diff --git a/docs/mobile-testing.md b/docs/mobile-testing.md index 3f71b3fe8..69062edf5 100644 --- a/docs/mobile-testing.md +++ b/docs/mobile-testing.md @@ -252,6 +252,25 @@ adb emu gsm call 5551234 Expected result: Paseo does not throw `RuntimeException: Audio focus request failed`; native audio reports an interruption and voice mode stops or pauses coherently. +### Releasing the audio session when idle + +Paseo must not hold the OS audio session once it is neither capturing nor playing, or the user's +background music stays paused. On iOS this is not just a "while recording" problem: the +`.playAndRecord`/`.voiceChat` category is non-mixing and survives backgrounding, and iOS re-asserts +it every time the app returns to the foreground — so one dictation turn kills music for the life of +the process, on every open, until the app is force-quit. Android holds +`AUDIOFOCUS_GAIN_TRANSIENT_EXCLUSIVE` with the same effect. + +`createAudioEngine` (`packages/app/src/voice/audio-engine.native.ts`) calls +`releaseAudioSession()` whenever capture stops and the playback queue drains, and the iOS module +also releases on `OnAppEntersBackground`. The native side re-guards on `isRecording` / +`speechPlayer.isPlaying`, because the native engine is a singleton shared by multiple JS engine +wrappers (voice provider + dictation) and only it knows the true state. + +This cannot be validated by a JS test — verify on a device: play music, open Paseo, use dictation +once, stop, and confirm the music resumes; then background/foreground the app and confirm it keeps +playing. + ## Unistyles + Reanimated ### The crash @@ -320,6 +339,111 @@ Platform-specific stream edges belong on `StreamStrategy`: If a chat footer looks duplicated or appears above the assistant message on mobile, start with `packages/app/src/agent-stream/layout.test.ts`. Do not add a React Native renderer test for this class of bug; make the pure layout invariant fail first. +## Local iOS device builds + +Native changes (anything under `packages/expo-two-way-audio/ios/`, or any new Expo module) **cannot be +tested over Metro or EAS Update**. Those ship JS only. A stale dev-client binary silently lacks the new +native functions, so the feature no-ops and the test proves nothing. Rebuild the binary. + +Symptom of a stale dev client: `Cannot find native module 'ExpoDocumentPicker'` at startup, followed by a +cascade of `Route "./X.tsx" is missing the required default export` warnings. The route warnings are not a +router bug — the module-level throw aborts `_layout.tsx` evaluation, so its default export never gets +assigned. One missing native module, N confusing warnings. + +**Prerequisite: Xcode needs a signed-in Apple ID for the signing team _with a valid keychain token_.** An +account that merely appears in Xcode → Settings → Accounts is not enough — if its `Xcode-Token` is missing, +the build below fails at code signing and blames a _certificate_ instead. See +[Signing needs a working Apple ID token in Xcode](#signing-needs-a-working-apple-id-token-in-xcode) before +you start, not after it fails. + +```bash +cd packages/app +npm --prefix ../.. run build:client +APP_VARIANT=development npx expo prebuild --platform ios +APP_VARIANT=development npx expo run:ios --device +``` + +`APP_VARIANT=development` is required. The `ios` npm script does not set it, and `app.config.js` defaults +to `production` — so a bare `npm run ios` builds `sh.paseo` and collides with the App Store install instead +of the `sh.paseo.debug` dev client. Ignore prebuild's `--non-interactive is not supported` warning; use +`CI=1` if you need non-interactive. + +### Signing needs a working Apple ID token in Xcode + +Device builds fail with a trio of errors that all point away from the real cause: + +``` +error: No Accounts: Add a new account in Accounts settings. +error: Provisioning profile "iOS Team Provisioning Profile: *" doesn't include + signing certificate "Apple Development: ". +error: Provisioning profile "iOS Team Provisioning Profile: *" doesn't include + the aps-environment entitlement. +``` + +**`No Accounts` does not reliably mean no account is configured.** Check the line _above_ it in the build +log before believing it: + +``` +DVTDeveloperAccountManager: Failed to load credentials for : + Invalid credentials in keychain for , missing Xcode-Token +``` + +That is the actual failure: the Apple ID is registered in Xcode, but its `Xcode-Token` keychain item is gone +(typically after an Apple ID password change or a keychain reset). Xcode reports it as `No Accounts`, then +falls back to whatever stale profile is cached — and if the machine holds a dev cert newer than that +profile, you get the _certificate_ mismatch as a second-order symptom. Chasing the certificate error is a +dead end. + +Verify what is actually configured rather than trusting the message: + +```bash +# Accounts Xcode knows about +defaults read com.apple.dt.Xcode DVTDeveloperAccountManagerAppleIDLists + +# The credential that is actually missing (error => no token) +security find-generic-password -s "Xcode-Token" +``` + +Fix: Xcode → Settings → Accounts, re-sign-in to the listed Apple ID (removing and re-adding forces a fresh +token). That restores `-allowProvisioningUpdates`, profile regeneration, and push. + +Signing an Xcode-_managed_ profile via `CODE_SIGN_STYLE=Manual` is not a workaround; Xcode rejects it with +`is Xcode managed, but signing settings require a manually managed profile`. + +If you need a build on the device before the token is fixed, build unsigned and sign by hand with a cert the +cached profile already embeds (list them with `security cms -D -i .mobileprovision`, then compare +against `security find-identity -v -p codesigning`): + +```bash +xcodebuild ... CODE_SIGNING_ALLOWED=NO CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY="" build +cp .mobileprovision "$APP/embedded.mobileprovision" +# sign nested frameworks/dylibs first, then the bundle, then: +xcrun devicectl device install app --device "$APP" +``` + +This is a stopgap, not a fix: `expo prebuild` regenerates `packages/app/ios/` and discards it. It also +requires stripping entitlements the cached profile lacks (e.g. `aps-environment`), so push is dead in such a +build. + +To verify only that native Swift compiles, skip signing entirely — the pods have their own schemes: + +```bash +cd packages/app/ios +xcodebuild -workspace PaseoDebug.xcworkspace -scheme ExpoTwoWayAudio \ + -sdk iphoneos -destination 'generic/platform=iOS' CODE_SIGNING_ALLOWED=NO build +``` + +Confirm the log compiled _your_ file and not a cached copy: the `SwiftCompile` line should reference +`packages/expo-two-way-audio/ios/...`, not a path inside DerivedData. + +### Headless launch cannot verify runtime behavior + +`xcrun devicectl device process launch` reports `The app terminated with the exit code 0` about a second +after launch when the phone's screen is off — the app is suspended and killed before the JS bundle loads. +This is **not** a build defect. Confirm with a control: launch `com.apple.Preferences` the same way and +watch it exit identically. Headless install and launch prove the binary is valid and that startup reaches +RN init; anything about on-screen behavior needs a human holding the phone. + ## iOS Simulator ```bash diff --git a/packages/app/src/voice/audio-engine.native.ts b/packages/app/src/voice/audio-engine.native.ts index 81f0fda85..465bab3c7 100644 --- a/packages/app/src/voice/audio-engine.native.ts +++ b/packages/app/src/voice/audio-engine.native.ts @@ -143,6 +143,23 @@ export function createAudioEngine( refs.initialized = true; } + /** + * Release the OS audio session as soon as we are neither capturing nor playing. + * Holding it keeps the user's background music paused — on iOS the non-mixing + * `.playAndRecord` category survives backgrounding and is re-asserted on every + * foreground, so an unreleased session means their music never comes back. + */ + function releaseSessionIfIdle(): void { + if (!refs.initialized || refs.destroyed) { + return; + } + if (refs.captureActive || refs.activePlayback || refs.queue.length > 0) { + return; + } + // The wrapper no-ops on binaries whose native module predates this function. + native.releaseAudioSession(); + } + async function ensureMicrophonePermission(): Promise { let permission = await native.getMicrophonePermissionsAsync().catch(() => null); if (!permission?.granted) { @@ -222,6 +239,7 @@ export function createAudioEngine( } } refs.processingQueue = false; + releaseSessionIfIdle(); } return { @@ -281,6 +299,7 @@ export function createAudioEngine( refs.captureActive = false; refs.muted = false; callbacks.onVolumeLevel(0); + releaseSessionIfIdle(); }, toggleMute() { @@ -313,6 +332,7 @@ export function createAudioEngine( active.settled = true; active.reject(new Error("Playback stopped")); } + releaseSessionIfIdle(); }, clearQueue() { @@ -320,6 +340,7 @@ export function createAudioEngine( refs.queue.shift()!.reject(new Error("Playback stopped")); } refs.processingQueue = false; + releaseSessionIfIdle(); }, isPlaying() { diff --git a/packages/expo-two-way-audio/android/src/main/java/expo/modules/twowayaudio/AudioEngine.kt b/packages/expo-two-way-audio/android/src/main/java/expo/modules/twowayaudio/AudioEngine.kt index cfa32ba59..9906e5a1d 100644 --- a/packages/expo-two-way-audio/android/src/main/java/expo/modules/twowayaudio/AudioEngine.kt +++ b/packages/expo-two-way-audio/android/src/main/java/expo/modules/twowayaudio/AudioEngine.kt @@ -169,6 +169,52 @@ class AudioEngine (context: Context) { } } + /** + * Give audio focus back once we are neither recording nor playing. The focus request is + * AUDIOFOCUS_GAIN_TRANSIENT_EXCLUSIVE, so holding it after a dictation turn leaves the + * user's music paused indefinitely. + */ + @SuppressLint("NewApi") + fun releaseAudioSession() { + if (isRecording || isPlaying) { + return + } + audioFocusRequest?.let { request -> + audioManager.abandonAudioFocusRequest(request) + audioFocusRequest = null + } + if (::audioTrack.isInitialized) { + audioTrack.pause() + } + } + + /** + * Take the audio session back before playing, mirroring iOS `activateAudioSessionIfNeeded()`. + * + * `releaseAudioSession()` abandons focus and pauses the track once we go idle, so a playback + * turn that starts after that release has to reclaim both. Without this, the assistant's + * response plays with no focus held and competes with whatever the user is listening to. + * A null [audioFocusRequest] is what "we released" looks like, so it is the guard against + * re-requesting focus mid-turn. + * + * Focus is best effort here, and deliberately not fatal. `playPCMData` is fire-and-forget + * from JS — `playAudio()` settles on a duration timer, not on a native ack — so refusing to + * queue would report a chunk as played that nobody heard. Playing unfocused is what this + * path did before the session was ever released, and it is the better failure. Leaving the + * request outstanding instead of abandoning it is also what lets a delayed grant arrive + * later through the listener, which is the point of `setAcceptsDelayedFocusGain(true)`. + */ + @SuppressLint("NewApi") + private fun acquireAudioSessionIfNeeded() { + if (audioFocusRequest != null) { + return + } + requestAudioFocus() + if (::audioTrack.isInitialized) { + audioTrack.play() + } + } + @SuppressLint("NewApi") private fun requestAudioFocus(): Boolean { audioFocusRequest?.let { request -> @@ -327,7 +373,9 @@ class AudioEngine (context: Context) { return isRecording } + @SuppressLint("NewApi") fun playPCMData(data: ByteArray) { + acquireAudioSessionIfNeeded() audioSampleQueue.add(data) playbackEvents += 1 playbackQueuedBytes += data.size.toLong() @@ -397,11 +445,18 @@ class AudioEngine (context: Context) { @RequiresApi(Build.VERSION_CODES.Q) fun resumeRecordingAndPlayer() { + val wasRecordingBeforePause = isRecordingBeforePause + // Only take the session back if something was actually live when we backgrounded. The + // activity lifecycle listener calls this on every onResume, so requesting + // AUDIOFOCUS_GAIN_TRANSIENT_EXCLUSIVE unconditionally re-paused the user's music every + // time the app came to the foreground — the exact symptom this change exists to remove. + if (!wasRecordingBeforePause && !isPlaying) { + return + } if (!requestAudioFocus()) { handleAudioFocusBlocked() return } - val wasRecordingBeforePause = isRecordingBeforePause isRecording = toggleRecording(wasRecordingBeforePause) if (wasRecordingBeforePause && !isRecording) { return @@ -423,7 +478,9 @@ class AudioEngine (context: Context) { Log.d("AudioEngine", "Playback paused") } + @SuppressLint("NewApi") fun resumePlayback() { + acquireAudioSessionIfNeeded() audioTrack.play() Log.d("AudioEngine", "Playback resumed") } diff --git a/packages/expo-two-way-audio/android/src/main/java/expo/modules/twowayaudio/ExpoTwoWayAudioModule.kt b/packages/expo-two-way-audio/android/src/main/java/expo/modules/twowayaudio/ExpoTwoWayAudioModule.kt index 95d6dfb3b..f9b107c99 100644 --- a/packages/expo-two-way-audio/android/src/main/java/expo/modules/twowayaudio/ExpoTwoWayAudioModule.kt +++ b/packages/expo-two-way-audio/android/src/main/java/expo/modules/twowayaudio/ExpoTwoWayAudioModule.kt @@ -33,6 +33,11 @@ class ExpoTwoWayAudioModule : Module() { } } + Function("releaseAudioSession") { + audioEngine?.releaseAudioSession() + null + } + Function("isRecording") { audioEngine?.isRecording ?: false } diff --git a/packages/expo-two-way-audio/ios/AudioEngine.swift b/packages/expo-two-way-audio/ios/AudioEngine.swift index 65dcf62e9..d385dab7a 100644 --- a/packages/expo-two-way-audio/ios/AudioEngine.swift +++ b/packages/expo-two-way-audio/ios/AudioEngine.swift @@ -10,6 +10,7 @@ class AudioEngine { public private(set) var voiceIOFormat: AVAudioFormat public private(set) var isRecording = false + public private(set) var isSessionActive = false public var onMicDataCallback: ((Data) -> Void)? public var onInputVolumeCallback: ((Float) -> Void)? @@ -27,6 +28,22 @@ class AudioEngine { private var hasFirstInputBeenDiscarded = false private var discardRecording = false private var discardFirstInputMillis = 2000 + + /// Buffers scheduled on `speechPlayer` that have not finished rendering yet. + /// + /// `AVAudioPlayerNode.isPlaying` cannot answer "is audio still in flight": it stays true from + /// `play()` until an explicit `stop()`/`pause()`, and draining every scheduled buffer does not + /// clear it. Using it as the release guard meant the session was never handed back after the + /// first spoken response. Decremented from the scheduling completion handler, which runs on an + /// AVAudioEngine-internal thread, so all access goes through `playbackCountLock`. + private var pendingPlaybackBuffers = 0 + private let playbackCountLock = NSLock() + + private var hasPendingPlayback: Bool { + playbackCountLock.lock() + defer { playbackCountLock.unlock() } + return pendingPlaybackBuffers > 0 + } enum AudioEngineError: Error { case audioFormatError @@ -79,26 +96,64 @@ class AudioEngine { func setupAudioSession() { let session = AVAudioSession.sharedInstance() - + do { try session.setCategory(.playAndRecord, mode: .voiceChat, options: [.defaultToSpeaker, .allowBluetooth, .allowBluetoothA2DP]) } catch { print("Could not set the audio category: \(error.localizedDescription)") } - + do { try session.setPreferredSampleRate(voiceIOFormat.sampleRate) } catch { print("Could not set the preferred sample rate: \(error.localizedDescription)") } - + do { try session.setActive(true) + isSessionActive = true } catch { print("Could not set the audio session as active") } } - + + func activateAudioSessionIfNeeded() { + guard !isSessionActive else { return } + setupAudioSession() + checkEngineIsRunning() + } + + /// Hand the audio session back to the system. + /// + /// `.playAndRecord` + `.voiceChat` does not mix, so while it is active the user's + /// background music stays dead — including across backgrounding, because iOS + /// re-asserts whatever category the app last set when it returns to the foreground. + /// Releasing when we are neither capturing nor playing is what lets their music come + /// back; `.notifyOthersOnDeactivation` is what tells the other app to resume, and + /// dropping to `.ambient` means any implicit reactivation mixes instead of interrupting. + func releaseAudioSession() { + // The native engine is a singleton shared by every JS-side engine wrapper, so it is the + // only layer that knows whether *anything* is still using the session. Never release + // while capture or playback is live — the JS callers each only see their own queue. + guard isSessionActive, !isRecording, !hasPendingPlayback else { return } + + avAudioEngine.pause() + + let session = AVAudioSession.sharedInstance() + do { + try session.setActive(false, options: [.notifyOthersOnDeactivation]) + } catch { + print("Could not deactivate the audio session: \(error.localizedDescription)") + } + do { + try session.setCategory(.ambient, options: [.mixWithOthers]) + } catch { + print("Could not reset the audio category: \(error.localizedDescription)") + } + + isSessionActive = false + } + func setup() { let input = avAudioEngine.inputNode do { @@ -183,6 +238,8 @@ class AudioEngine { } func playPCMData(_ pcmData: Data) { + activateAudioSessionIfNeeded() + // Looks like we don't get a proper AEC for the very first chunks of audio that we play. // To work around this, we will discard microphone input for the first few milliseconds. // This will give the AEC time to adapt to the playback audio. @@ -200,8 +257,18 @@ class AudioEngine { print("Failed to create audio buffer") return } - speechPlayer.scheduleBuffer(buffer) - + playbackCountLock.lock() + pendingPlaybackBuffers += 1 + playbackCountLock.unlock() + speechPlayer.scheduleBuffer(buffer, completionCallbackType: .dataPlayedBack) { [weak self] _ in + guard let self = self else { return } + self.playbackCountLock.lock() + if self.pendingPlaybackBuffers > 0 { + self.pendingPlaybackBuffers -= 1 + } + self.playbackCountLock.unlock() + } + if !speechPlayer.isPlaying { speechPlayer.play() } @@ -247,6 +314,7 @@ class AudioEngine { inputBuffer = [Float](repeating: 0, count: 2048) updateInputVolume() } else { + activateAudioSessionIfNeeded() avAudioEngine.inputNode.isVoiceProcessingInputMuted = false } print("Recording \(isRecording ? "started" : "stopped")") @@ -255,22 +323,23 @@ class AudioEngine { } func stopRecordingAndPlayer(){ - do { - try AVAudioSession.sharedInstance().setActive(false) - } catch { - print("Could not set the audio session to inactive: \(error)") - } toggleRecording(false) speechPlayer.stop() + resetPendingPlayback() updateOutputVolume() + releaseAudioSession() } - + + /// `stop()` discards whatever is still scheduled, so the pending count has to be cleared with + /// it. Leaving it non-zero would block every later release. + private func resetPendingPlayback() { + playbackCountLock.lock() + pendingPlaybackBuffers = 0 + playbackCountLock.unlock() + } + func resumeRecordingAndPlayer(){ - do { - try AVAudioSession.sharedInstance().setActive(true) - } catch { - print("Could not set the audio session to active: \(error)") - } + activateAudioSessionIfNeeded() self.checkEngineIsRunning() isRecording = toggleRecording(true) speechPlayer.play() @@ -287,6 +356,7 @@ class AudioEngine { func stopPlayback() { speechPlayer.stop() + resetPendingPlayback() // Clear any scheduled buffers outputBuffer = [Float](repeating: 0, count: 2048) updateOutputVolume() @@ -306,6 +376,11 @@ class AudioEngine { } private func checkEngineIsRunning() { + // Starting the engine implicitly reactivates the audio session. Never do that while the + // session is released, or a stray configuration-change notification (which releasing + // itself can trigger, since it changes the category) would silently grab the user's + // audio back and undo the release. + guard isSessionActive else { return } if !avAudioEngine.isRunning { start() } @@ -340,6 +415,9 @@ class AudioEngine { private func handleMediaServicesWereReset() { self.avAudioEngine.stop() self.setup() + // Only bring the engine back up if we still own the session; otherwise wait until the + // next capture/playback reactivates it. + guard isSessionActive else { return } self.start() } diff --git a/packages/expo-two-way-audio/ios/ExpoTwoWayAudioModule.swift b/packages/expo-two-way-audio/ios/ExpoTwoWayAudioModule.swift index f198580ac..baec74799 100644 --- a/packages/expo-two-way-audio/ios/ExpoTwoWayAudioModule.swift +++ b/packages/expo-two-way-audio/ios/ExpoTwoWayAudioModule.swift @@ -39,6 +39,18 @@ public class ExpoTwoWayAudioModule: Module { } } + // Safety net for the JS-driven release in `stopCapture`/`processQueue`: never leave + // the non-mixing `.playAndRecord` session active while backgrounded, or iOS + // re-asserts it on the next foreground and kills the user's music again. + OnAppEntersBackground { + guard self.audioEngine?.isRecording == false else { return } + self.audioEngine?.releaseAudioSession() + } + + Function("releaseAudioSession") { + self.audioEngine?.releaseAudioSession() + } + Function("isRecording") { () -> Bool in guard let audioEngine = self.audioEngine else { print("AudioEngine not initialized") diff --git a/packages/expo-two-way-audio/src/core.ts b/packages/expo-two-way-audio/src/core.ts index c5d92aa7c..1a54e0e9c 100644 --- a/packages/expo-two-way-audio/src/core.ts +++ b/packages/expo-two-way-audio/src/core.ts @@ -21,6 +21,20 @@ export function isRecording(): boolean { return ExpoTwoWayAudioModule.isRecording(); } +/** + * Hand the OS audio session / audio focus back once nothing is being captured or played, + * so other apps' audio can resume. Safe to call when already released. + */ +export function releaseAudioSession() { + // COMPAT(releaseAudioSession): added in v0.2.6. An OTA JS update can land on an older + // binary whose native module lacks this function, so probe the native object rather than + // this wrapper (which always exists). Drop the guard once the binary floor is >= v0.2.6. + if (typeof ExpoTwoWayAudioModule.releaseAudioSession !== "function") { + return; + } + return ExpoTwoWayAudioModule.releaseAudioSession(); +} + export function tearDown() { return ExpoTwoWayAudioModule.tearDown(); }