Release the audio session when voice capture and playback go idle (#2866)
* Release the audio session when voice capture and playback go idle Opening Paseo permanently paused music playing in other apps. Once the mic had been used a single time, every subsequent foreground of the app killed the user's audio again until the app was force-quit. iOS AudioEngine.setupAudioSession() takes .playAndRecord/.voiceChat — a non-mixing category — and calls setActive(true), but the only setActive(false) was reachable from tearDown(). Nothing handed the session back after a normal dictation turn, so iOS re-asserted the last category the app set on every foreground. The iOS module had no app-lifecycle handling at all, while Android already had a lifecycle listener; that asymmetry is what made the platforms diverge. Android had the same leak in a different shape: AUDIOFOCUS_GAIN_TRANSIENT_EXCLUSIVE was requested per recording and never abandoned. Add releaseAudioSession() to both platforms and call it once capture stops and the playback queue drains. iOS pauses the engine, calls setActive(false, .notifyOthersOnDeactivation) — this is what tells the other app to resume — then drops to .ambient/.mixWithOthers so any implicit reactivation mixes instead of interrupting. The session is re-acquired lazily on toggleRecording(true)/playPCMData, so voice keeps working; resumeRecordingAndPlayer() re-activates before restarting the engine, which keeps post-interruption resume intact. Releasing is guarded on isRecording/isPlaying, and checkEngineIsRunning() now refuses to start the engine while the session is released, since starting it would silently re-take the session. The JS wrapper probes the native module before calling, because a JS bundle shipped over EAS Update can land on a binary predating this function. * Correct the iOS signing diagnosis in mobile-testing docs The "No Accounts" error does not mean no Apple ID is configured. The account can be registered in Xcode while its Xcode-Token keychain item is missing, which Xcode reports as No Accounts and then compounds by falling back to a stale cached profile — surfacing as a certificate mismatch that sends you chasing the wrong thing. Document the DVTDeveloperAccountManager log line that names the real cause, the two commands that check what is actually configured, and the build-unsigned-then-sign-manually stopgap. * Reacquire Android audio focus when playback restarts releaseAudioSession() abandons the AUDIOFOCUS_GAIN_TRANSIENT_EXCLUSIVE request and pauses the AudioTrack once capture and playback go idle, but nothing took either back. The next playback turn therefore ran with no focus held, so the assistant's response competed with whatever the user was listening to instead of interrupting it. Add acquireAudioSessionIfNeeded(), the mirror of the iOS activateAudioSessionIfNeeded() already called from playPCMData, and use it from both playPCMData() and resumePlayback(). A null audioFocusRequest is what "we released" looks like, so it doubles as the guard against re-requesting focus mid-turn. Reported by Greptile on #2866. * Make Android playback focus reacquisition non-fatal The previous commit dropped PCM data when focus reacquisition did not return AUDIOFOCUS_REQUEST_GRANTED. That is worse than playing unfocused: playPCMData is fire-and-forget from JS, and playAudio() settles on a duration timer rather than a native ack, so the runtime acknowledged and discarded a chunk that was never queued. It also broke delayed focus. The request is built with setAcceptsDelayedFocusGain(true), so AUDIOFOCUS_REQUEST_DELAYED is a normal outcome, but routing it through handleAudioFocusBlocked() abandoned the pending request and the later grant could never arrive. Request focus, then play regardless of the result, and leave the request outstanding so a delayed grant still lands via the listener. Reported by Greptile on #2866. * Track scheduled buffers so iOS actually releases after playback releaseAudioSession() guarded on !speechPlayer.isPlaying, but AVAudioPlayerNode.isPlaying stays true from play() until an explicit stop()/pause() — draining every scheduled buffer does not clear it. The normal completion path is processQueue() -> releaseSessionIfIdle(), and stopPlayback() is only reached from the explicit stop() API, so after the first spoken response the guard blocked every release and the session stayed active. The music-pausing bug returned in full for anyone using voice responses. Count buffers that are scheduled but not yet rendered, decrementing from the .dataPlayedBack completion handler, and guard on that instead. The handler runs on an AVAudioEngine-internal thread, so the count is behind a lock. Both stop() paths reset it, since stop() discards what is scheduled and a stale count would block every later release. Found by Codex review on #2866. * Stop re-taking Android audio focus on every foreground ExpoTwoWayAudioLifeCycleListener calls resumeRecordingAndPlayer() from onResume, and that requested AUDIOFOCUS_GAIN_TRANSIENT_EXCLUSIVE before looking at isRecordingBeforePause. Foregrounding the app while nothing was recording therefore took exclusive focus back and re-paused the user's music — the same "every foreground kills my music" symptom this change set exists to remove, just reached through the Android lifecycle instead of the iOS category. Return early unless capture or playback was actually live when we backgrounded. Found by Codex review on #2866.
This commit is contained in:
@@ -252,6 +252,25 @@ adb emu gsm call 5551234
|
|||||||
|
|
||||||
Expected result: Paseo does not throw `RuntimeException: Audio focus request failed`; native audio reports an interruption and voice mode stops or pauses coherently.
|
Expected result: Paseo does not throw `RuntimeException: Audio focus request failed`; native audio reports an interruption and voice mode stops or pauses coherently.
|
||||||
|
|
||||||
|
### Releasing the audio session when idle
|
||||||
|
|
||||||
|
Paseo must not hold the OS audio session once it is neither capturing nor playing, or the user's
|
||||||
|
background music stays paused. On iOS this is not just a "while recording" problem: the
|
||||||
|
`.playAndRecord`/`.voiceChat` category is non-mixing and survives backgrounding, and iOS re-asserts
|
||||||
|
it every time the app returns to the foreground — so one dictation turn kills music for the life of
|
||||||
|
the process, on every open, until the app is force-quit. Android holds
|
||||||
|
`AUDIOFOCUS_GAIN_TRANSIENT_EXCLUSIVE` with the same effect.
|
||||||
|
|
||||||
|
`createAudioEngine` (`packages/app/src/voice/audio-engine.native.ts`) calls
|
||||||
|
`releaseAudioSession()` whenever capture stops and the playback queue drains, and the iOS module
|
||||||
|
also releases on `OnAppEntersBackground`. The native side re-guards on `isRecording` /
|
||||||
|
`speechPlayer.isPlaying`, because the native engine is a singleton shared by multiple JS engine
|
||||||
|
wrappers (voice provider + dictation) and only it knows the true state.
|
||||||
|
|
||||||
|
This cannot be validated by a JS test — verify on a device: play music, open Paseo, use dictation
|
||||||
|
once, stop, and confirm the music resumes; then background/foreground the app and confirm it keeps
|
||||||
|
playing.
|
||||||
|
|
||||||
## Unistyles + Reanimated
|
## Unistyles + Reanimated
|
||||||
|
|
||||||
### The crash
|
### The crash
|
||||||
@@ -320,6 +339,111 @@ Platform-specific stream edges belong on `StreamStrategy`:
|
|||||||
|
|
||||||
If a chat footer looks duplicated or appears above the assistant message on mobile, start with `packages/app/src/agent-stream/layout.test.ts`. Do not add a React Native renderer test for this class of bug; make the pure layout invariant fail first.
|
If a chat footer looks duplicated or appears above the assistant message on mobile, start with `packages/app/src/agent-stream/layout.test.ts`. Do not add a React Native renderer test for this class of bug; make the pure layout invariant fail first.
|
||||||
|
|
||||||
|
## Local iOS device builds
|
||||||
|
|
||||||
|
Native changes (anything under `packages/expo-two-way-audio/ios/`, or any new Expo module) **cannot be
|
||||||
|
tested over Metro or EAS Update**. Those ship JS only. A stale dev-client binary silently lacks the new
|
||||||
|
native functions, so the feature no-ops and the test proves nothing. Rebuild the binary.
|
||||||
|
|
||||||
|
Symptom of a stale dev client: `Cannot find native module 'ExpoDocumentPicker'` at startup, followed by a
|
||||||
|
cascade of `Route "./X.tsx" is missing the required default export` warnings. The route warnings are not a
|
||||||
|
router bug — the module-level throw aborts `_layout.tsx` evaluation, so its default export never gets
|
||||||
|
assigned. One missing native module, N confusing warnings.
|
||||||
|
|
||||||
|
**Prerequisite: Xcode needs a signed-in Apple ID for the signing team _with a valid keychain token_.** An
|
||||||
|
account that merely appears in Xcode → Settings → Accounts is not enough — if its `Xcode-Token` is missing,
|
||||||
|
the build below fails at code signing and blames a _certificate_ instead. See
|
||||||
|
[Signing needs a working Apple ID token in Xcode](#signing-needs-a-working-apple-id-token-in-xcode) before
|
||||||
|
you start, not after it fails.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd packages/app
|
||||||
|
npm --prefix ../.. run build:client
|
||||||
|
APP_VARIANT=development npx expo prebuild --platform ios
|
||||||
|
APP_VARIANT=development npx expo run:ios --device
|
||||||
|
```
|
||||||
|
|
||||||
|
`APP_VARIANT=development` is required. The `ios` npm script does not set it, and `app.config.js` defaults
|
||||||
|
to `production` — so a bare `npm run ios` builds `sh.paseo` and collides with the App Store install instead
|
||||||
|
of the `sh.paseo.debug` dev client. Ignore prebuild's `--non-interactive is not supported` warning; use
|
||||||
|
`CI=1` if you need non-interactive.
|
||||||
|
|
||||||
|
### Signing needs a working Apple ID token in Xcode
|
||||||
|
|
||||||
|
Device builds fail with a trio of errors that all point away from the real cause:
|
||||||
|
|
||||||
|
```
|
||||||
|
error: No Accounts: Add a new account in Accounts settings.
|
||||||
|
error: Provisioning profile "iOS Team Provisioning Profile: *" doesn't include
|
||||||
|
signing certificate "Apple Development: <name>".
|
||||||
|
error: Provisioning profile "iOS Team Provisioning Profile: *" doesn't include
|
||||||
|
the aps-environment entitlement.
|
||||||
|
```
|
||||||
|
|
||||||
|
**`No Accounts` does not reliably mean no account is configured.** Check the line _above_ it in the build
|
||||||
|
log before believing it:
|
||||||
|
|
||||||
|
```
|
||||||
|
DVTDeveloperAccountManager: Failed to load credentials for <apple-id>:
|
||||||
|
Invalid credentials in keychain for <apple-id>, missing Xcode-Token
|
||||||
|
```
|
||||||
|
|
||||||
|
That is the actual failure: the Apple ID is registered in Xcode, but its `Xcode-Token` keychain item is gone
|
||||||
|
(typically after an Apple ID password change or a keychain reset). Xcode reports it as `No Accounts`, then
|
||||||
|
falls back to whatever stale profile is cached — and if the machine holds a dev cert newer than that
|
||||||
|
profile, you get the _certificate_ mismatch as a second-order symptom. Chasing the certificate error is a
|
||||||
|
dead end.
|
||||||
|
|
||||||
|
Verify what is actually configured rather than trusting the message:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Accounts Xcode knows about
|
||||||
|
defaults read com.apple.dt.Xcode DVTDeveloperAccountManagerAppleIDLists
|
||||||
|
|
||||||
|
# The credential that is actually missing (error => no token)
|
||||||
|
security find-generic-password -s "Xcode-Token"
|
||||||
|
```
|
||||||
|
|
||||||
|
Fix: Xcode → Settings → Accounts, re-sign-in to the listed Apple ID (removing and re-adding forces a fresh
|
||||||
|
token). That restores `-allowProvisioningUpdates`, profile regeneration, and push.
|
||||||
|
|
||||||
|
Signing an Xcode-_managed_ profile via `CODE_SIGN_STYLE=Manual` is not a workaround; Xcode rejects it with
|
||||||
|
`is Xcode managed, but signing settings require a manually managed profile`.
|
||||||
|
|
||||||
|
If you need a build on the device before the token is fixed, build unsigned and sign by hand with a cert the
|
||||||
|
cached profile already embeds (list them with `security cms -D -i <profile>.mobileprovision`, then compare
|
||||||
|
against `security find-identity -v -p codesigning`):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
xcodebuild ... CODE_SIGNING_ALLOWED=NO CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY="" build
|
||||||
|
cp <profile>.mobileprovision "$APP/embedded.mobileprovision"
|
||||||
|
# sign nested frameworks/dylibs first, then the bundle, then:
|
||||||
|
xcrun devicectl device install app --device <udid> "$APP"
|
||||||
|
```
|
||||||
|
|
||||||
|
This is a stopgap, not a fix: `expo prebuild` regenerates `packages/app/ios/` and discards it. It also
|
||||||
|
requires stripping entitlements the cached profile lacks (e.g. `aps-environment`), so push is dead in such a
|
||||||
|
build.
|
||||||
|
|
||||||
|
To verify only that native Swift compiles, skip signing entirely — the pods have their own schemes:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd packages/app/ios
|
||||||
|
xcodebuild -workspace PaseoDebug.xcworkspace -scheme ExpoTwoWayAudio \
|
||||||
|
-sdk iphoneos -destination 'generic/platform=iOS' CODE_SIGNING_ALLOWED=NO build
|
||||||
|
```
|
||||||
|
|
||||||
|
Confirm the log compiled _your_ file and not a cached copy: the `SwiftCompile` line should reference
|
||||||
|
`packages/expo-two-way-audio/ios/...`, not a path inside DerivedData.
|
||||||
|
|
||||||
|
### Headless launch cannot verify runtime behavior
|
||||||
|
|
||||||
|
`xcrun devicectl device process launch` reports `The app terminated with the exit code 0` about a second
|
||||||
|
after launch when the phone's screen is off — the app is suspended and killed before the JS bundle loads.
|
||||||
|
This is **not** a build defect. Confirm with a control: launch `com.apple.Preferences` the same way and
|
||||||
|
watch it exit identically. Headless install and launch prove the binary is valid and that startup reaches
|
||||||
|
RN init; anything about on-screen behavior needs a human holding the phone.
|
||||||
|
|
||||||
## iOS Simulator
|
## iOS Simulator
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -143,6 +143,23 @@ export function createAudioEngine(
|
|||||||
refs.initialized = true;
|
refs.initialized = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Release the OS audio session as soon as we are neither capturing nor playing.
|
||||||
|
* Holding it keeps the user's background music paused — on iOS the non-mixing
|
||||||
|
* `.playAndRecord` category survives backgrounding and is re-asserted on every
|
||||||
|
* foreground, so an unreleased session means their music never comes back.
|
||||||
|
*/
|
||||||
|
function releaseSessionIfIdle(): void {
|
||||||
|
if (!refs.initialized || refs.destroyed) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (refs.captureActive || refs.activePlayback || refs.queue.length > 0) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// The wrapper no-ops on binaries whose native module predates this function.
|
||||||
|
native.releaseAudioSession();
|
||||||
|
}
|
||||||
|
|
||||||
async function ensureMicrophonePermission(): Promise<void> {
|
async function ensureMicrophonePermission(): Promise<void> {
|
||||||
let permission = await native.getMicrophonePermissionsAsync().catch(() => null);
|
let permission = await native.getMicrophonePermissionsAsync().catch(() => null);
|
||||||
if (!permission?.granted) {
|
if (!permission?.granted) {
|
||||||
@@ -222,6 +239,7 @@ export function createAudioEngine(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
refs.processingQueue = false;
|
refs.processingQueue = false;
|
||||||
|
releaseSessionIfIdle();
|
||||||
}
|
}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
@@ -281,6 +299,7 @@ export function createAudioEngine(
|
|||||||
refs.captureActive = false;
|
refs.captureActive = false;
|
||||||
refs.muted = false;
|
refs.muted = false;
|
||||||
callbacks.onVolumeLevel(0);
|
callbacks.onVolumeLevel(0);
|
||||||
|
releaseSessionIfIdle();
|
||||||
},
|
},
|
||||||
|
|
||||||
toggleMute() {
|
toggleMute() {
|
||||||
@@ -313,6 +332,7 @@ export function createAudioEngine(
|
|||||||
active.settled = true;
|
active.settled = true;
|
||||||
active.reject(new Error("Playback stopped"));
|
active.reject(new Error("Playback stopped"));
|
||||||
}
|
}
|
||||||
|
releaseSessionIfIdle();
|
||||||
},
|
},
|
||||||
|
|
||||||
clearQueue() {
|
clearQueue() {
|
||||||
@@ -320,6 +340,7 @@ export function createAudioEngine(
|
|||||||
refs.queue.shift()!.reject(new Error("Playback stopped"));
|
refs.queue.shift()!.reject(new Error("Playback stopped"));
|
||||||
}
|
}
|
||||||
refs.processingQueue = false;
|
refs.processingQueue = false;
|
||||||
|
releaseSessionIfIdle();
|
||||||
},
|
},
|
||||||
|
|
||||||
isPlaying() {
|
isPlaying() {
|
||||||
|
|||||||
+58
-1
@@ -169,6 +169,52 @@ class AudioEngine (context: Context) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Give audio focus back once we are neither recording nor playing. The focus request is
|
||||||
|
* AUDIOFOCUS_GAIN_TRANSIENT_EXCLUSIVE, so holding it after a dictation turn leaves the
|
||||||
|
* user's music paused indefinitely.
|
||||||
|
*/
|
||||||
|
@SuppressLint("NewApi")
|
||||||
|
fun releaseAudioSession() {
|
||||||
|
if (isRecording || isPlaying) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
audioFocusRequest?.let { request ->
|
||||||
|
audioManager.abandonAudioFocusRequest(request)
|
||||||
|
audioFocusRequest = null
|
||||||
|
}
|
||||||
|
if (::audioTrack.isInitialized) {
|
||||||
|
audioTrack.pause()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Take the audio session back before playing, mirroring iOS `activateAudioSessionIfNeeded()`.
|
||||||
|
*
|
||||||
|
* `releaseAudioSession()` abandons focus and pauses the track once we go idle, so a playback
|
||||||
|
* turn that starts after that release has to reclaim both. Without this, the assistant's
|
||||||
|
* response plays with no focus held and competes with whatever the user is listening to.
|
||||||
|
* A null [audioFocusRequest] is what "we released" looks like, so it is the guard against
|
||||||
|
* re-requesting focus mid-turn.
|
||||||
|
*
|
||||||
|
* Focus is best effort here, and deliberately not fatal. `playPCMData` is fire-and-forget
|
||||||
|
* from JS — `playAudio()` settles on a duration timer, not on a native ack — so refusing to
|
||||||
|
* queue would report a chunk as played that nobody heard. Playing unfocused is what this
|
||||||
|
* path did before the session was ever released, and it is the better failure. Leaving the
|
||||||
|
* request outstanding instead of abandoning it is also what lets a delayed grant arrive
|
||||||
|
* later through the listener, which is the point of `setAcceptsDelayedFocusGain(true)`.
|
||||||
|
*/
|
||||||
|
@SuppressLint("NewApi")
|
||||||
|
private fun acquireAudioSessionIfNeeded() {
|
||||||
|
if (audioFocusRequest != null) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
requestAudioFocus()
|
||||||
|
if (::audioTrack.isInitialized) {
|
||||||
|
audioTrack.play()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
@SuppressLint("NewApi")
|
@SuppressLint("NewApi")
|
||||||
private fun requestAudioFocus(): Boolean {
|
private fun requestAudioFocus(): Boolean {
|
||||||
audioFocusRequest?.let { request ->
|
audioFocusRequest?.let { request ->
|
||||||
@@ -327,7 +373,9 @@ class AudioEngine (context: Context) {
|
|||||||
return isRecording
|
return isRecording
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@SuppressLint("NewApi")
|
||||||
fun playPCMData(data: ByteArray) {
|
fun playPCMData(data: ByteArray) {
|
||||||
|
acquireAudioSessionIfNeeded()
|
||||||
audioSampleQueue.add(data)
|
audioSampleQueue.add(data)
|
||||||
playbackEvents += 1
|
playbackEvents += 1
|
||||||
playbackQueuedBytes += data.size.toLong()
|
playbackQueuedBytes += data.size.toLong()
|
||||||
@@ -397,11 +445,18 @@ class AudioEngine (context: Context) {
|
|||||||
|
|
||||||
@RequiresApi(Build.VERSION_CODES.Q)
|
@RequiresApi(Build.VERSION_CODES.Q)
|
||||||
fun resumeRecordingAndPlayer() {
|
fun resumeRecordingAndPlayer() {
|
||||||
|
val wasRecordingBeforePause = isRecordingBeforePause
|
||||||
|
// Only take the session back if something was actually live when we backgrounded. The
|
||||||
|
// activity lifecycle listener calls this on every onResume, so requesting
|
||||||
|
// AUDIOFOCUS_GAIN_TRANSIENT_EXCLUSIVE unconditionally re-paused the user's music every
|
||||||
|
// time the app came to the foreground — the exact symptom this change exists to remove.
|
||||||
|
if (!wasRecordingBeforePause && !isPlaying) {
|
||||||
|
return
|
||||||
|
}
|
||||||
if (!requestAudioFocus()) {
|
if (!requestAudioFocus()) {
|
||||||
handleAudioFocusBlocked()
|
handleAudioFocusBlocked()
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
val wasRecordingBeforePause = isRecordingBeforePause
|
|
||||||
isRecording = toggleRecording(wasRecordingBeforePause)
|
isRecording = toggleRecording(wasRecordingBeforePause)
|
||||||
if (wasRecordingBeforePause && !isRecording) {
|
if (wasRecordingBeforePause && !isRecording) {
|
||||||
return
|
return
|
||||||
@@ -423,7 +478,9 @@ class AudioEngine (context: Context) {
|
|||||||
Log.d("AudioEngine", "Playback paused")
|
Log.d("AudioEngine", "Playback paused")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@SuppressLint("NewApi")
|
||||||
fun resumePlayback() {
|
fun resumePlayback() {
|
||||||
|
acquireAudioSessionIfNeeded()
|
||||||
audioTrack.play()
|
audioTrack.play()
|
||||||
Log.d("AudioEngine", "Playback resumed")
|
Log.d("AudioEngine", "Playback resumed")
|
||||||
}
|
}
|
||||||
|
|||||||
+5
@@ -33,6 +33,11 @@ class ExpoTwoWayAudioModule : Module() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
Function("releaseAudioSession") {
|
||||||
|
audioEngine?.releaseAudioSession()
|
||||||
|
null
|
||||||
|
}
|
||||||
|
|
||||||
Function("isRecording") {
|
Function("isRecording") {
|
||||||
audioEngine?.isRecording ?: false
|
audioEngine?.isRecording ?: false
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ class AudioEngine {
|
|||||||
|
|
||||||
public private(set) var voiceIOFormat: AVAudioFormat
|
public private(set) var voiceIOFormat: AVAudioFormat
|
||||||
public private(set) var isRecording = false
|
public private(set) var isRecording = false
|
||||||
|
public private(set) var isSessionActive = false
|
||||||
|
|
||||||
public var onMicDataCallback: ((Data) -> Void)?
|
public var onMicDataCallback: ((Data) -> Void)?
|
||||||
public var onInputVolumeCallback: ((Float) -> Void)?
|
public var onInputVolumeCallback: ((Float) -> Void)?
|
||||||
@@ -28,6 +29,22 @@ class AudioEngine {
|
|||||||
private var discardRecording = false
|
private var discardRecording = false
|
||||||
private var discardFirstInputMillis = 2000
|
private var discardFirstInputMillis = 2000
|
||||||
|
|
||||||
|
/// Buffers scheduled on `speechPlayer` that have not finished rendering yet.
|
||||||
|
///
|
||||||
|
/// `AVAudioPlayerNode.isPlaying` cannot answer "is audio still in flight": it stays true from
|
||||||
|
/// `play()` until an explicit `stop()`/`pause()`, and draining every scheduled buffer does not
|
||||||
|
/// clear it. Using it as the release guard meant the session was never handed back after the
|
||||||
|
/// first spoken response. Decremented from the scheduling completion handler, which runs on an
|
||||||
|
/// AVAudioEngine-internal thread, so all access goes through `playbackCountLock`.
|
||||||
|
private var pendingPlaybackBuffers = 0
|
||||||
|
private let playbackCountLock = NSLock()
|
||||||
|
|
||||||
|
private var hasPendingPlayback: Bool {
|
||||||
|
playbackCountLock.lock()
|
||||||
|
defer { playbackCountLock.unlock() }
|
||||||
|
return pendingPlaybackBuffers > 0
|
||||||
|
}
|
||||||
|
|
||||||
enum AudioEngineError: Error {
|
enum AudioEngineError: Error {
|
||||||
case audioFormatError
|
case audioFormatError
|
||||||
}
|
}
|
||||||
@@ -94,11 +111,49 @@ class AudioEngine {
|
|||||||
|
|
||||||
do {
|
do {
|
||||||
try session.setActive(true)
|
try session.setActive(true)
|
||||||
|
isSessionActive = true
|
||||||
} catch {
|
} catch {
|
||||||
print("Could not set the audio session as active")
|
print("Could not set the audio session as active")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func activateAudioSessionIfNeeded() {
|
||||||
|
guard !isSessionActive else { return }
|
||||||
|
setupAudioSession()
|
||||||
|
checkEngineIsRunning()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Hand the audio session back to the system.
|
||||||
|
///
|
||||||
|
/// `.playAndRecord` + `.voiceChat` does not mix, so while it is active the user's
|
||||||
|
/// background music stays dead — including across backgrounding, because iOS
|
||||||
|
/// re-asserts whatever category the app last set when it returns to the foreground.
|
||||||
|
/// Releasing when we are neither capturing nor playing is what lets their music come
|
||||||
|
/// back; `.notifyOthersOnDeactivation` is what tells the other app to resume, and
|
||||||
|
/// dropping to `.ambient` means any implicit reactivation mixes instead of interrupting.
|
||||||
|
func releaseAudioSession() {
|
||||||
|
// The native engine is a singleton shared by every JS-side engine wrapper, so it is the
|
||||||
|
// only layer that knows whether *anything* is still using the session. Never release
|
||||||
|
// while capture or playback is live — the JS callers each only see their own queue.
|
||||||
|
guard isSessionActive, !isRecording, !hasPendingPlayback else { return }
|
||||||
|
|
||||||
|
avAudioEngine.pause()
|
||||||
|
|
||||||
|
let session = AVAudioSession.sharedInstance()
|
||||||
|
do {
|
||||||
|
try session.setActive(false, options: [.notifyOthersOnDeactivation])
|
||||||
|
} catch {
|
||||||
|
print("Could not deactivate the audio session: \(error.localizedDescription)")
|
||||||
|
}
|
||||||
|
do {
|
||||||
|
try session.setCategory(.ambient, options: [.mixWithOthers])
|
||||||
|
} catch {
|
||||||
|
print("Could not reset the audio category: \(error.localizedDescription)")
|
||||||
|
}
|
||||||
|
|
||||||
|
isSessionActive = false
|
||||||
|
}
|
||||||
|
|
||||||
func setup() {
|
func setup() {
|
||||||
let input = avAudioEngine.inputNode
|
let input = avAudioEngine.inputNode
|
||||||
do {
|
do {
|
||||||
@@ -183,6 +238,8 @@ class AudioEngine {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func playPCMData(_ pcmData: Data) {
|
func playPCMData(_ pcmData: Data) {
|
||||||
|
activateAudioSessionIfNeeded()
|
||||||
|
|
||||||
// Looks like we don't get a proper AEC for the very first chunks of audio that we play.
|
// Looks like we don't get a proper AEC for the very first chunks of audio that we play.
|
||||||
// To work around this, we will discard microphone input for the first few milliseconds.
|
// To work around this, we will discard microphone input for the first few milliseconds.
|
||||||
// This will give the AEC time to adapt to the playback audio.
|
// This will give the AEC time to adapt to the playback audio.
|
||||||
@@ -200,7 +257,17 @@ class AudioEngine {
|
|||||||
print("Failed to create audio buffer")
|
print("Failed to create audio buffer")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
speechPlayer.scheduleBuffer(buffer)
|
playbackCountLock.lock()
|
||||||
|
pendingPlaybackBuffers += 1
|
||||||
|
playbackCountLock.unlock()
|
||||||
|
speechPlayer.scheduleBuffer(buffer, completionCallbackType: .dataPlayedBack) { [weak self] _ in
|
||||||
|
guard let self = self else { return }
|
||||||
|
self.playbackCountLock.lock()
|
||||||
|
if self.pendingPlaybackBuffers > 0 {
|
||||||
|
self.pendingPlaybackBuffers -= 1
|
||||||
|
}
|
||||||
|
self.playbackCountLock.unlock()
|
||||||
|
}
|
||||||
|
|
||||||
if !speechPlayer.isPlaying {
|
if !speechPlayer.isPlaying {
|
||||||
speechPlayer.play()
|
speechPlayer.play()
|
||||||
@@ -247,6 +314,7 @@ class AudioEngine {
|
|||||||
inputBuffer = [Float](repeating: 0, count: 2048)
|
inputBuffer = [Float](repeating: 0, count: 2048)
|
||||||
updateInputVolume()
|
updateInputVolume()
|
||||||
} else {
|
} else {
|
||||||
|
activateAudioSessionIfNeeded()
|
||||||
avAudioEngine.inputNode.isVoiceProcessingInputMuted = false
|
avAudioEngine.inputNode.isVoiceProcessingInputMuted = false
|
||||||
}
|
}
|
||||||
print("Recording \(isRecording ? "started" : "stopped")")
|
print("Recording \(isRecording ? "started" : "stopped")")
|
||||||
@@ -255,22 +323,23 @@ class AudioEngine {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func stopRecordingAndPlayer(){
|
func stopRecordingAndPlayer(){
|
||||||
do {
|
|
||||||
try AVAudioSession.sharedInstance().setActive(false)
|
|
||||||
} catch {
|
|
||||||
print("Could not set the audio session to inactive: \(error)")
|
|
||||||
}
|
|
||||||
toggleRecording(false)
|
toggleRecording(false)
|
||||||
speechPlayer.stop()
|
speechPlayer.stop()
|
||||||
|
resetPendingPlayback()
|
||||||
updateOutputVolume()
|
updateOutputVolume()
|
||||||
|
releaseAudioSession()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `stop()` discards whatever is still scheduled, so the pending count has to be cleared with
|
||||||
|
/// it. Leaving it non-zero would block every later release.
|
||||||
|
private func resetPendingPlayback() {
|
||||||
|
playbackCountLock.lock()
|
||||||
|
pendingPlaybackBuffers = 0
|
||||||
|
playbackCountLock.unlock()
|
||||||
}
|
}
|
||||||
|
|
||||||
func resumeRecordingAndPlayer(){
|
func resumeRecordingAndPlayer(){
|
||||||
do {
|
activateAudioSessionIfNeeded()
|
||||||
try AVAudioSession.sharedInstance().setActive(true)
|
|
||||||
} catch {
|
|
||||||
print("Could not set the audio session to active: \(error)")
|
|
||||||
}
|
|
||||||
self.checkEngineIsRunning()
|
self.checkEngineIsRunning()
|
||||||
isRecording = toggleRecording(true)
|
isRecording = toggleRecording(true)
|
||||||
speechPlayer.play()
|
speechPlayer.play()
|
||||||
@@ -287,6 +356,7 @@ class AudioEngine {
|
|||||||
|
|
||||||
func stopPlayback() {
|
func stopPlayback() {
|
||||||
speechPlayer.stop()
|
speechPlayer.stop()
|
||||||
|
resetPendingPlayback()
|
||||||
// Clear any scheduled buffers
|
// Clear any scheduled buffers
|
||||||
outputBuffer = [Float](repeating: 0, count: 2048)
|
outputBuffer = [Float](repeating: 0, count: 2048)
|
||||||
updateOutputVolume()
|
updateOutputVolume()
|
||||||
@@ -306,6 +376,11 @@ class AudioEngine {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private func checkEngineIsRunning() {
|
private func checkEngineIsRunning() {
|
||||||
|
// Starting the engine implicitly reactivates the audio session. Never do that while the
|
||||||
|
// session is released, or a stray configuration-change notification (which releasing
|
||||||
|
// itself can trigger, since it changes the category) would silently grab the user's
|
||||||
|
// audio back and undo the release.
|
||||||
|
guard isSessionActive else { return }
|
||||||
if !avAudioEngine.isRunning {
|
if !avAudioEngine.isRunning {
|
||||||
start()
|
start()
|
||||||
}
|
}
|
||||||
@@ -340,6 +415,9 @@ class AudioEngine {
|
|||||||
private func handleMediaServicesWereReset() {
|
private func handleMediaServicesWereReset() {
|
||||||
self.avAudioEngine.stop()
|
self.avAudioEngine.stop()
|
||||||
self.setup()
|
self.setup()
|
||||||
|
// Only bring the engine back up if we still own the session; otherwise wait until the
|
||||||
|
// next capture/playback reactivates it.
|
||||||
|
guard isSessionActive else { return }
|
||||||
self.start()
|
self.start()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -39,6 +39,18 @@ public class ExpoTwoWayAudioModule: Module {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Safety net for the JS-driven release in `stopCapture`/`processQueue`: never leave
|
||||||
|
// the non-mixing `.playAndRecord` session active while backgrounded, or iOS
|
||||||
|
// re-asserts it on the next foreground and kills the user's music again.
|
||||||
|
OnAppEntersBackground {
|
||||||
|
guard self.audioEngine?.isRecording == false else { return }
|
||||||
|
self.audioEngine?.releaseAudioSession()
|
||||||
|
}
|
||||||
|
|
||||||
|
Function("releaseAudioSession") {
|
||||||
|
self.audioEngine?.releaseAudioSession()
|
||||||
|
}
|
||||||
|
|
||||||
Function("isRecording") { () -> Bool in
|
Function("isRecording") { () -> Bool in
|
||||||
guard let audioEngine = self.audioEngine else {
|
guard let audioEngine = self.audioEngine else {
|
||||||
print("AudioEngine not initialized")
|
print("AudioEngine not initialized")
|
||||||
|
|||||||
@@ -21,6 +21,20 @@ export function isRecording(): boolean {
|
|||||||
return ExpoTwoWayAudioModule.isRecording();
|
return ExpoTwoWayAudioModule.isRecording();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Hand the OS audio session / audio focus back once nothing is being captured or played,
|
||||||
|
* so other apps' audio can resume. Safe to call when already released.
|
||||||
|
*/
|
||||||
|
export function releaseAudioSession() {
|
||||||
|
// COMPAT(releaseAudioSession): added in v0.2.6. An OTA JS update can land on an older
|
||||||
|
// binary whose native module lacks this function, so probe the native object rather than
|
||||||
|
// this wrapper (which always exists). Drop the guard once the binary floor is >= v0.2.6.
|
||||||
|
if (typeof ExpoTwoWayAudioModule.releaseAudioSession !== "function") {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
return ExpoTwoWayAudioModule.releaseAudioSession();
|
||||||
|
}
|
||||||
|
|
||||||
export function tearDown() {
|
export function tearDown() {
|
||||||
return ExpoTwoWayAudioModule.tearDown();
|
return ExpoTwoWayAudioModule.tearDown();
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user