Release the audio session when voice capture and playback go idle (#2866)
* Release the audio session when voice capture and playback go idle Opening Paseo permanently paused music playing in other apps. Once the mic had been used a single time, every subsequent foreground of the app killed the user's audio again until the app was force-quit. iOS AudioEngine.setupAudioSession() takes .playAndRecord/.voiceChat — a non-mixing category — and calls setActive(true), but the only setActive(false) was reachable from tearDown(). Nothing handed the session back after a normal dictation turn, so iOS re-asserted the last category the app set on every foreground. The iOS module had no app-lifecycle handling at all, while Android already had a lifecycle listener; that asymmetry is what made the platforms diverge. Android had the same leak in a different shape: AUDIOFOCUS_GAIN_TRANSIENT_EXCLUSIVE was requested per recording and never abandoned. Add releaseAudioSession() to both platforms and call it once capture stops and the playback queue drains. iOS pauses the engine, calls setActive(false, .notifyOthersOnDeactivation) — this is what tells the other app to resume — then drops to .ambient/.mixWithOthers so any implicit reactivation mixes instead of interrupting. The session is re-acquired lazily on toggleRecording(true)/playPCMData, so voice keeps working; resumeRecordingAndPlayer() re-activates before restarting the engine, which keeps post-interruption resume intact. Releasing is guarded on isRecording/isPlaying, and checkEngineIsRunning() now refuses to start the engine while the session is released, since starting it would silently re-take the session. The JS wrapper probes the native module before calling, because a JS bundle shipped over EAS Update can land on a binary predating this function. * Correct the iOS signing diagnosis in mobile-testing docs The "No Accounts" error does not mean no Apple ID is configured. The account can be registered in Xcode while its Xcode-Token keychain item is missing, which Xcode reports as No Accounts and then compounds by falling back to a stale cached profile — surfacing as a certificate mismatch that sends you chasing the wrong thing. Document the DVTDeveloperAccountManager log line that names the real cause, the two commands that check what is actually configured, and the build-unsigned-then-sign-manually stopgap. * Reacquire Android audio focus when playback restarts releaseAudioSession() abandons the AUDIOFOCUS_GAIN_TRANSIENT_EXCLUSIVE request and pauses the AudioTrack once capture and playback go idle, but nothing took either back. The next playback turn therefore ran with no focus held, so the assistant's response competed with whatever the user was listening to instead of interrupting it. Add acquireAudioSessionIfNeeded(), the mirror of the iOS activateAudioSessionIfNeeded() already called from playPCMData, and use it from both playPCMData() and resumePlayback(). A null audioFocusRequest is what "we released" looks like, so it doubles as the guard against re-requesting focus mid-turn. Reported by Greptile on #2866. * Make Android playback focus reacquisition non-fatal The previous commit dropped PCM data when focus reacquisition did not return AUDIOFOCUS_REQUEST_GRANTED. That is worse than playing unfocused: playPCMData is fire-and-forget from JS, and playAudio() settles on a duration timer rather than a native ack, so the runtime acknowledged and discarded a chunk that was never queued. It also broke delayed focus. The request is built with setAcceptsDelayedFocusGain(true), so AUDIOFOCUS_REQUEST_DELAYED is a normal outcome, but routing it through handleAudioFocusBlocked() abandoned the pending request and the later grant could never arrive. Request focus, then play regardless of the result, and leave the request outstanding so a delayed grant still lands via the listener. Reported by Greptile on #2866. * Track scheduled buffers so iOS actually releases after playback releaseAudioSession() guarded on !speechPlayer.isPlaying, but AVAudioPlayerNode.isPlaying stays true from play() until an explicit stop()/pause() — draining every scheduled buffer does not clear it. The normal completion path is processQueue() -> releaseSessionIfIdle(), and stopPlayback() is only reached from the explicit stop() API, so after the first spoken response the guard blocked every release and the session stayed active. The music-pausing bug returned in full for anyone using voice responses. Count buffers that are scheduled but not yet rendered, decrementing from the .dataPlayedBack completion handler, and guard on that instead. The handler runs on an AVAudioEngine-internal thread, so the count is behind a lock. Both stop() paths reset it, since stop() discards what is scheduled and a stale count would block every later release. Found by Codex review on #2866. * Stop re-taking Android audio focus on every foreground ExpoTwoWayAudioLifeCycleListener calls resumeRecordingAndPlayer() from onResume, and that requested AUDIOFOCUS_GAIN_TRANSIENT_EXCLUSIVE before looking at isRecordingBeforePause. Foregrounding the app while nothing was recording therefore took exclusive focus back and re-paused the user's music — the same "every foreground kills my music" symptom this change set exists to remove, just reached through the Android lifecycle instead of the iOS category. Return early unless capture or playback was actually live when we backgrounded. Found by Codex review on #2866.
This commit is contained in:
Kevin Aspesi
@@ -10,6 +10,7 @@ class AudioEngine {
|
||||
|
||||
public private(set) var voiceIOFormat: AVAudioFormat
|
||||
public private(set) var isRecording = false
|
||||
public private(set) var isSessionActive = false
|
||||
|
||||
public var onMicDataCallback: ((Data) -> Void)?
|
||||
public var onInputVolumeCallback: ((Float) -> Void)?
|
||||
@@ -27,6 +28,22 @@ class AudioEngine {
|
||||
private var hasFirstInputBeenDiscarded = false
|
||||
private var discardRecording = false
|
||||
private var discardFirstInputMillis = 2000
|
||||
|
||||
/// Buffers scheduled on `speechPlayer` that have not finished rendering yet.
|
||||
///
|
||||
/// `AVAudioPlayerNode.isPlaying` cannot answer "is audio still in flight": it stays true from
|
||||
/// `play()` until an explicit `stop()`/`pause()`, and draining every scheduled buffer does not
|
||||
/// clear it. Using it as the release guard meant the session was never handed back after the
|
||||
/// first spoken response. Decremented from the scheduling completion handler, which runs on an
|
||||
/// AVAudioEngine-internal thread, so all access goes through `playbackCountLock`.
|
||||
private var pendingPlaybackBuffers = 0
|
||||
private let playbackCountLock = NSLock()
|
||||
|
||||
private var hasPendingPlayback: Bool {
|
||||
playbackCountLock.lock()
|
||||
defer { playbackCountLock.unlock() }
|
||||
return pendingPlaybackBuffers > 0
|
||||
}
|
||||
|
||||
enum AudioEngineError: Error {
|
||||
case audioFormatError
|
||||
@@ -79,26 +96,64 @@ class AudioEngine {
|
||||
|
||||
func setupAudioSession() {
|
||||
let session = AVAudioSession.sharedInstance()
|
||||
|
||||
|
||||
do {
|
||||
try session.setCategory(.playAndRecord, mode: .voiceChat, options: [.defaultToSpeaker, .allowBluetooth, .allowBluetoothA2DP])
|
||||
} catch {
|
||||
print("Could not set the audio category: \(error.localizedDescription)")
|
||||
}
|
||||
|
||||
|
||||
do {
|
||||
try session.setPreferredSampleRate(voiceIOFormat.sampleRate)
|
||||
} catch {
|
||||
print("Could not set the preferred sample rate: \(error.localizedDescription)")
|
||||
}
|
||||
|
||||
|
||||
do {
|
||||
try session.setActive(true)
|
||||
isSessionActive = true
|
||||
} catch {
|
||||
print("Could not set the audio session as active")
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
func activateAudioSessionIfNeeded() {
|
||||
guard !isSessionActive else { return }
|
||||
setupAudioSession()
|
||||
checkEngineIsRunning()
|
||||
}
|
||||
|
||||
/// Hand the audio session back to the system.
|
||||
///
|
||||
/// `.playAndRecord` + `.voiceChat` does not mix, so while it is active the user's
|
||||
/// background music stays dead — including across backgrounding, because iOS
|
||||
/// re-asserts whatever category the app last set when it returns to the foreground.
|
||||
/// Releasing when we are neither capturing nor playing is what lets their music come
|
||||
/// back; `.notifyOthersOnDeactivation` is what tells the other app to resume, and
|
||||
/// dropping to `.ambient` means any implicit reactivation mixes instead of interrupting.
|
||||
func releaseAudioSession() {
|
||||
// The native engine is a singleton shared by every JS-side engine wrapper, so it is the
|
||||
// only layer that knows whether *anything* is still using the session. Never release
|
||||
// while capture or playback is live — the JS callers each only see their own queue.
|
||||
guard isSessionActive, !isRecording, !hasPendingPlayback else { return }
|
||||
|
||||
avAudioEngine.pause()
|
||||
|
||||
let session = AVAudioSession.sharedInstance()
|
||||
do {
|
||||
try session.setActive(false, options: [.notifyOthersOnDeactivation])
|
||||
} catch {
|
||||
print("Could not deactivate the audio session: \(error.localizedDescription)")
|
||||
}
|
||||
do {
|
||||
try session.setCategory(.ambient, options: [.mixWithOthers])
|
||||
} catch {
|
||||
print("Could not reset the audio category: \(error.localizedDescription)")
|
||||
}
|
||||
|
||||
isSessionActive = false
|
||||
}
|
||||
|
||||
func setup() {
|
||||
let input = avAudioEngine.inputNode
|
||||
do {
|
||||
@@ -183,6 +238,8 @@ class AudioEngine {
|
||||
}
|
||||
|
||||
func playPCMData(_ pcmData: Data) {
|
||||
activateAudioSessionIfNeeded()
|
||||
|
||||
// Looks like we don't get a proper AEC for the very first chunks of audio that we play.
|
||||
// To work around this, we will discard microphone input for the first few milliseconds.
|
||||
// This will give the AEC time to adapt to the playback audio.
|
||||
@@ -200,8 +257,18 @@ class AudioEngine {
|
||||
print("Failed to create audio buffer")
|
||||
return
|
||||
}
|
||||
speechPlayer.scheduleBuffer(buffer)
|
||||
|
||||
playbackCountLock.lock()
|
||||
pendingPlaybackBuffers += 1
|
||||
playbackCountLock.unlock()
|
||||
speechPlayer.scheduleBuffer(buffer, completionCallbackType: .dataPlayedBack) { [weak self] _ in
|
||||
guard let self = self else { return }
|
||||
self.playbackCountLock.lock()
|
||||
if self.pendingPlaybackBuffers > 0 {
|
||||
self.pendingPlaybackBuffers -= 1
|
||||
}
|
||||
self.playbackCountLock.unlock()
|
||||
}
|
||||
|
||||
if !speechPlayer.isPlaying {
|
||||
speechPlayer.play()
|
||||
}
|
||||
@@ -247,6 +314,7 @@ class AudioEngine {
|
||||
inputBuffer = [Float](repeating: 0, count: 2048)
|
||||
updateInputVolume()
|
||||
} else {
|
||||
activateAudioSessionIfNeeded()
|
||||
avAudioEngine.inputNode.isVoiceProcessingInputMuted = false
|
||||
}
|
||||
print("Recording \(isRecording ? "started" : "stopped")")
|
||||
@@ -255,22 +323,23 @@ class AudioEngine {
|
||||
}
|
||||
|
||||
func stopRecordingAndPlayer(){
|
||||
do {
|
||||
try AVAudioSession.sharedInstance().setActive(false)
|
||||
} catch {
|
||||
print("Could not set the audio session to inactive: \(error)")
|
||||
}
|
||||
toggleRecording(false)
|
||||
speechPlayer.stop()
|
||||
resetPendingPlayback()
|
||||
updateOutputVolume()
|
||||
releaseAudioSession()
|
||||
}
|
||||
|
||||
|
||||
/// `stop()` discards whatever is still scheduled, so the pending count has to be cleared with
|
||||
/// it. Leaving it non-zero would block every later release.
|
||||
private func resetPendingPlayback() {
|
||||
playbackCountLock.lock()
|
||||
pendingPlaybackBuffers = 0
|
||||
playbackCountLock.unlock()
|
||||
}
|
||||
|
||||
func resumeRecordingAndPlayer(){
|
||||
do {
|
||||
try AVAudioSession.sharedInstance().setActive(true)
|
||||
} catch {
|
||||
print("Could not set the audio session to active: \(error)")
|
||||
}
|
||||
activateAudioSessionIfNeeded()
|
||||
self.checkEngineIsRunning()
|
||||
isRecording = toggleRecording(true)
|
||||
speechPlayer.play()
|
||||
@@ -287,6 +356,7 @@ class AudioEngine {
|
||||
|
||||
func stopPlayback() {
|
||||
speechPlayer.stop()
|
||||
resetPendingPlayback()
|
||||
// Clear any scheduled buffers
|
||||
outputBuffer = [Float](repeating: 0, count: 2048)
|
||||
updateOutputVolume()
|
||||
@@ -306,6 +376,11 @@ class AudioEngine {
|
||||
}
|
||||
|
||||
private func checkEngineIsRunning() {
|
||||
// Starting the engine implicitly reactivates the audio session. Never do that while the
|
||||
// session is released, or a stray configuration-change notification (which releasing
|
||||
// itself can trigger, since it changes the category) would silently grab the user's
|
||||
// audio back and undo the release.
|
||||
guard isSessionActive else { return }
|
||||
if !avAudioEngine.isRunning {
|
||||
start()
|
||||
}
|
||||
@@ -340,6 +415,9 @@ class AudioEngine {
|
||||
private func handleMediaServicesWereReset() {
|
||||
self.avAudioEngine.stop()
|
||||
self.setup()
|
||||
// Only bring the engine back up if we still own the session; otherwise wait until the
|
||||
// next capture/playback reactivates it.
|
||||
guard isSessionActive else { return }
|
||||
self.start()
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user