Prompt is too long

This commit is contained in:
Mohamed Boudra
2025-10-22 09:06:18 +02:00
parent 8235d8d1b1
commit e646f410be
128 changed files with 9346 additions and 7090 deletions
BIN
View File
Binary file not shown.
-391
View File
@@ -1,391 +0,0 @@
# Voice Assistant Mobile App - Technical Specification
## Project Overview
React Native mobile app that replicates the existing web-based voice assistant. Same functionality, same server connection, just on mobile with in-call experience for hands-free use.
## Core Architecture
### What We Have (Web Prototype)
```
Browser (React + Vite)
↓ WebSocket
Express Server on Laptop
├── OpenAI Whisper (STT)
├── OpenAI GPT-4 (LLM streaming)
├── OpenAI TTS
└── tmux terminal manager
```
### What We're Building (Mobile App)
```
React Native App (Expo)
↓ WebSocket (same protocol)
Express Server on Laptop (unchanged)
├── OpenAI Whisper (STT)
├── OpenAI GPT-4 (LLM streaming)
├── OpenAI TTS
└── tmux terminal manager
```
**Connection:** Direct WebSocket to laptop server (Tailscale tunnel)
**Server changes:** ZERO. The existing server works as-is.
## Monorepo Structure
```
voice-dev/
├── packages/
│ ├── voice-assistant/ # Existing web app (stays as-is)
│ │ └── src/
│ │ └── server/
│ │ ├── messages.ts # WebSocket message types (SHARED)
│ │ └── types.ts # Server types (SHARED)
│ └── voice-mobile/ # NEW - React Native app
│ ├── app/ # Expo Router screens
│ ├── components/ # RN components (NOT shared with web)
│ ├── hooks/ # WebSocket, audio hooks
│ ├── lib/ # Audio capture/playback
│ ├── app.json
│ ├── package.json
│ └── tsconfig.json
└── package.json # Root workspace config
```
**Code Sharing:**
- ✅ Share Zod schemas and types from `voice-assistant/src/server/` (WebSocket message types)
- ❌ Don't share UI components, hooks, or other code between web and mobile
## Technology Stack
- **React Native** with **Expo SDK 54+**
- **TypeScript** (strict mode)
- **Expo Router** (file-based routing)
- **NativeWind v5** (Tailwind CSS for React Native)
- **expo-audio** for audio capture and playback
- **react-native-incall-manager** for Android in-call experience
- **WebSocket** (native WebSocket API, no socket.io needed)
## What the App Does
**Exactly what the web app does:**
1. Connect to laptop server via WebSocket
2. Record audio from microphone
3. Stream audio chunks to server
4. Display transcription results
5. Show streaming AI responses
6. Play TTS audio responses
7. Display activity logs (tool calls, terminal output)
**Additional mobile features:**
- Android sticky notification (like being in a phone call)
- Background audio (works when screen locked)
- Bluetooth/headphone support
- Earpiece/speaker routing
## User Interface
### Styling: NativeWind v5
All styling uses Tailwind utility classes:
```tsx
<View className="flex-1 bg-black">
<Text className="text-white text-lg font-semibold">
Voice Assistant
</Text>
<Pressable className="bg-red-500 rounded-full p-4">
<Text className="text-white">Record</Text>
</Pressable>
</View>
```
**Color Scheme (Dark Mode Primary):**
```
- bg-black - Main background
- bg-zinc-900 - Surface/cards
- text-white - Primary text
- text-zinc-400 - Secondary text
- bg-red-500 - Recording indicator
- bg-blue-500 - Processing indicator
- bg-green-500 - Speaking indicator
```
### Screens
**1. Main Screen**
The conversation interface (matches web app layout).
```
┌─────────────────────────────┐
│ ● Connected │ ← Status bar
├─────────────────────────────┤
│ │
│ You: Run git status │ ← User message (right aligned)
│ │
│ Assistant: Running git... │ ← AI response (left aligned)
│ [Activity: git status] │ ← Expandable activity log
│ │
│ You: What changed? │
│ │
│ Assistant: You modified... │
│ │
│ [Scrollable conversation] │
│ │
├─────────────────────────────┤
│ 🎤 │ ← Voice button (large, centered)
└─────────────────────────────┘
```
**Voice Button States:**
- Idle: Gray circle
- Recording: Red pulsing circle
- Transcribing: Blue spinner
- AI Speaking: Green animated waveform
**Conversation Items:**
- User messages (right aligned, dark background)
- AI responses (left aligned, lighter background)
- Activity logs (collapsed by default, tap to expand)
- Timestamps (subtle, relative time)
**2. Settings Screen**
```
┌─────────────────────────────┐
│ Settings │
├─────────────────────────────┤
│ │
│ Server Configuration │
│ ┌─────────────────────────┐ │
│ │ ws://100.x.x.x:3000 │ │ ← Tailscale URL
│ └─────────────────────────┘ │
│ │
│ Audio │
│ □ Use Speaker │ ← Toggle earpiece/speaker
│ □ Keep Screen On │
│ │
│ Theme │
│ ○ Dark ○ Light ○ Auto │
│ │
└─────────────────────────────┘
```
## Audio System
### Android In-Call Experience
**What it looks like:**
- Persistent notification: "Voice Assistant • In Call"
- Notification shows hang-up button
- Phone treats it like an active call
- Audio routes properly (earpiece/speaker/bluetooth)
- Works when screen is locked
- Works when app is backgrounded
**Implementation:**
```typescript
import InCallManager from 'react-native-incall-manager';
// Start voice session
InCallManager.start({ media: 'audio' });
InCallManager.setKeepScreenOn(true);
InCallManager.setSpeakerphoneOn(false); // Use earpiece
// End voice session
InCallManager.stop();
```
### Audio Capture
**Same configuration as web:**
```typescript
{
sampleRate: 16000,
numberOfChannels: 1,
echoCancellation: true,
noiseSuppression: true,
autoGainControl: true,
}
```
**Streaming:**
- Capture in 100ms chunks (same as web)
- Send via WebSocket as base64
- Same message format as web app uses
### Audio Playback
- Receive MP3 audio from server (base64)
- Decode and play via expo-audio
- Same as web app's audio playback
## WebSocket Protocol
**Uses the existing protocol - no changes.**
The mobile app imports message types directly from the server:
```typescript
import type {
WSInboundMessage,
WSOutboundMessage,
SessionOutboundMessage
} from '../../voice-assistant/src/server/messages';
```
**Client → Server:**
- `audio_chunk` - Audio data
- `user_message` - Text input
- `recording_state` - Started/stopped
- `ping` - Keepalive
**Server → Client:**
- `activity_log` - Tool calls, events
- `assistant_chunk` - Streaming response
- `audio_output` - TTS audio
- `transcription_result` - STT output
- `status` - Connection state
- `pong` - Keepalive response
**Connection:**
```typescript
const ws = new WebSocket('ws://100.x.x.x:3000'); // Tailscale IP
ws.onmessage = (event) => {
const message: WSOutboundMessage = JSON.parse(event.data);
// Type-safe message handling with shared types
};
```
## Data Flow
### Voice Input (Same as Web)
```
1. User taps mic button
2. Start recording with expo-audio
3. Capture 100ms chunks
4. Send chunks via WebSocket
5. Server transcribes with Whisper
6. Display transcription in real-time
```
### AI Response (Same as Web)
```
1. Server streams GPT-4 response chunks
2. Display text word-by-word
3. Server generates TTS audio
4. Send audio chunks to mobile
5. Play audio via expo-audio
```
### Activity Logs (Same as Web)
```
1. Claude executes tool (e.g., "create file")
2. Server sends activity_log message
3. Display in conversation with collapse/expand
```
## Android Features
### Required Permissions
```xml
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<uses-permission android:name="android.permission.INTERNET" />
```
### Foreground Service
When voice session is active:
- App runs as foreground service
- Persistent notification visible
- Android won't kill the app
- Audio works in background
- Works when screen is locked
## State Management
**Simple, no libraries:**
- React Context for WebSocket connection
- React Context for conversation state
- Local state for UI (button press, etc.)
- AsyncStorage for settings (Tailscale URL)
```typescript
interface AppState {
isConnected: boolean;
isRecording: boolean;
messages: Message[];
activityLogs: ActivityLog[];
serverUrl: string;
}
```
## What We're NOT Building
**Keep it simple. These are out of scope:**
- ❌ Code diff viewer (just show text)
- ❌ Terminal emulator (just show logs)
- ❌ File browser
- ❌ Push notifications
- ❌ Conversation history sync
- ❌ Multi-device support
- ❌ Authentication (Tailscale handles it)
- ❌ Encryption layer (Tailscale handles it)
- ❌ Relay server (direct connection only)
## Success Criteria
**It works if:**
- ✅ Connects to laptop server (same as web app)
- ✅ Audio quality matches web app
- ✅ Shows conversation same as web app
- ✅ Works with phone locked (Android)
- ✅ Shows sticky notification (Android)
- ✅ Routes audio to bluetooth/headphones
- ✅ Latency same as web app (< 3 seconds)
- ✅ Can use hands-free in car
**Visual/UX:**
- ✅ Dark mode looks good
- ✅ Voice button is obvious and large
- ✅ Conversation scrolls smoothly
- ✅ Works one-handed
## Development Approach
**Start simple:**
1. Set up Expo + NativeWind
2. Port WebSocket connection (copy from web)
3. Port audio capture (expo-audio instead of MediaRecorder)
4. Port audio playback (expo-audio instead of Web Audio)
5. Port UI (React Native + Tailwind instead of React + CSS)
6. Add InCallManager for Android experience
**No fancy stuff:**
- No state management libraries
- No complex routing (just 2 screens)
- No animations (except button states)
- No offline support
- No caching
- Keep it simple
## Personal Use Optimizations
**Built for one user (you):**
- Tailscale URL can be hardcoded or stored locally
- No user accounts
- No analytics
- No crash reporting (just use it and fix bugs)
- No app store deployment (sideload APK)
- No iOS initially (Android only)
-5
View File
@@ -1,5 +0,0 @@
{
"android": {
"package": "com.moboudra.voicedev"
}
}
+9037 -1386
View File
File diff suppressed because it is too large Load Diff
+8 -12
View File
@@ -3,22 +3,18 @@
"version": "1.0.0",
"private": true,
"workspaces": [
"packages/voice-assistant"
"packages/server",
"packages/app"
],
"scripts": {
"dev": "npm run dev --workspace=@voice-dev/web",
"dev:agent": "npm run dev --workspace=@voice-dev/agent",
"dev:mcp": "npm run dev --workspace=@voice-dev/mcp-server",
"dev": "npm run dev --workspace=@voice-dev/server",
"dev:app": "npm run start --workspace=@voice-dev/app",
"build": "npm run build --workspaces --if-present",
"build:agent": "npm run build --workspace=@voice-dev/agent",
"build:mcp": "npm run build --workspace=@voice-dev/mcp-server",
"build:web": "npm run build --workspace=@voice-dev/web",
"typecheck": "npm run typecheck --workspaces --if-present",
"start": "npm run start --workspace=@voice-dev/web",
"start:agent": "npm run start --workspace=@voice-dev/agent",
"start:mcp": "npm run start --workspace=@voice-dev/mcp-server",
"android": "expo run:android",
"ios": "expo run:ios"
"start": "npm run start --workspace=@voice-dev/server",
"android": "npm run android --workspace=@voice-dev/app",
"ios": "npm run ios --workspace=@voice-dev/app",
"web": "npm run web --workspace=@voice-dev/app"
},
"devDependencies": {
"typescript": "^5.9.3"
@@ -1,11 +1,11 @@
{
"expo": {
"name": "voice-mobile",
"slug": "voice-mobile",
"name": "Voice Dev",
"slug": "voice-dev",
"version": "1.0.0",
"orientation": "portrait",
"icon": "./assets/images/icon.png",
"scheme": "voicemobile",
"scheme": "voicedev",
"userInterfaceStyle": "automatic",
"newArchEnabled": true,
"runtimeVersion": {
@@ -19,7 +19,7 @@
"infoPlist": {
"NSMicrophoneUsageDescription": "This app needs access to the microphone for voice commands."
},
"bundleIdentifier": "com.moboudra.voicemobile"
"bundleIdentifier": "com.moboudra.voicedev"
},
"android": {
"adaptiveIcon": {
@@ -36,7 +36,7 @@
"android.permission.RECORD_AUDIO",
"android.permission.MODIFY_AUDIO_SETTINGS"
],
"package": "com.moboudra.voicemobile"
"package": "com.moboudra.voicedev"
},
"web": {
"output": "static",
@@ -61,7 +61,7 @@
"expo-build-properties",
{
"android": {
"minSdkVersion": 26
"minSdkVersion": 29
}
}
]
@@ -25,8 +25,8 @@ import type {
ActivityLogPayload,
SessionInboundMessage,
WSInboundMessage,
} from '@voice-assistant/server/messages';
import type { AgentStatus } from '@voice-assistant/server/acp/types';
} from '@server/server/messages';
import type { AgentStatus } from '@server/server/acp/types';
import type { SessionNotification } from '@agentclientprotocol/sdk';
type MessageEntry =
@@ -95,29 +95,26 @@ export default function VoiceAssistantScreen() {
// Realtime mode state (defined early so we can use it in audioRecorder)
const [isRealtimeMode, setIsRealtimeMode] = useState(false);
const [isVADActive, setIsVADActive] = useState(false);
const pulseAnim = useRef(new Animated.Value(1)).current;
const audioRecorder = useAudioRecorder();
const audioPlayer = useAudioPlayer({ useSpeaker: settings.useSpeaker });
const insets = useSafeAreaInsets();
// Realtime audio with VAD
// Realtime audio with WebRTC
const realtimeAudio = useRealtimeAudio({
onSpeechStart: () => {
console.log('[App] Realtime speech started');
setIsVADActive(true);
console.log('[App] Speech detected');
// Pause audio playback if playing
if (isPlayingAudio) {
audioPlayer.pause();
}
},
onSpeechEnd: () => {
console.log('[App] Realtime speech ended');
setIsVADActive(false);
console.log('[App] Speech ended');
},
onAudioSegment: async (audioData: string, format: string) => {
console.log('[App] Received audio segment, length:', audioData.length);
onAudioSegment: (base64Audio: string) => {
console.log('[App] Sending audio segment, length:', base64Audio.length);
// Send audio segment to server
try {
@@ -125,12 +122,11 @@ export default function VoiceAssistantScreen() {
type: 'session',
message: {
type: 'audio_chunk',
audio: audioData,
format: format,
isLast: true,
audio: base64Audio,
format: 'audio/pcm',
isLast: true, // Complete segment
},
});
console.log('[App] Sent audio segment to server');
} catch (error) {
console.error('[App] Failed to send audio segment:', error);
}
@@ -148,7 +144,7 @@ export default function VoiceAssistantScreen() {
},
]);
},
speechThreshold: 0.5,
volumeThreshold: 0.02,
silenceDuration: 1000,
});
@@ -174,9 +170,9 @@ export default function VoiceAssistantScreen() {
const scrollViewRef = useRef<ScrollView>(null);
// Pulse animation for VAD indicator
// Pulse animation for speech indicator
useEffect(() => {
if (isVADActive) {
if (realtimeAudio.isSpeaking) {
Animated.loop(
Animated.sequence([
Animated.timing(pulseAnim, {
@@ -195,7 +191,7 @@ export default function VoiceAssistantScreen() {
pulseAnim.stopAnimation();
pulseAnim.setValue(1);
}
}, [isVADActive, pulseAnim]);
}, [realtimeAudio.isSpeaking, pulseAnim]);
// Keep screen awake if setting is enabled (mobile only)
useEffect(() => {
@@ -830,7 +826,6 @@ export default function VoiceAssistantScreen() {
try {
await realtimeAudio.stop();
setIsRealtimeMode(false);
setIsVADActive(false);
console.log('[App] Realtime mode disabled');
setMessages((prev) => [

Before

Width:  |  Height:  |  Size: 17 KiB

After

Width:  |  Height:  |  Size: 17 KiB

Before

Width:  |  Height:  |  Size: 77 KiB

After

Width:  |  Height:  |  Size: 77 KiB

Before

Width:  |  Height:  |  Size: 4.0 KiB

After

Width:  |  Height:  |  Size: 4.0 KiB

Before

Width:  |  Height:  |  Size: 1.1 KiB

After

Width:  |  Height:  |  Size: 1.1 KiB

Before

Width:  |  Height:  |  Size: 384 KiB

After

Width:  |  Height:  |  Size: 384 KiB

Before

Width:  |  Height:  |  Size: 5.0 KiB

After

Width:  |  Height:  |  Size: 5.0 KiB

Before

Width:  |  Height:  |  Size: 6.2 KiB

After

Width:  |  Height:  |  Size: 6.2 KiB

Before

Width:  |  Height:  |  Size: 14 KiB

After

Width:  |  Height:  |  Size: 14 KiB

Before

Width:  |  Height:  |  Size: 21 KiB

After

Width:  |  Height:  |  Size: 21 KiB

Before

Width:  |  Height:  |  Size: 17 KiB

After

Width:  |  Height:  |  Size: 17 KiB

@@ -1,5 +1,5 @@
import { View, Text, Pressable, ScrollView } from 'react-native';
import type { AgentStatus } from '@voice-assistant/server/acp/types';
import type { AgentStatus } from '@server/server/acp/types';
export interface ActiveProcessesProps {
agents: Array<{
@@ -1,7 +1,7 @@
import { useState, useEffect, useRef } from 'react';
import { View, Text, ScrollView, Pressable } from 'react-native';
import { useSafeAreaInsets } from 'react-native-safe-area-context';
import type { AgentStatus } from '@voice-assistant/server/acp/types';
import type { AgentStatus } from '@server/server/acp/types';
import type { SessionNotification } from '@agentclientprotocol/sdk';
export interface AgentStreamViewProps {
@@ -5,7 +5,7 @@ import { MessageSquare, X, Plus, Trash2 } from 'lucide-react-native';
import AsyncStorage from '@react-native-async-storage/async-storage';
import type { UseWebSocketReturn } from '../hooks/use-websocket';
const STORAGE_KEY = '@voice-assistant:conversation-id';
const STORAGE_KEY = '@voice-dev:conversation-id';
interface Conversation {
id: string;
+210
View File
@@ -0,0 +1,210 @@
import { useState, useEffect, useRef } from 'react';
import { Platform } from 'react-native';
import { useAudioRecorder } from '@siteed/expo-audio-studio';
import { check, request, PERMISSIONS, RESULTS } from 'react-native-permissions';
export interface RealtimeAudioConfig {
onAudioSegment?: (audioData: string) => void;
onSpeechStart?: () => void;
onSpeechEnd?: () => void;
onError?: (error: Error) => void;
volumeThreshold?: number; // RMS threshold for speech detection (0-1)
silenceDuration?: number; // ms of silence before ending segment
}
export interface RealtimeAudio {
start: () => Promise<void>;
stop: () => Promise<void>;
isActive: boolean;
isSpeaking: boolean;
}
/**
* Hook for realtime audio streaming with volume-based VAD
* Uses expo-audio-studio for recording with echo cancellation and noise suppression
*/
export function useRealtimeAudio(config: RealtimeAudioConfig): RealtimeAudio {
const [isActive, setIsActive] = useState(false);
const [isSpeaking, setIsSpeaking] = useState(false);
const audioRecorder = useAudioRecorder();
const audioBufferRef = useRef<string[]>([]);
const silenceStartRef = useRef<number | null>(null);
const isSpeakingRef = useRef(false);
const lastVolumeCheckRef = useRef<number>(Date.now());
const VOLUME_THRESHOLD = config.volumeThreshold ?? 0.02;
const SILENCE_DURATION_MS = config.silenceDuration ?? 1000;
const VOLUME_CHECK_INTERVAL = 200; // Check volume every 200ms
async function requestMicrophonePermission(): Promise<boolean> {
try {
const permission = Platform.OS === 'ios'
? PERMISSIONS.IOS.MICROPHONE
: PERMISSIONS.ANDROID.RECORD_AUDIO;
const status = await check(permission);
if (status === RESULTS.GRANTED) {
return true;
}
const result = await request(permission);
return result === RESULTS.GRANTED;
} catch (error) {
console.error('[RealtimeAudio] Permission error:', error);
return false;
}
}
function calculateVolume(base64Audio: string): number {
try {
// Decode base64 to binary
const binary = atob(base64Audio);
const bytes = new Uint8Array(binary.length);
for (let i = 0; i < binary.length; i++) {
bytes[i] = binary.charCodeAt(i);
}
// Calculate RMS from PCM data (assuming 16-bit PCM)
let sum = 0;
for (let i = 0; i < bytes.length - 1; i += 2) {
const sample = (bytes[i] | (bytes[i + 1] << 8)) / 32768.0; // Convert to -1 to 1
sum += sample * sample;
}
const rms = Math.sqrt(sum / (bytes.length / 2));
return rms;
} catch (error) {
console.error('[RealtimeAudio] Volume calculation error:', error);
return 0;
}
}
async function start(): Promise<void> {
if (isActive) {
console.log('[RealtimeAudio] Already active');
return;
}
try {
console.log('[RealtimeAudio] Starting realtime audio...');
// Request permissions
const hasPermission = await requestMicrophonePermission();
if (!hasPermission) {
throw new Error('Microphone permission denied');
}
// Start recording with audio streaming
await audioRecorder.startRecording({
sampleRate: 16000,
channels: 1,
encoding: 'pcm_16bit',
interval: 100, // Get chunks every 100ms
onAudioStream: async (event: any) => {
if (!event.data) return;
const audioData = event.data; // base64 PCM data
// Check volume periodically
const now = Date.now();
if (now - lastVolumeCheckRef.current >= VOLUME_CHECK_INTERVAL) {
lastVolumeCheckRef.current = now;
const volume = calculateVolume(audioData);
const speechDetected = volume > VOLUME_THRESHOLD;
if (speechDetected && !isSpeakingRef.current) {
// Speech START
console.log('[RealtimeAudio] Speech started (volume:', volume.toFixed(4), ')');
isSpeakingRef.current = true;
silenceStartRef.current = null;
audioBufferRef.current = [audioData];
setIsSpeaking(true);
config.onSpeechStart?.();
} else if (speechDetected && isSpeakingRef.current) {
// Continuing speech
audioBufferRef.current.push(audioData);
silenceStartRef.current = null;
} else if (!speechDetected && isSpeakingRef.current) {
// Potential speech END
audioBufferRef.current.push(audioData);
if (silenceStartRef.current === null) {
silenceStartRef.current = Date.now();
} else {
const silenceDuration = Date.now() - silenceStartRef.current;
if (silenceDuration >= SILENCE_DURATION_MS) {
// Speech END confirmed
console.log('[RealtimeAudio] Speech ended after', silenceDuration, 'ms silence');
isSpeakingRef.current = false;
silenceStartRef.current = null;
setIsSpeaking(false);
config.onSpeechEnd?.();
// Send buffered audio segment
if (audioBufferRef.current.length > 0) {
const combinedAudio = audioBufferRef.current.join('');
config.onAudioSegment?.(combinedAudio);
audioBufferRef.current = [];
}
}
}
}
} else if (isSpeakingRef.current) {
// Just buffer if we're speaking but not checking volume yet
audioBufferRef.current.push(audioData);
}
},
});
setIsActive(true);
console.log('[RealtimeAudio] Realtime audio started successfully');
} catch (error) {
console.error('[RealtimeAudio] Start error:', error);
const err = error instanceof Error ? error : new Error(String(error));
config.onError?.(err);
// Cleanup on error
await stop();
throw error;
}
}
async function stop(): Promise<void> {
console.log('[RealtimeAudio] Stopping realtime audio...');
// Stop recording
if (audioRecorder.isRecording) {
try {
await audioRecorder.stopRecording();
} catch (error) {
console.error('[RealtimeAudio] Error stopping recording:', error);
}
}
// Reset state
audioBufferRef.current = [];
isSpeakingRef.current = false;
silenceStartRef.current = null;
setIsActive(false);
setIsSpeaking(false);
console.log('[RealtimeAudio] Realtime audio stopped');
}
// Cleanup on unmount
useEffect(() => {
return () => {
if (isActive) {
stop();
}
};
}, []);
return {
start,
stop,
isActive,
isSpeaking,
};
}
@@ -1,7 +1,7 @@
import { useState, useEffect, useCallback } from 'react';
import AsyncStorage from '@react-native-async-storage/async-storage';
const STORAGE_KEY = '@voice-assistant:settings';
const STORAGE_KEY = '@voice-dev:settings';
export interface Settings {
serverUrl: string;
@@ -3,7 +3,7 @@ import type {
WSInboundMessage,
WSOutboundMessage,
SessionOutboundMessage
} from '@voice-assistant/server/messages';
} from '@server/server/messages';
export interface UseWebSocketReturn {
isConnected: boolean;
@@ -44,8 +44,8 @@
"react-native-reanimated": "~4.1.1",
"react-native-safe-area-context": "~5.6.0",
"react-native-screens": "~4.16.0",
"react-native-vad": "^1.0.44",
"react-native-web": "~0.21.0",
"react-native-webrtc": "^124.0.7",
"react-native-worklets": "0.5.1"
},
"devDependencies": {
@@ -15258,16 +15258,6 @@
"react-native": "*"
}
},
"node_modules/react-native-vad": {
"version": "1.0.44",
"resolved": "https://registry.npmjs.org/react-native-vad/-/react-native-vad-1.0.44.tgz",
"integrity": "sha512-n8Ufijqp63EIpfwjiVhaYEGdeFYN0scHvGGe1AgeTYg6ivUam3N5HlbQg1Wwj4OX5BjXkZT558NFckhHMddFzg==",
"license": "MIT",
"peerDependencies": {
"react": "^18.3.1",
"react-native": ">=0.70.1"
}
},
"node_modules/react-native-web": {
"version": "0.21.2",
"resolved": "https://registry.npmjs.org/react-native-web/-/react-native-web-0.21.2.tgz",
@@ -15300,6 +15290,55 @@
"integrity": "sha512-rkpe71W0N0c0Xz6QD0eJETuWAJGnJ9afsl1srmwPrI+yBCkge5EycXXbYRyvL29zZVUWQCY7InPRCv3GDXuZNw==",
"license": "MIT"
},
"node_modules/react-native-webrtc": {
"version": "124.0.7",
"resolved": "https://registry.npmjs.org/react-native-webrtc/-/react-native-webrtc-124.0.7.tgz",
"integrity": "sha512-gnXPdbUS8IkKHq9WNaWptW/yy5s6nMyI6cNn90LXdobPVCgYSk6NA2uUGdT4c4J14BRgaFA95F+cR28tUPkMVA==",
"license": "MIT",
"dependencies": {
"base64-js": "1.5.1",
"debug": "4.3.4",
"event-target-shim": "6.0.2"
},
"peerDependencies": {
"react-native": ">=0.60.0"
}
},
"node_modules/react-native-webrtc/node_modules/debug": {
"version": "4.3.4",
"resolved": "https://registry.npmjs.org/debug/-/debug-4.3.4.tgz",
"integrity": "sha512-PRWFHuSU3eDtQJPvnNY7Jcket1j0t5OuOsFzPPzsekD52Zl8qUfFIPEiswXqIvHWGVHOgX+7G/vCNNhehwxfkQ==",
"license": "MIT",
"dependencies": {
"ms": "2.1.2"
},
"engines": {
"node": ">=6.0"
},
"peerDependenciesMeta": {
"supports-color": {
"optional": true
}
}
},
"node_modules/react-native-webrtc/node_modules/event-target-shim": {
"version": "6.0.2",
"resolved": "https://registry.npmjs.org/event-target-shim/-/event-target-shim-6.0.2.tgz",
"integrity": "sha512-8q3LsZjRezbFZ2PN+uP+Q7pnHUMmAOziU2vA2OwoFaKIXxlxl38IylhSSgUorWu/rf4er67w0ikBqjBFk/pomA==",
"license": "MIT",
"engines": {
"node": ">=10.13.0"
},
"funding": {
"url": "https://github.com/sponsors/mysticatea"
}
},
"node_modules/react-native-webrtc/node_modules/ms": {
"version": "2.1.2",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.1.2.tgz",
"integrity": "sha512-sGkPx+VjMtmA6MX27oA4FBFELFCZZ4S4XqeGOXCv68tT+jb3vk/RyaKWP0PTKyWtmLSM0b+adUTEvbs1PEaH2w==",
"license": "MIT"
},
"node_modules/react-native-worklets": {
"version": "0.5.1",
"resolved": "https://registry.npmjs.org/react-native-worklets/-/react-native-worklets-0.5.1.tgz",
@@ -1,5 +1,5 @@
{
"name": "voice-mobile",
"name": "@voice-dev/app",
"main": "expo-router/entry",
"version": "1.0.0",
"scripts": {
@@ -47,8 +47,8 @@
"react-native-reanimated": "~4.1.1",
"react-native-safe-area-context": "~5.6.0",
"react-native-screens": "~4.16.0",
"react-native-vad": "^1.0.44",
"react-native-web": "~0.21.0",
"react-native-webrtc": "^124.0.7",
"react-native-worklets": "0.5.1"
},
"devDependencies": {
@@ -63,9 +63,6 @@
},
"overrides": {
"lightningcss": "1.30.1",
"react-native-vad": {
"react": "19.1.0"
},
"@siteed/expo-audio-studio": {
"expo-modules-core": "3.0.22"
}
@@ -6,8 +6,8 @@
"@/*": [
"./*"
],
"@voice-assistant/*": [
"../voice-assistant/src/*"
"@server/*": [
"../server/src/*"
]
}
},
@@ -1,15 +1,13 @@
{
"name": "@voice-dev/voice-assistant",
"name": "@voice-dev/server",
"version": "0.1.0",
"description": "Voice-controlled terminal assistant - single service implementation",
"description": "Voice assistant backend server",
"type": "module",
"scripts": {
"dev": "NODE_ENV=development tsx watch --ignore '**/*.timestamp-*' src/server/index.ts",
"build:ui": "vite build",
"build:server": "tsc -p tsconfig.server.json",
"build": "npm run build:ui && npm run build:server",
"build": "tsc -p tsconfig.server.json",
"start": "NODE_ENV=production node dist/server/index.js",
"typecheck": "tsc --noEmit && tsc -p tsconfig.server.json --noEmit",
"typecheck": "tsc -p tsconfig.server.json --noEmit",
"test": "vitest run",
"test:watch": "vitest",
"test:ui": "vitest --ui",
@@ -23,15 +21,12 @@
"@deepgram/sdk": "^3.4.0",
"@modelcontextprotocol/sdk": "^1.20.1",
"@openrouter/ai-sdk-provider": "^1.2.0",
"@ricky0123/vad-web": "^0.0.28",
"ai": "^5.0.76",
"dotenv": "^17.2.3",
"express": "^4.18.2",
"express-basic-auth": "^1.2.1",
"lucide-react": "^0.546.0",
"openai": "^4.20.0",
"playwright": "^1.56.1",
"react-markdown": "^10.1.0",
"tiny-invariant": "^1.3.3",
"uuid": "^9.0.1",
"ws": "^8.14.2",
@@ -41,19 +36,11 @@
"@playwright/test": "^1.56.1",
"@types/express": "^4.17.20",
"@types/node": "^20.9.0",
"@types/react": "^19.1.0",
"@types/react-dom": "^19.1.0",
"@types/uuid": "^9.0.7",
"@types/ws": "^8.5.8",
"@vitejs/plugin-react": "^4.2.0",
"@vitest/ui": "^3.2.4",
"concurrently": "^8.2.2",
"react": "^19.1.0",
"react-dom": "^19.1.0",
"tsx": "^4.6.0",
"typescript": "^5.2.2",
"vite": "^7.1.10",
"vite-plugin-static-copy": "^3.1.4",
"vitest": "^3.2.4"
}
}
@@ -1,13 +1,7 @@
import "dotenv/config";
import express from "express";
import basicAuth from "express-basic-auth";
import path from "path";
import { fileURLToPath } from "url";
import { createServer as createHTTPServer, Server as HttpServer } from "http";
import { readFile } from "fs/promises";
import { createServer as createViteServer } from "vite";
import type { ViteDevServer } from "vite";
import type { ServerConfig } from "./types.js";
import { createServer as createHTTPServer } from "http";
import { VoiceAssistantWebSocketServer } from "./websocket-server.js";
import { initializeSTT } from "./agent/stt-openai.js";
import { initializeTTS } from "./agent/tts-openai.js";
@@ -16,10 +10,7 @@ import {
deleteConversation,
} from "./persistence.js";
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
async function createServer(httpServer: HttpServer, config: ServerConfig) {
function createServer() {
const app = express();
// Basic authentication
@@ -61,75 +52,14 @@ async function createServer(httpServer: HttpServer, config: ServerConfig) {
}
});
let vite: ViteDevServer | undefined;
if (config.isDev) {
// Development: Create Vite server in middleware mode
vite = await createViteServer({
server: {
middlewareMode: true,
allowedHosts: ["localhost", "mohameds-macbook-pro.tail8fe838.ts.net"],
hmr: {
server: httpServer,
port: config.port,
protocol: "wss",
path: "/hmr",
},
},
appType: "custom",
});
// Use Vite's middleware to handle HMR, module transformation, etc.
app.use(vite.middlewares);
} else {
// Production: Serve static files from dist/ui
const uiPath = path.join(__dirname, "../ui");
app.use(express.static(uiPath));
}
// SPA fallback - serve index.html for all non-API routes
app.use("*", async (req, res, next) => {
if (req.originalUrl.startsWith("/api")) {
return res.status(404).json({ error: "Not found" });
}
try {
const indexHtmlPath = config.isDev
? path.resolve(__dirname, "../../src/ui/index.html")
: path.join(__dirname, "../ui/index.html");
let html = await readFile(indexHtmlPath, "utf-8");
// In development, transform the HTML with Vite
if (config.isDev && vite) {
html = await vite.transformIndexHtml(req.originalUrl, html);
}
res.status(200).set({ "Content-Type": "text/html" }).end(html);
} catch (e) {
if (config.isDev && vite) {
vite.ssrFixStacktrace(e as Error);
}
next(e);
}
});
return { app, vite };
return app;
}
async function main() {
function main() {
const port = parseInt(process.env.PORT || "6767", 10);
const isDev = process.env.NODE_ENV !== "production";
const config: ServerConfig = { port, isDev };
const httpServer = createHTTPServer((req, res) => {
app(req, res);
});
const { app, vite } = await createServer(httpServer, config);
// Create HTTP server
const app = createServer();
const httpServer = createHTTPServer(app);
// Initialize WebSocket server
const wsServer = new VoiceAssistantWebSocketServer(httpServer);
@@ -167,18 +97,12 @@ async function main() {
console.log(
`\n✓ Voice Assistant server running on http://localhost:${port}`
);
if (isDev) {
console.log(`✓ Vite dev server integrated (HMR enabled)\n`);
}
});
// Graceful shutdown
process.on("SIGTERM", async () => {
process.on("SIGTERM", () => {
console.log("\nSIGTERM received, shutting down gracefully...");
wsServer.close();
if (vite) {
await vite.close();
}
httpServer.close(() => {
console.log("Server closed");
process.exit(0);
@@ -186,7 +110,4 @@ async function main() {
});
}
main().catch((error) => {
console.error("Failed to start server:", error);
process.exit(1);
});
main();
@@ -1,13 +0,0 @@
{
"permissions": {
"allow": [
"WebFetch(domain:vite.dev)",
"Read(//Users/moboudra/dev/voice-dev/packages/web/app/hooks/**)",
"Read(//Users/moboudra/dev/voice-dev/packages/web/app/**)",
"Read(//Users/moboudra/dev/voice-dev/**)",
"Bash(cat:*)"
],
"deny": [],
"ask": []
}
}
Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 134 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

Some files were not shown because too many files have changed in this diff Show More