Building Biscuit: An On-Device Android AI Assistant with Voice Match & MCP

Why Build an On-Device Android Assistant?

Commercial mobile assistants (Google Assistant, Siri) are rigid: they cannot touch arbitrary apps, cannot execute multi-step workflows without official app APIs, and cannot connect to private developer tools.

Biscuit is my personal Android assistant and agent runtime. It runs natively on my phone, listens for custom wake words, verifies my voice biometric signature on-device, executes actions across any installed app via Android’s Accessibility tree, and connects to external tools using Anthropic’s Model Context Protocol (MCP).

Core Architectural Pillars

1. Native Accessibility Engine (AgentService.kt)

Rather than relying on screen scraping with high-latency OCR, Biscuit taps directly into Android’s native AccessibilityService. It reads the parsed view-hierarchy accessibility tree in milliseconds, navigating Jetpack Compose and legacy views without requiring root or external ADB connections.

2. Streaming Audio & Voice Match Pipeline

• Wake Word: Local wake-word trigger trained on personal audio samples (train_wake_word.py).
• Speaker Matcher: Extracts on-device MFCC voice embeddings to ensure only my voice triggers sensitive device actions.
• Low-Latency Duplex Audio: Pipes audio into Whisper STT, generating streaming response audio via OpenAI tts-1 with instant playback interruption on user speech.

3. MCP (Model Context Protocol) on Android

Biscuit acts as a native mobile MCP client. Through an OAuth connector, it securely binds to remote and local MCP servers, granting the LLM direct access to query Google Calendars, interact with local SQLite databases, and trigger smart home automation endpoints.

4. Floating Overlay & Human-in-the-Loop Supervision

A persistent floating Compose overlay (Overlay.kt) displays the agent’s real-time reasoning thoughts, action intents, and dedicated Pause / Cancel buttons, ensuring complete user control over autonomous app interactions.