One Pepper, Every Door
I built Pepper’s voice features with one customer in mind: my dad. Reading a screen isn’t always easy for him, and I wanted him to be able to just talk to Pepper. The first time I showed him, on my iPhone’s speaker, it kept cutting itself off. I’d ask a question, Pepper would start to answer, and a fraction of a second later it would stop and wait. I was frustrated, because it just wasn’t good enough yet. When I reproduced it later and pulled up the trajectory, the first two answers had been interrupted about 25 milliseconds in, with no speech from me at all. Muting my microphone between questions made it go away. Pepper was hearing its own voice come back out of the speaker and politely yielding the floor to itself.
My mistake was testing only on headphones at first. If I built a voice app again, I’d test on a speakerphone from the first day. A speaker flushes out the hardest problems right away, and if it works there, it works even better on headphones and a handset.
That bug was a good introduction to what voice is like to build. Chris wrote about everywhere you can talk to Pepper now. There’s a voice button in the app and Talk to Pepper on your watch. There’s a phone number you can call, a morning brief that rings you, and an answering service that picks up when you can’t. This is the other half of that story: how those pieces fit together, and the three problems that took most of the work. Hooking a model up to a voice is very different from hooking it up to text. Each of those problems came down to who’s on the other end of the conversation and what the model knows about it. The fundamentals I’ve written about before still carried the work: observability, trajectories, and good old-fashioned debugging.
The place I use voice most is the car. Pepper has my calendar and my email, so I can ask for the address in a message someone sent me or book an appointment. I can throw together a quick note to friends about getting together this weekend. I do all of it without taking my eyes off the road. It’s much more like talking to a person than drafting an email. I don’t have to be precise, because Pepper turns what I said into the message. I wouldn’t write anything long this way, and everyone should know their own limits behind the wheel. That only works because a voice conversation has every tool the rest of Pepper has. Keeping those tools was the one requirement I wouldn’t give up, and it shaped most of what follows.
The Model Owns the Turn
Everything Pepper did before voice was text, and text is turn-based. A message arrives, our agent loop calls the model, looks for tool calls, runs them through our policy gates, appends the results, and loops until there’s an answer. Pepper decides when a turn starts and when it ends.
The Gemini Live API inverts that. It’s a stateful, two-way WebSocket. You stream audio in continuously and the model decides everything else: when you’ve stopped talking, when you’ve interrupted, and when it wants to call a tool. Our loop, with its round cap and its fallback logic, doesn’t carry over at all. That mismatch, not the audio plumbing, was the bulk of the work.
The obvious alternative is a pipeline: speech to text, our normal loop, then text to speech, which hands the turn back to us. We didn’t build it. You speak, then wait for the whole pipeline to run, and that gap feels uncanny. You lose interruptions, and most of what makes a conversation feel fluid. Pepper is meant to be an assistant for people who don’t want AI as a hobby, so it has to feel like talking to a person. A pipeline also wouldn’t have fixed the hardest problem in this post. Whoever detects speech still has to tell your voice from Pepper’s coming out of the speaker.
So we wrote a second runner that sits beside the text loop rather than inside it. It opens the live connection with the same system prompt, memory, and tools the text path assembles. Then it pumps audio in both directions. When the model asks for a tool, the runner sends the call through the same gate the text path uses and hands back the result. It records the whole call as one trajectory, in the same event vocabulary as a text turn. Everything I wrote about in The Observability Gap applies to a phone call too.
A trajectory is our record of a single turn: the request, every tool call and its result, the model’s reasoning, and the reply. Anyone can read their own. pepper trajectories show in our CLI gives you yours. I read mine that way when I’m testing, on a local instance or in production. Nobody at Vycari reads a customer’s trajectories. Those are read only by agents, mostly in an automated process that runs overnight that I call the nightly reflection. It reads the last day’s trajectories, grades them, anonymizes the bad ones, and files bugs or opens fix PRs on its own. I mentioned it in The Backlog Was a Coping Mechanism as the source of most of our issues. A voice call that doesn’t leave a good trajectory is a call that process can’t learn from.
The one real decision was who holds the connection to Gemini. The fastest option is to let the client connect directly with a short-lived token. But then the tool calls show up on the client, and Pepper’s tools, memory, and policy checks either get rebuilt on every device or quietly disappear. Pepper is its tools, and a voice mode without my calendar and mail would be a much smaller product. So the server terminates the Gemini socket on every surface, the app and the Telnyx phone line alike, and we pay a network hop for it. I worried that hop would bring back the same lag that ruled out a pipeline. In practice I don’t notice it.
Pepper was built for this kind of growth before voice existed. Email, SMS, WhatsApp, and chat each come in through their own channel adapter in our comms gateway. The adapter verifies the webhook and turns the message into one common inbound shape the agent loop understands. The same gateway delivers replies on whichever channel you chose. So the gateway grows sideways as people ask for new ways to reach Pepper, and the agent loop stays simple. Voice fits that picture with one twist, shown in orange below. In-app voice and the watch connect straight to the live runner, because they already carry a credential and there’s no carrier in the way. The phone comes in through the gateway like every other channel, through a Telnyx voice bridge that relays the call to the same runner.
How voice and phone reach Pepper
Messaging channels enter through the comms gateway and become one common inbound message for the agent loop. In-app and watch voice connect straight to the live session runner. Phone calls enter through the Telnyx voice bridge in the gateway, which hands the call to the same live session runner. Both runners share Pepper’s tools, memory, policy gate and trajectories.
WHERE YOU ARE
COMMS GATEWAY
CORE
Web and app chat
Email
SMS
WhatsApp
Voice in the app
web, iPhone, Apple Watch
You call Pepper
plus Answering, when forwarded
Pepper calls you
briefs, reminders, “call me”
One adapter per channel
verifies the webhook, parses the message
Common inbound message
verify, claim, enrich, run the turn, send
Delivery
reaches you on the channel you chose
New channels are added here; the agent
loop doesn’t change.
Telnyx voice bridge
caller ID gate, keypad PIN or SMS code
disclosure before capture
restricted floor until verified
answering leg: no step-up at all
outbound: delivery dials, then bridges
Agent loop
text turns: Pepper owns the turn
Live session runner
the model owns the turn
one runner for every voice surface
Shared by both
tools: calendar, mail, todos, maps
memory and system prompt
policy gate and trajectories
Gemini Live
audio WebSocket, straight to core
chosen as delivery
Holding the Gemini socket on the server is what made the rest of our features easy to implement. The runner’s inputs are an authenticated user, a stream of audio in, a place to send audio out, and a context for the tools. The web button was the first client. The iPhone app needed a bearer token on the WebSocket and a tag saying which surface the call came from, and the runner didn’t change. The watch is a native app that dials the same endpoint. The phone needed two small additions, both additive. One lets a caller narrow which tools a session may use. The other lets the server say something mid-call. Inbound calls, outbound calls, and the answering service are all the same runner with a different edge in front of it. In code the contract looks like this, trimmed:
async def run_live_session(
*,
user: User,
audio_in: AsyncIterator[bytes], # PCM16 @ 16 kHz from any client
send_audio: AudioSink, # PCM16 @ 24 kHz back out
send_event: EventSink, # transcripts, status, tool activity
channel: str = "voice_web",
extra_interceptors: list[Interceptor] | None = None,
private_context: bool = True,
proactive_call: bool = False,
...
) -> UUID: # the call's trajectory id
Those last three parameters are where the rest of this post lives.
Pepper Kept Interrupting Itself
Back to the bug I opened with. The Live API’s voice activity detection runs on the server. It gets one mono microphone stream and no copy of what Pepper is playing. It has no way to tell “the user started talking” from “Pepper’s voice came back through the speaker.” Worse, when it decides you’ve interrupted, the in-flight answer is cancelled and dropped from the session’s history. Pepper didn’t just stop talking. It forgot it had answered.
Why Pepper interrupted itself, and how the loop was broken
Left: Pepper’s voice plays from the phone speaker, the microphone picks it up, Gemini’s speech detector hears it as the user interrupting, and the answer is cancelled and dropped from the session’s history, so Pepper stops. Right: the app now gates the microphone while Pepper is audible plus a quarter second, so the loop never closes; on a phone call the carrier sends only the caller’s side of the line, so the loop cannot form.
ON A SPEAKER, BEFORE
NOW
Pepper starts answering
audio plays from the speaker
The mic hears it
one mono stream, no reference
Gemini’s speech detector
can’t tell Pepper’s voice
from yours: “you interrupted”
Answer cancelled
dropped from session history;
Pepper stops and waits
about 25 ms in
Raising the detector’s threshold breaks real interruptions;
lowering it brings the echo back. The fix had to be on the client.
In the app and on the watch
Pepper speaking
Speech detector
Mic gate
no mic audio while Pepper is
audible, plus 250 ms
On a phone call
Pepper speaking
Speech detector
The carrier sends back only the caller’s side of the line,
so Pepper’s own voice never reaches the detector.
caller only
Our first round of fixes reduced the opportunity without removing the cause. We lowered the detector’s start sensitivity, which makes it demand a more confident onset before it barges in. We made tool calls non-blocking and told the model to say a short “let me check” before reaching for one. The silence during a slow calendar lookup was where a lot of the false triggers landed. Sensitivity is a single threshold, though. Raise it and real interruptions stop working. Lower it and the echo comes back.
Getting a conversation to feel natural took a lot of iterations, and almost every one depended on observability we built for voice specifically. So we stopped guessing and measured. Every interruption now records how much of Pepper’s audio was still waiting to play. It also records what fraction of the words the microphone heard were words Pepper was saying at the time. My first version of that overlap metric scored the wrong text and read zero on every healthy turn. We caught it by reading the trajectory from one of our own calls in production, not from a test. Once it was right, two sessions in the same room at the same volume told the two surfaces apart. In Safari on the iPhone, every interruption landed half a second into an answer with more than a second still unplayed. That’s echo. The browser’s echo cancellation doesn’t cover Web Audio playback on iOS Safari. On the native app the backlog read zero, so I concluded Apple’s hardware echo cancellation was doing its job.
It wasn’t. A different bug was hiding it. The Gemini SDK’s receive call ends after each turn, not after each session, and our adapter read that clean end as the server hanging up. The runner had been opening a brand new Live session after every single turn since the day voice shipped. That one bug was behind most of what had looked like model quality problems. Tool calls repeated, answers never arrived, and once the whole reply was the word “try.” It also meant playback had always drained during the reconnect gap, so the native app’s interruptions looked like dead air instead of echo. When we fixed the reconnects, the same phone showed half a second to two seconds of Pepper still playing at every interruption.
The fix on both surfaces is a half-duplex microphone gate. While Pepper is audible, plus a quarter second of tail, the client doesn’t send microphone audio. That costs one thing. In the app you can’t talk over Pepper; you wait for it to finish. On a phone call the problem can’t arise at all, so interrupting works the way you’d expect. The carrier sends us only the caller’s side of the line, so Pepper’s own voice is never in what it hears.
That path had five defects in two days, and we found four of them by reading trajectories from our own calls in production. Two of them were introduced by the fix for the one before. The rule I took from it is now written at the top of the design doc: whatever the client is told, the trajectory records. A status the user can see and the record can’t is a diagnosis you can’t finish.
Caller ID Is Not a Credential
Chris built the phone side on Telnyx, and it’s where the hard questions moved from audio to trust. The web and the apps arrive with a credential: a session cookie or a personal access token. A phone call arrives with a caller ID, and caller ID can be forged. This bridge hands whoever is on the line an agent that can read the account holder’s mail.
So the number only opens a narrow door. It has to be a number that’s already passed signup verification, or Pepper says one line and hangs up. A verified number gets a restricted session. Tools that read private accounts or change anything are refused until the caller proves who they are. They do that on the keypad, with a PIN or a code we text them. A refusal is a string the model reads, so it comes out of Pepper’s mouth as “I need to check it’s really you first.”
The first version gated only the tools, and that left the exposure in plain sight. Pepper’s memory, a summary of today’s conversations, and the list of connected accounts were all loaded into the system prompt when the call connected, before any tool could run. Someone who spoofed your number didn’t need a tool call. They could just ask Pepper what your week looked like. A string that’s already in the prompt isn’t something any gate gets to refuse. The fix was to make the floor cover the prompt as well. A restricted call starts with none of your private context. Once you verify, Pepper hands that context over as a message mid-call, so you get the whole Pepper for the rest of the conversation.
Starting cold created a problem of its own. A model handed no memories and no explanation will make one up. “I can’t seem to reach your calendar” sounds like a broken integration, and a caller who thinks Pepper is broken hangs up instead of verifying. A plausible guess about your week is worse, because to the person on the line it’s indistinguishable from a leak. So a restricted session is told it’s restricted, why, and what fixes it. It’s also told plainly that “I don’t know” is the correct answer.
Verifying doesn’t have to mean looking at the keypad. If you’ve set a PIN, you can type it before Pepper asks. I do that in the car. I have Pepper saved in my contacts as its number, a few commas, and my PIN. Each comma tells the phone to pause before it sends the next digits. The call connects, my PIN goes in on its own, and I’m verified without taking my eyes off the road. That puts the PIN wherever my contacts go, and I’m comfortable with that. The floor exists so a stranger who spoofs my number can’t read my mail, not to stop someone holding my phone. I can rotate the PIN in settings anytime, and four wrong guesses lock it until I verify another way. If your threat level is higher, don’t store it. The texted code is always there.
The disclosure needed the same care. Every call opens with “Hi, this is Pepper. This call is transcribed. How can I help you?” before anything the caller says is captured. We first played that line with the carrier’s text-to-speech. Callers heard a stranger’s voice say one sentence, then a different voice for the rest of the call. Now Pepper says it in its own voice. The bridge drops the caller’s audio until that opening turn finishes, so nothing is captured before the disclosure has been heard. We lost the carrier’s guarantee that the wording is exact. An instruction and a test pin it now, which lowers the risk without removing it.
I wrote about this kind of layering in Putting Up the Guardrails. The phone was the first place I had to build it for a real attacker. It taught me that the prompt belongs to the attack surface as much as the tools do.
The Turn Can’t See the Phone
When Pepper calls you, the trust problem flips. The number wasn’t presented by a caller. We chose it, and it’s yours and already verified. What we can’t know is who picked up. Chris wanted the best wake-up call we could build: you answer the phone just after you wake up, and someone on the other end tells you exactly what you need to know that morning. A call that demands a code before it says anything can’t do that, so the update is spoken first, and anything private beyond it still waits for the keypad. That means whoever answers hears it. Pepper never calls you unless you’ve turned it on, so think about which phone you register and don’t point it at a shared one. Most people don’t answer someone else’s cell phone, so I think the risk is low. A call nobody answers is over: one ring cycle, no voicemail, and no fallback to email. A voicemail is different. It’s your morning brief recorded on someone else’s servers, where anyone near the phone can replay it later.
The first outbound call is my favorite bug in the whole project. It rang correctly. Then Pepper told Chris it can’t place outbound phone calls, and kept saying so when he pointed out that it obviously just had.
Both halves of that came from the same blind spot. The scheduled task’s prompt was literally “Call me right now.” The system decided how to deliver the result only after the turn ran. The turn didn’t know a call was coming. The tool for placing a call is deliberately hidden on scheduled runs, because the delivery is what dials and a run that could dial would ring you twice. So the model answered from its tool list, “I can’t place outbound phone calls,” and the delivery faithfully rang Chris and read him that sentence. On the live call, the model had been told once at the start that it placed the call. That was one message early in the history, and it didn’t hold. Looking at its tools, it had no way to dial from inside the call, which is true, because it was already on one.
The first outbound call
A scheduled task with the prompt “Call me right now” ran as an ordinary turn. The call tool is hidden on scheduled runs and the delivery channel had not been chosen yet, so the model wrote “I can’t place outbound phone calls.” Only after the turn did delivery pick a phone call, so the phone rang and Pepper read that sentence aloud. Now the delivery is checked before the turn, and a call Pepper placed carries a standing instruction saying so.
THE FIRST CALL
the layer that writes
the layer that knows
1. Scheduled task fires
prompt: “Call me right now.”
runs as an ordinary turn
2. The model answers
no call tool on scheduled runs,
so it writes: “I can’t place
outbound phone calls.”
3. Delivery is chosen
only after the turn ends:
this one goes by phone call
4. The phone rings
Pepper reads the reply aloud:
“I can’t place outbound
phone calls.”
NOW
Before the turn runs
The delivery is checked first. If the reply will be spoken
on a call Pepper places, the turn gets a note saying so.
What the model needs to know about delivery has to
reach it before it writes.
On the live call
A standing line in the system instructions: you placed
this call, so never say you can’t, and never contradict
a user who says you just did.
One opening message early in the history didn’t hold.
The fixes are small. A turn whose reply will be spoken on a call now gets a note saying so before it runs. A call Pepper placed gets a standing line in its system instructions that says it placed this call and should never contradict a user who says so. Across all of these surfaces, a turn composes text for a delivery it can’t see. How the reply will reach the person, and whether they’re verified, has to be told to the model. The layer that knows runs after the layer that writes.
The answering service takes that to its end point. A forwarded call is never you, so the path that would let a caller prove they’re the owner doesn’t exist on that leg at all. Pepper answers from the one note you chose and nothing else. The safest agent for that job is one that doesn’t have what it shouldn’t say.
Voice Is a Different Interface
Most of what I’ve learned building agents came from text, where the model writes and a person reads, usually with a moment to spare. Voice takes away that margin. The model decides when you’re done talking. You can’t scroll back, and you hear a wrong answer in real time. And on a phone, there’s no screen at all to explain who is on the other end.
What surprised me was how much of the work came down to telling the model facts about the conversation it couldn’t otherwise see. Whether the caller is verified. Whether Pepper placed the call. That the reply will be spoken. Even today’s date and the user’s language, which the live session originally had no way to know. Most of those started as a confident, wrong answer from a model reasoning correctly from what it had.
None of that made the old fundamentals less important. The reconnect bug looked like a model getting worse until a trajectory showed a new session after every turn. The echo metric read zero until a trajectory from one of our own calls showed it was scoring the wrong text. Observability, trajectories, and plain debugging found the bugs in voice the same way they find them in text.
The architecture held up better than I expected. One runner took six surfaces with only additive changes, which is the argument for putting the boundary at the server and keeping the tools there.
My dad is now our heaviest voice user. He talks to Pepper more than anyone else on the platform, almost never through the screen. That’s the part of this I’m proudest of. For a lot of people a voice interface is a convenience. For some it’s the way in.
A user wrote to me recently that they’d called Pepper more in the last month than they’d called their mom. I love hearing that. Please still call your mom, and if finding the time is the hard part, Pepper can help with that too.
If you want to try it, once you have an account, tap the voice button in the iPhone app, or call +1 727 558-4453 and ask what’s next on your calendar. If Pepper ever talks over itself, you now know why. Pull up your trajectory with the CLI and send it to me.
#Agents #AI #Developer #Startups
