Building a NotebookLM-like podcast on my phone
You've probably heard of NotebookLM. Feed it a few sources and it makes a podcast-style conversation. The output isn't as good as a real podcast, and it gives me some AI slop vibes. But it's still pretty impressive.
So I wondered: how good could this be with all inference running on my phone?
tl;dr, if you don't care about the details: jump to the demo
What I tried
Speech
I tried five speech models on my phone's CPU, and on its NPU (Qualcomm's Hexagon Tensor Processor, HTP) wherever I could get them to run. Two made it. In these warm short-input tests, the prepared HTP versions were over twice as fast.
| Model | CPU RTF | NPU RTF | What happened |
|---|---|---|---|
| Supertonic 3 | 0.42 | 0.17 | Both tested samples sounded good |
| Pocket TTS | 0.41 | 0.18 | Both tested samples sounded good |
| Kokoro | 0.98 | blocked | The HTP version failed a check before it reached the phone |
| MOSS-TTS-Nano | 1.12 | blocked | The HTP runtime failed to initialize on the phone |
| VibeVoice 1.5B | no speech | not tried | Silent, incomplete output; the package alone was 9.1 GB |
Script writing
For the script I need a language model. The app uses Gemma 4 E2B on the GPU. I also tried it on the NPU, where it ran but repeated itself on long prompts.
| Model | Backend | Prefill, tokens/s | Decode, tokens/s | Note |
|---|---|---|---|---|
| Gemma 4 E2B | GPU | 981 | 16.3 | What the app uses |
| Gemma 4 E2B | CPU | 68 | 8.7 | |
| Gemma 4 E2B | NPU | 741 | not comparable | Different build. Repeated itself on long prompts |
| LFM2.5-2.6B | GPU | 64 | 7.0 | |
| LFM2.5-2.6B | CPU | 29 | 6.9 | |
| Qwen3.5-2B | GPU | 86 | 8.6 | |
| Qwen3.5-2B | CPU | 34 | 6.4 |
Where I ended up
The whole thing runs on my Galaxy S23 Ultra. Only the one-time model preparation happens on a computer.
- Adapt ONNX graphs Fixed shapes Computer
- Compile HTP contexts Built once Copied to the phone for the two NPU steps Computer
- Add sources Text + instructions Phone CPU
- Write script Gemma 4 E2B Phone GPU
- Review & edit Chat with Gemma Phone CPU
- Revise script Applies your request Optional loop back to review Phone GPU
- Split into chunks Sized for the voice model Phone CPU
- Predict + encode Length + text features Phone NPU
- Generate one voice Refine, then vocoder Phone NPU
- Join audio Adds pauses Phone CPU
- Play / export WAV One episode Phone CPU
Swimlane diagram with one lane per device. On the computer, Supertonic's existing ONNX graphs are adapted and compiled into reusable HTP contexts, once, before use; they are copied to the phone and used by the two NPU steps. On the phone, the CPU runs the app: you add text or Markdown or JSON sources and instructions. The GPU runs Gemma 4 E2B, which writes the script. Back on the CPU you can review and edit the script by chatting with Gemma; each request goes to the GPU, which revises the script, and you can repeat this as often as you like. The CPU splits the script into chunks sized for the voice model. The NPU runs Supertonic 3 to predict each chunk's length and encode its text, then generate one fixed voice. The CPU joins the audio with pauses and plays or exports the episode as a WAV file.
Pocket TTS on my phone
The errors you run into while working on TTS can be interesting. Before Supertonic, I tried Pocket TTS. On my laptop it said this sentence fine. Here it is twice on my phone's CPU, with the same weights, text, voice and noise. The first sounds right. The second is my first port, and it runs 10.72 seconds instead of 4.48 and sounds unhinged.
Pocket TTS runs entirely on this Android phone using the original floating point model.
So what went wrong?
The model expects a special NaN marker as its first input and swaps in its learned start vector. My phone code freed that buffer too early, so the model got zeros instead. It started from the wrong place and never found the end, so it kept talking until I cut it off. The voice went wrong too, because every sound it makes is built on that wrong start.
- Original code
- Buffer kept alive
The demo
Demo video goes here. Add it as docs/qoralo-blog/media/demo.mp4.
If you wanna take a look, here it is.