Building an AI browser for Android on Chromium

TaffyGo is a web browser for Android that I’m building at Matterward Labs. It blocks ads and trackers from the first page you open, and it has one assistant, Taffy, which can answer questions about the page you are reading or take on a task, such as comparing a price on two shops.

Most of the engineering is about the second part. An assistant that can click and type inside a browser has a lot of reach. This article covers how the code is arranged so that reach stays inside rules a person can see. The first-party source is public under the Mozilla Public License 2.0, so everything here can be checked in the repository.

Why start from Chromium

A browser has to be a good browser before anything else matters. Page compatibility, media, downloads, the security model and years of fixes come with Chromium, and I did not want to rebuild them. TaffyGo is built from Chromium 152, runs on 64-bit ARM phones with Android 10 or later, and carries no Google API keys. That last choice means there is no Google sign-in or sync, and the product says so directly instead of leaving people to find out.

The cost of building on Chromium is the fork. Chromium moves quickly, and a downstream browser that drifts too far from it becomes expensive to keep current and secure.

Keeping the fork small

The repository does not contain a Chromium checkout. The checkout lives outside the tree and can be reproduced from three inputs: a pinned upstream revision, a numbered patch queue, and the taffy-core/ folder that holds all of TaffyGo’s own code.

That folder is mounted into the checkout at //taffy. New product code goes there and never into Chromium’s files, so it rebases without conflicts. The patch queue is reserved for the edits that genuinely have to touch upstream code, such as registering the product target, packing TaffyGo’s string catalogues, or turning off defaults that point at Google services. Each patch has a written specification next to it explaining why it exists.

A check counts the realised patches and the upstream lines they modify against a budget for fork debt. When the queue reaches that budget, new edits to upstream files are refused until something is retired. Some entries have been. The plan to change the user agent was dropped before it was written, and TaffyGo keeps Chromium’s. A Compose status island, the largest and riskiest patch in the queue, was retired after four days. Removing a patch has been as useful as adding one.

The Chromium pin moves through a milestone rebase roughly every four weeks, within five working days of an upstream stable release.

Four parts, four kinds of authority

Inside the app, the work is split between parts that are trusted with different things.

• The browser process is Chromium’s, written in C++. It owns the network, storage, tabs and the person’s keys. It makes every request, including the ones to the AI provider.
• The core service holds Taffy’s logic, written in Rust, inside a sandboxed utility process. It never receives a key, a cookie, a file path or a network connection. Within it, task-engine proposes a step, policy-engine decides whether that step is allowed, and audit-engine records it.
• The renderer reports the structure of a page and carries out actions on it. The browser authorises each action separately, and the renderer never holds that authority itself.
• The Compose UI, in Kotlin, talks to the browser through one generated interface.

The core can propose anything, and it can do nothing by itself. A bug in task planning produces a bad proposal that policy can reject. It cannot quietly read a cookie it was never given.

Contracts between processes

Each boundary between those parts is a generated contract with golden fixtures, kept under taffy-core/contracts/. There are five: the page protocol between renderer and browser, a browsing seam for tabs, navigation and downloads, the UI facade, the browser-to-Rust service protocol, and a tool runtime protocol. A deliberately small generator, built only on the standard library, emits the projections each schema declares, across Rust, Kotlin, TypeScript, C++ and Mojo.

The schemas use append-only ordinals and closed enumerations. An unknown enum value stops processing. In C++ this needed special care, because casting an integer to an enum that has no matching value compiles fine and is undefined behaviour. At a trust boundary, that would let a bad value travel on and fall through every later switch. So the C++ projection is a header of decoders that return an optional value, the counterpart of the from_wire functions in Rust and Kotlin.

One rule came from a mistake. A contract emits only the projections it declares, and adding a language is the same change as wiring up the target that compiles it. Before that rule, the repository carried three thousand lines of generated Kotlin that no build target compiled. They had never been through a compiler, so they could never fail, and in review they looked as if they had been checked.

Asking before acting

The visible behaviour follows from that architecture. Taffy does its tasks in tabs of its own, kept apart from the person’s tabs, and describes each step in plain words. The person can take over, pause or stop at any point.

Taffy needs permission to open more tabs. Before it fills in a form, it shows the exact values it intends to enter. That approval works once, and it never submits the form. Passwords and one-time codes are handed back to the person; Taffy will not type them.

Blocking is on for every site by default, with a switch for each site. Matching runs on adblock-rust against EasyList and EasyPrivacy, and both lists ship inside the app, so blocking does not depend on a TaffyGo server.

The provider belongs to the person

TaffyGo sells no AI access and runs no server of its own. Taffy talks to a provider the person sets up: an API key, an existing plan at one of the providers that support account sign-in, or their own model server, such as Ollama, LM Studio, llama.cpp, vLLM or any OpenAI-compatible endpoint. The Android keystore seals each key, and the key is sent only to its provider.

There is no account. History, bookmarks and Taffy’s Memory stay on the phone. An optional backup is encrypted with AES-256-GCM under a recovery key that only the person keeps, and it never contains passwords, sign-ins, provider keys or browsing history. Without a provider, TaffyGo is still a complete browser with blocking, and Taffy waits until one is connected.

Building it, and being honest about it

A full Chromium build needs an x86-64 Linux machine with plenty of memory and around 400 GB of free disk. Most work does not need that. The Rust core, the Compose UI and the website each have fast loops that run on Linux, macOS or WSL2 without a Chromium checkout:

cargo test --workspace --locked                     # the Rust core
./gradlew --no-daemon lintDebug testDebugUnitTest   # the Compose UI
./tools/check fast                                  # every check this host can run

When a check’s toolchain is missing, it says it skipped and why. It never reports a pass for something it did not run. The same habit applies to the product’s description of itself: 1.0 is the first release, and so far it has been tested on one physical phone. The README says that, and asks people to open an issue with their phone model if TaffyGo misbehaves.

TaffyGo is on Google Play, and signed APKs with checksums are published on GitHub. If you want to help, the repository has issues labelled for first contributions, and every commit carries a Developer Certificate of Origin sign-off.