How OpenAlly is put together

OpenAlly is the AI agent product I’m building at Matterward Labs. People use it to run parts of a business from a phone and to get through everyday work: a store, an order, an invoice, a plan, a post. The case study describes what it does. This piece describes how the code is arranged as it stands today.

Interfaces in TypeScript, the runtime in Rust

The repository is a pnpm and Turborepo monorepo with two app heads. The mobile app is built with Expo and React Native and runs on iPhone and Android. The desktop app is a Tauri shell, with macOS as the current desktop target. Both draw on shared TypeScript packages for UI, features, state, data, navigation and platform code.

Everything that acts happens in a Rust kernel. It owns the agent loop, the connections to model providers, messaging channels, skills, tools and on-device inference. The interfaces present work and collect decisions; the kernel carries the work out. On Android the kernel ships as a native library built for arm64, and on iOS it is packaged as an XCFramework.

How the screens reach the kernel

On mobile, React Native reaches the kernel through JSI, the native interface in React Native’s current architecture. The app sends a request such as chat:send through a dispatch call and receives a stream of kernel events back: text as it arrives, tool calls, tool results and the end of a turn.

On the Mac, the Tauri shell links the kernel crates directly. A small bridge forwards kernel events into the webview as one named event stream, and stored credentials are sealed with the macOS Keychain. The two heads use different native plumbing, yet they send the same requests to the same runtime.

That gives the product one definition of an agent run. A new screen does not get its own idea of how a tool is called, retried or refused.

Replacing the embedded Node runtime

The kernel was not the first version. Earlier builds ran an embedded Node.js runtime on the phone with a JavaScript agent core inside it, and the interface talked to it as a local gateway. The move to Rust was planned in numbered phases, and the core architecture document states the motive: keep the agent behaviour, remove the Node runtime, avoid garbage-collection pauses on the hot streaming path and get type-checked composition.

The final phase deleted the gateway. There is no Node process on the device now, and the repository notes say that any reference to a local gateway port describes plumbing that has been removed. I like keeping notes like that next to the code. They save the next person from debugging something that no longer exists.

A kernel in layers

The Rust workspace is large, so it is organised in rough layers. At the bottom are stores for agents, sessions, apps, automations, tasks, inventory, payments and credentials. Above them sit families of tools, then channels, then model providers, and finally the RPC handlers that the interfaces call.

The agent loop has a familiar shape. It takes the conversation, compacts it when it grows too long, streams a response from the selected provider, runs any tool calls and feeds the results back until the model stops. Tool calls run in a bounded number of slots, and each call has its own timeout. When a timeout fires, the result the model receives says the outcome is unknown and asks it to check the real state before trying again.

Agents own their model

In OpenAlly, the model belongs to the agent. A chat request must name an agent, and the kernel resolves the model and credentials from that agent’s record. The model picker beside the composer edits the selected agent, so there is no separate global setting that can disagree with it.

People can connect providers with their own keys, use supported subscriptions, point at a self-hosted endpoint or choose an on-device option where the feature and hardware support it. Provider keys stay in secure storage on the device and are presented only to the provider the person chose. The OpenAlly trust page describes the other side of that boundary: a request sent to a remote model goes to that model’s service, and OpenAlly Cloud requests are processed by OpenAlly.

Skills, tools and grants

Agents gain abilities along two separate axes. A skill carries a teaching prompt, so granting one costs model context. A feature tool, such as the PDF reader, carries no prompt and costs nothing until it is called. Each agent has a grant on each axis: all, none or a named list, plus a deny list for removing one item from an otherwise full grant.

The deny list is enforced twice. Denied tools are removed from the schemas the model is offered, and the dispatcher refuses them again if a model calls one by name anyway. I wrote more about that pattern in Designing tool calls for agents that act.

Built-in skills reach devices through a signed catalog rather than inside the app binary, so a prompt change ships with a catalog release instead of an app release. A few skill engines on Android are WebAssembly modules, and the kernel loads one only when its file matches a pinned hash.

Where data lives

The storage boundaries differ by kind of data, and I try not to blur them. Messages and knowledge are stored on the device. Business records and notes are kept in the person’s OpenAlly account, so products, orders and invoices match on every device they sign in on. A backup is encrypted on the device before it goes to Google Drive, and OpenAlly does not hold the recovery phrase that opens it.

Aster stays separate

Screen control on Android belongs to Aster, a separately installed companion. On Android, OpenAlly does not request accessibility access, notification access or permission to draw over other apps. When someone installs Aster, OpenAlly drives it over Binder IPC on the same phone; Three ways into one Android phone covers that link.

The companion face follows the same split. OpenAlly computes the face and streams its frames, while Aster owns the window that draws it over other apps. Each app holds only the permissions its own job needs, and the core product works without Aster installed at all.