Giving AI an Android phone

Giving an AI agent access to an Android phone turns a conversation into a systems problem. The agent needs a way to observe the interface, request an action, and find out what changed. The person holding the phone needs to understand and control that access.

Aster is the Android companion I built around that boundary. It is separate from OpenAlly, the cross-platform AI product I’m building at Matterward Labs.

A request is not an outcome

An instruction such as “open that app” sounds complete in conversation. On a device, it is a request made against a particular state. The app may already be open. A dialog may be covering it. The screen may have changed since the agent last looked.

I think of that interaction as a short loop: observe the available state, choose an action, then observe again. Treating an attempted action as proof of success would hide from both the agent and the person whether the action worked.

This is a design principle I take from working across interfaces and systems. It does not require a model to be infallible. It requires the surrounding software to keep the difference between intention and evidence visible.

Give the device layer a clear job

Aster exposes Android capabilities to agents. I built it as an open-source project for the OpenClaw era, with connection options that do not require a second computer. The Android app can host its own Model Context Protocol server for clients that connect directly to the phone. OpenAlly’s native Android integration uses a separate Binder IPC path between apps on the same device.

The on-device MCP server embeds Ktor and uses Streamable HTTP. Binder IPC avoids the network altogether, with token authentication and on-device approval. Aster also has an optional remote WebSocket mode through a separate server. The Aster repository documents all three transports and their different tool catalogs; the project is MIT-licensed.

A stable boundary is useful even when the interface on either side changes. The agent decides what it wants to attempt. The device layer decides which capabilities it exposes, under what permissions, and what result it can report. Those responsibilities should remain understandable independently.

Permission belongs in the experience

Android accessibility access is powerful because it can reach an interface rather than a narrow purpose-built API. A person needs an explicit way to approve access and stop it. Aster includes device approval and an on-device stop control.

The design question is broader than whether a connection is technically possible. Can the person tell that control is active? Can they intervene when the agent is wrong? What happens if the connection is lost? These questions are useful requirements for any agent that acts outside its chat window.

Keep platform claims precise

Aster’s screen-control role is Android-specific. OpenAlly has a broader cross-platform scope. Combining those two descriptions into “an AI Android app” would lose the architecture and confuse what is available on another device.

OpenAlly can share an agent runtime without promising identical device capabilities everywhere. I prefer to explain the shared job first, then identify which capability needs a particular platform.

The part I keep returning to

The interesting work is the connection between a useful intention and a dependable action. A model can suggest the next step; the product still has to manage state, permissions, and feedback.