Giving AI an Android phone

Satyajit Pradhan — Mon, 14 Sep 2026 12:00:00 GMT

Giving an AI agent access to an Android phone turns a conversation into a systems problem. The agent needs a way to observe the interface, request an action, and find out what changed. The person holding the phone needs to understand and control that access.

Aster is the Android companion I built around that boundary. It is separate from OpenAlly, the cross-platform AI product I’m building at Matterward Labs.

A request is not an outcome

An instruction such as “open that app” sounds complete in conversation. On a device, it is a request made against a particular state. The app may already be open. A dialog may be covering it. The screen may have changed since the agent last looked.

I think of that interaction as a short loop: observe the available state, choose an action, then observe again. Treating an attempted action as proof of success would hide from both the agent and the person whether the action worked.

This is a design principle I take from working across interfaces and systems. It does not require a model to be infallible. It requires the surrounding software to keep the difference between intention and evidence visible.

Give the device layer a clear job

Aster exposes Android capabilities to agents. I built it as an open-source project for the OpenClaw era, with connection options that do not require a second computer. The Android app can host its own Model Context Protocol server for clients that connect directly to the phone. OpenAlly’s native Android integration uses a separate Binder IPC path between apps on the same device.

The on-device MCP server embeds Ktor and uses Streamable HTTP. Binder IPC avoids the network altogether, with token authentication and on-device approval. Aster also has an optional remote WebSocket mode through a separate server. The Aster repository documents all three transports and their different tool catalogs; the project is MIT-licensed.

A stable boundary is useful even when the interface on either side changes. The agent decides what it wants to attempt. The device layer decides which capabilities it exposes, under what permissions, and what result it can report. Those responsibilities should remain understandable independently.

Permission belongs in the experience

Android accessibility access is powerful because it can reach an interface rather than a narrow purpose-built API. A person needs an explicit way to approve access and stop it. Aster includes device approval and an on-device stop control.

The design question is broader than whether a connection is technically possible. Can the person tell that control is active? Can they intervene when the agent is wrong? What happens if the connection is lost? These questions are useful requirements for any agent that acts outside its chat window.

Keep platform claims precise

Aster’s screen-control role is Android-specific. OpenAlly has a broader cross-platform scope. Combining those two descriptions into “an AI Android app” would lose the architecture and confuse what is available on another device.

OpenAlly can share an agent runtime without promising identical device capabilities everywhere. I prefer to explain the shared job first, then identify which capability needs a particular platform.

The part I keep returning to

The interesting work is the connection between a useful intention and a dependable action. A model can suggest the next step; the product still has to manage state, permissions, and feedback.

Designing interactions around a laptop hinge

Satyajit Pradhan — Mon, 14 Sep 2026 12:00:00 GMT

A laptop hinge is an unusually good invitation to think about interaction. It already moves for a reason. It has a physical range. And the person using it does not need to learn a new gesture.

UnfoldMyMac began around that possibility: connect desktop effects to the MacBook lid. The app now includes lid effects, dynamic wallpapers, and creative scenes, with a native Swift, AppKit, and Metal implementation.

Begin with the physical relationship

When an effect responds to a hinge, the relationship between movement and image matters more than an elaborate animation. If the lid moves a little, the visual response should make that movement understandable. If the lid stops, the effect should not seem to continue on an unrelated schedule.

That is a different design problem from a timed transition. A transition has a beginning and an end chosen by software. A physical input can reverse halfway through, pause, or arrive in small, uneven updates. The visual system needs a meaningful state for all of those moments.

A useful model of the interaction

I think about the experience in three parts:

Input: what movement or device state is available?
Mapping: how does that input become a position, amount, or visual parameter?
Presentation: how does the result appear without fighting the desktop?

Separating those questions makes the behaviour easier to reason about. A rendering change need not redefine the input. A different device capability need not mean inventing a new visual language.

This is a design model, rather than a claim that every effect uses the same implementation. The UnfoldMyMac source has the details of the current native app.

Let people see what a design does

A still image explains appearance, but it does not explain an interaction. A person choosing a lid effect needs to understand the relationship to movement. A person choosing a wallpaper needs to know whether it reacts to a different input or behaves independently.

So the UnfoldMyMac library has to show a preview and say what each effect responds to. The question is not simply “Do I like this picture?” It is also “Do I want my desktop to behave like this?”

Motion has a lifecycle

A delightful effect is still a running system. It has resources to manage and moments when it is no longer visible. Its behaviour around closing, reopening, and changing context is part of the design.

That concern carries into this portfolio. The machine has short responses to a person’s selection, and the page remains useful without them.

Follow the small question

What interested me was noticing a control already present in the object, rather than adding another to a screen. Following that question led from a laptop hinge into native rendering and a collection of effects.

Satyajit Pradhan: Notes from building