Designing tool calls for agents that act

A tool call looks tidy in a transcript: a name, some JSON arguments and a result. When the tool taps a button on a phone or deletes a contact, every part of that exchange needs design. Most of what I know about it came from building Aster, the Android companion, and the agent runtime inside OpenAlly, and from failures where an agent did something reasonable with information that was slightly wrong.

Schemas say what is valid

Each Aster action has a short description and a JSON schema for its parameters. The registry’s convention is that every declared parameter is required unless the definition says otherwise. That default is safe for most tools and wrong for a few.

tap is one of the exceptions. It accepts either an element ref from the latest observation or an x and y coordinate pair. A schema that marked all three as required would reject every valid call, so the definition marks none as required and the handler checks the combination:

"tap" to ToolDef(
    "Tap an element by ref (preferred) or by x,y coordinates",
    mapOf(
        "ref" to PropDef("string", "Element ref from screen_observe"),
        "x" to PropDef("number", "X coordinate (coordinate fallback)"),
        "y" to PropDef("number", "Y coordinate (coordinate fallback)")
    ),
    required = emptyList()
)

That snippet is simplified from the registry. When neither form is present, the handler answers with a plain instruction to provide ref, or both x and y.

Normalise arguments in one place

Models follow the schema they are shown, which in Aster’s case uses snake_case. Some older handlers read camelCase. For a while a mismatched key was simply ignored, so an argument the model had chosen carefully never reached the code.

The fix was one normalisation step that runs before dispatch on all three of Aster’s transports. I prefer that to teaching each handler two spellings. The rule lives in one place, and a new handler cannot forget it.

Results should describe outcomes

A result that says only success: true invites the agent to assume the world changed. Aster’s screen actions return a fixed shape instead: whether the act succeeded, which strategy resolved the element, whether anything on screen changed, which app is in front afterwards and whether the screen settled. changed is allowed to be false. If a tap landed and nothing moved, the agent should hear that.

Audit records follow the same rule. On the IPC path, the log row for a screen action records the target text, how the element was resolved, the risk level and any approval. Those values come from what the runtime actually sent. When one is missing, the log stores nothing for it rather than a plausible default.

Errors should point at the next step

An error message is part of the interface for the model, and a vague one leads to a guess. In OpenAlly’s kernel, hard tool failures become ordinary error results, so the model always sees one shape and can correct itself.

The wording matters as much as the shape. If a model calls a tool that does not exist, the kernel checks whether the name belongs to a skill or is close to a real tool, and says so. If nothing matches, it tells the model to search for a tool or load a skill. A refusal for an unknown foreground app once told the model, and through it the owner, to look for a setting that does not exist. It now asks the model to pass the foreground_package from its most recent observation, which is the actual fix.

Some outcomes cannot be known. When a tool times out, the kernel asks the call to stop and returns a message saying the outcome is unknown and that the agent should check the action’s real state before retrying. For an action on a payment screen, a blind retry is the dangerous choice.

Unknown is a valid state

The device side takes the same position. Aster’s tracked execution protocol, which OpenAlly uses for screen-control actions, binds each request to its exact payload bytes, a SHA-256 hash and a request ID before submission. A status query never runs the command. A separate call can seal a request that has not started, so a delayed submit cannot run it later.

Losing the connection does not prove that nothing happened. The protocol notes put it directly: a new boot, a missing status or a lost transport is never proof of completion. A duplicate submit observes the original record instead of acting twice.

Permission checks belong at dispatch

Offering a tool and allowing a tool are separate decisions, and I enforce both. Each OpenAlly agent has a grant, and denied tools are removed from the schemas it is offered. Models still sometimes call a name they were never shown, so the dispatcher checks the deny list again and refuses the call as not permitted for that conversation.

Screen actions add a risk check before anything reaches the phone. The classifier is deterministic and never uses a model. An action counts as high risk when its target text means send, pay, confirm, delete, post or similar words in several languages, when the foreground app is a payments or banking app, or when it toggles a system setting or a power action. A high-risk action is staged instead of run. The model receives a pending_confirmation with an ID and a summary, and the owner decides. Low-risk actions go through with the verdict stamped into their parameters, so Aster can record it.

Aster checks again on its own side. Its package policy refuses control of denylisted apps even if the kernel has never synced a policy to it, and it refuses when it cannot tell which app is in front.

Destructive actions can carry their own verification as well. delete_contacts_verified re-checks each name before deleting it, and it is the call OpenAlly sends after the owner approves a contacts clean-up.

Fit the tool list to the model

A schema costs tokens every time it is sent. On a small on-device model with a 4,096-token window, the core set of tool schemas used for cloud models measured roughly 3,900 to 4,500 tokens by itself, enough to fill the window before the conversation began.

For that tier, the kernel sends a small set of full schemas up front: the core tools that tier can use, the tools most relevant to the turn and anything the owner pinned. The rest stay listed in a directory the model can search and activate during the run. The grant does not change. The selection decides which schemas are sent first, never which tools the agent may reach.

Where the fixes usually land

A tool definition is a contract between a model that can be wrong and software that has to be dependable. Looking back through these changes, the model was rarely the part that needed fixing. More often it was a schema that rejected valid calls, a result that hid what happened, or an error message that pointed somewhere unhelpful.