One Decision Cycle for Interactive and Autonomous Agents

A real-time voice agent and a background task agent feel like two different products. One streams audio over a phone line and has to answer in under a second. The other grinds through a multi-step goal with no human watching. The obvious move is to build them as two systems with two codebases — and then spend the rest of your life keeping their behaviour from drifting apart.

Matrix doesn't do that. Interactive and autonomous AI agents run the same decision cycle. They share working memory, the same typed action space, the same execution path, and the same memory writes. The only thing that branches is one seam — DECIDE — selected by a single persisted property, Agent.mode.

This post shows the cycle, the four seams, and how to launch an autonomous run yourself.

The insight: dialogue is just an action

The architecture is modelled on CoALA — Cognitive Architectures for Language Agents, and its load-bearing idea is deceptively small: talking to a human is just another grounding action.

If dialogue is an action, then "agent talks to a person" and "agent calls a tool with no person present" aren't two architectures. They're one decision cycle where the available grounding actions and the decider differ. Interactive and autonomous collapse into a single loop with a swappable brain.

For the higher-level case for running a cognitive architecture instead of a while loop, see From Chatbot to Cognitive Architecture: CoALA in Production. This post is the mechanism.

Four seams, one branch

Every agent turn — chat message, voice turn, or autonomous step — runs through four seams. Three are shared verbatim across both modes. Agent.mode swaps only the second one.

Read that table top to bottom and notice how little is mode-specific. PERCEIVE, ACT, and LEARN are identical machinery. Even the principal seam isn't a special case the runtime invents for autonomy — it's the same axis access control already keyed on. The cognition got unified along the boundary that authorization was already drawn on.

PERCEIVE — one working memory for everything

Both modes build the same WorkingMemory object via WorkingMemoryAssembler.assemble(...). It's the CoALA working-memory hub: persona + objective + recalled long-term memory + the typed action space + progress so far, read once into a snapshot per cycle. Long-term memory is read a single time into a MemorySnapshot rather than re-queried per field.

There's exactly one PERCEIVE path for chat, voice, and autonomy. Voice and chat compose through the same hub the autonomous loop does, which is how byte-for-byte prompt parity stays an enforced invariant instead of a hope.

ACT — the same executor, the same typed actions

Whatever the decider picks, execution goes through ToolDispatcher.invoke against a typed AgentAction. Every callable is classified REASONING / RETRIEVAL / LEARNING / GROUNDING, and each dispatch is logged with its category. A telephony tool call and an autonomous tool call hit the identical executor — there is no parallel "autonomous tool runner" to drift.

LEARN — same write path, different cadence

Both modes write memory through MemoryService.write. The cadence differs: interactive writes per turn plus a post-session extractor pass; autonomous writes EPISODIC memory per step plus a final summary. Same machinery, two schedules.

The principal seam — agents are principals too

When an agent runs a tool, it runs as a principal. Agent.mode decides scope: INTERACTIVE composes agent ∩ caller (an agent can never surface, through the model, data the human caller couldn't see); AUTONOMOUS composes the agent's own grant. This isn't decoration — unifying the autonomous path under a non-admin agent principal closed a real privilege-escalation gap where background tasks used to run as a platform-admin context that bypassed access control entirely.

The one branch: DECIDE

So what actually changes between modes? Who proposes the next action, and how.

Interactive. The realtime or chat provider is a co-located brain. It proposes and selects the next action in-turn — that's what keeps latency low enough for a live phone call. InteractiveDriver is deliberately thin: it adapts a provider's tool-call into the shared Decision shape and gets out of the way. No separate planning round-trip, because the conversation model already did the planning.

Autonomous. There is no human turn and no co-located brain proposing for free, so Matrix supplies the brain. AutonomousDriver is a full multi-candidate planner: one structured LLM call proposes K candidates ({tool, args, rationale} or {finish, message}), a second call scores them, and the best is selected — reasoning over the full working-memory projection. K is matrix.runtime.decision-candidates (default 3). The deeper case for not letting an agent commit to its first idea is in Multi-Candidate Decisions: Don't Let Agents Take the First Idea.

Same Decision shape comes out either way. Everything downstream — ACT, OBSERVE, LEARN — doesn't know or care which driver produced it.

The autonomous loop

AgentRuntime.runAutonomous(...) is the engine. Stripped to its shape:

loop (step ≤ maxSteps):
  wm   = assemble(AUTONOMOUS, goal, retrieved-memory, action-space, observations)   # PERCEIVE
  d    = autonomousDriver.decide(wm)                                                 # DECIDE
  if d.finish: write EPISODIC result; return completed
  res  = toolDispatcher.invoke(toolCtx, callbacks, d.tool, d.args)                   # ACT
  observations += summarize(d, res)                                                  # OBSERVE
  write EPISODIC step                                                                # LEARN
return budget-exhausted

The OBSERVE step folds each action's result back into the observations, which feed the next PERCEIVE — so the agent's progress accumulates in working memory across steps. The loop ends one of two ways: the driver returns finish (objective met, run COMPLETED), or the step budget runs out (run FAILED). Every cycle appends one row to the run's step log.

Running an autonomous task

You launch a no-human-in-the-loop run with one POST.

POST /api/orgs/{slug}/tasks
{
  "name": "research-and-summarize",
  "agentId": 1234,
  "assigneeKind": "AGENT",
  "payload": {
    "goal": "<the objective>",
    "maxSteps": 6,
    "callerUserId": null,
    "contactContext": null
  }
}

The dispatched TaskRun accumulates the propose → evaluate → select → act → observe log in stepsJson (one row per cycle), writes EPISODIC Memory rows per step and for the final result, and ends COMPLETED (objective met) or FAILED (step budget exhausted). The goal can also arrive under objective, instructions, or description — the payload keys are tried in that order. maxSteps defaults to 8.

One caveat worth flagging

Autonomous deliberation needs the Vertex bean — matrix.gcp.auth.enabled=true, which provides the VertexTextClient that runs the propose-and-score calls (with thinking disabled, for a structured, predictable plan). When that bean is absent — a local boot without GCP auth, say — the driver finishes immediately rather than hanging. Autonomous tasks degrade gracefully: you get a terminal run, not a stuck one. It's a deliberate fail-soft, but it does mean you won't see real multi-step deliberation until the Vertex path is configured.

Why this matters

The payoff isn't architectural elegance for its own sake. It's that you author an agent once and it behaves consistently however you run it. The persona, the tools, the memory, the access rules — none of it forks when you point the same agent at a background goal instead of a live caller. Add a tool, tighten a grant, fix a prompt: it lands in chat, voice, and autonomous runs at once, because there's one composition path and one execution path underneath all three.

That's the difference between an agent runtime and three chatbots that happen to share a logo.

Takeaway: Interactive and autonomous AI agents aren't two systems to build and reconcile. They're one PERCEIVE → DECIDE → ACT → LEARN cycle that swaps a single seam — DECIDE — by Agent.mode. The realtime provider is the brain in-turn for low-latency conversation; AutonomousDriver is the brain that proposes-N and scores when no human is in the loop. Everything else is shared.

Want to run one? Spin up a workspace, set an agent to AUTONOMOUS, and POST /api/orgs/{slug}/tasks with a goal and a step budget — then watch the cycle log accumulate in the run's stepsJson. The full runtime lives in docs/COGNITIVE_CORE.md, and the sibling posts on CoALA in production and multi-candidate decisions cover the surrounding architecture.