Retriever: Composing Closed-Loop Asynchronous Robot Programs

A programming model and runtime for composing closed-loop asynchronous robot programs with explicit clocks, synchronization policies, and deterministic replay.

Linfeng Zhao 1
Haojie Huang 2
Jiayuan Mao 3
Weiyu Liu 1
Mykel Kochenderfer 1,*
Lawson L. S. Wong 2,*
1 Stanford University
2 Northeastern University
3 MIT
*Equal advising

Retriever spans an asynchronous decision model, a programming model, and a runtime for closed-loop robot agents. Agents are graphs of stateful Flows executed on explicit clocks; each edge declares how upstream history is consumed. The same graph can be stepped locally or executed asynchronously across supported backends while preserving its temporal meaning.

The problem

General-purpose robot agents need perception, memory, planning, monitoring, learned skills, and low-level control to keep running together while the world changes — all at very different rates. Existing middleware operates at the message-passing level; Retriever raises the abstraction to the behavior level, where the programmer describes what the agent does and when, not how messages are routed. The core challenge is giving the whole agent a programming model where composition, feedback, timing, and replay are explicit parts of the program rather than hidden in callback glue or scheduler behavior.

Why a programming framework, now?

Three trends shape this bet:

  1. Programs are the interface to foundation models. General-purpose coding agents can already write and debug programs together with roboticists, while end-to-end VLM and video models for robot control are still maturing. Programs are the key intermediate representation between foundation models and robots.
  2. Modular is inevitable; integrated is coming. As data scales, robot systems will gradually shift from modular pipelines toward more integrated, end-to-end systems — but each hand-built modular system specializes to one domain. The durable investment is the abstraction beneath both: explicit time, modular composition, and deterministic dataflow.
  3. Deterministic dataflow is differentiable. Because a Retriever graph is deterministic, execution yields replayable data and gradients along the graph. The same abstraction that coordinates today’s modular pipelines can support training tomorrow’s integrated ones.

Retriever-0: a concrete closed-loop agent

Retriever-0 is our real-robot case study built inside one Retriever pipeline. It combines a VLM planner, belief and memory, an execution monitor, a VLA skill policy, and a joint controller in one coordinated loop — slow reasoning, medium-rate learned skills, and high-rate control all exchanging information during execution.

Canonical multi-rate closed-loop robot agent pipeline with planning, skill, and control modules.

The Retriever-0 pipeline: slow VLM planning (with belief and memory), execution monitoring, medium-rate VLA skill execution, and high-rate control — all running simultaneously at mismatched clocks inside one program.

Real-robot task demonstrations

The same asynchronous pipeline supports two distinct long-horizon behaviors: belief-guided search across closed drawers and bimanual retrieval from a deformable shopping bag.

Nine numbered frames show a bimanual robot opening and inspecting the top-left drawer, closing it after the spice is absent, then opening the top-right drawer, picking the spice, seasoning food, closing the drawer, and finishing.
Belief-guided spice search and seasoning. The robot inspects the top-left drawer, updates its belief when the spice is absent, continues the search in the top-right drawer, then picks the spice, seasons the food, and closes the drawer.
Twelve numbered frames show a bimanual robot lifting a deformable shopping bag, retrieving produce, sorting the objects onto two colored plates, and releasing the bag.
Shopping-bag retrieval and sorting. One arm stabilizes and lifts the deformable bag while the other retrieves objects, sorts them by category, and releases the bag.

Closed-loop by design

The Retriever-0 pipeline is not a feedforward stack — feedback loops keep belief, plans, and control state current during execution:

PerceptionBelief / memoryPlanningMonitorSkill / control
feedback updates belief, plans, and control
01

Perception-planning loop

New observations update belief and memory, so the planner reasons over the latest task state instead of a stale prompt snapshot.

02

Planning-execution loop

The execution monitor connects plan chunks and progress-prediction handoff, allowing replanning and skill switching while execution continues.

03

One multi-rate program

Slow planning, medium-rate skill inference, and high-rate control run together because clocks and sync policies define the timing contract.

What a Retriever program looks like

Retriever centers the program around a small vocabulary: Flow is the compositional unit, Clock says when each Flow runs, Sync policy says what upstream history is consumed at each step, and Pipeline wires Flows into a closed-loop graph. The runtime maps the graph onto supported backends without changing its semantics.

Program surface

Define clocked Flows, connect them with edge-level sync policies, then run the same graph locally for debugging or on a deployment backend.

Minimal Retriever pipeline (Python)
# 1) Define Flows (what computes) and clocks (when they run)
top_cam   = CameraSource(id=0)       @Rate(hz=30)
wrist_cam = CameraSource(id=1)       @Rate(hz=30)
belief    = BeliefMemoryFlow()       @Trigger("inspection_done")
monitor   = ExecutionMonitorFlow()   @Trigger("belief_updated", "progress_prediction")
planner   = VLMPlanFlow("gemini")    @Trigger("replan")
vla       = VLASkillFlow("pi05")     @Rate(hz=2)
robot     = ControllerFlow(id=0)     @Rate(hz=200)


# 2) Compose the graph and declare sync policies on edges
pipe = Pipeline("Closed-loop Agent")
with pipe:
    wrist_cam.then(vla, sync=Latest()) \
             .then(robot, sync=ActionChunking())


    top_cam.then(belief, sync=Latest()) \
           .then(planner, sync=Latest()) \
           .then(monitor, sync=Latest()) \
           .then(vla, sync=Latest())


# 3) Debug locally or deploy asynchronously
pipe.step(dt=0.1)
pipe.run(backend="dora")

Interactive timeline

This is one robot program shown as time moving left to right. Each row is a Flow that wakes up on its own clock; arrows are computation, and blocks are outputs that remain usable until replaced.

t = 1.2s · high-rate control keeps flowing between slow policy calls
camera flow
Rate(10Hz)
belief/memory flow
update belief t = 5sExecuting an information-gathering action produces new evidence, so memory updates before planning continues. update belief t = 10sProgress prediction opens the next handoff point for the monitor.
planner
compute plan t = 2sThe planner proposes a plan chunk that can be consumed while execution continues. replan t = 7sA progress prediction opens a revision step, so the planner reads current belief again.
VLA skill policy
infer action t = 4sThe VLA skill emits a time-extended action chunk plus progress prediction. t = 6sThe skill policy refreshes commands using the latest synchronized inputs. t = 8sThe skill policy continues refreshing short action chunks while control runs. t = 10sProgress prediction selects the next policy step without stopping control. t = 12sThe final visible action chunk carries the controller toward the end of this window.
controller flow

Dots mark Flow starts. Dashed links show one event causing another; they are not regular global ticks.

clocked procedure start computation time trigger / switching edge plan / action chunk high-rate control tick

Read the full technical blog post →