Perception-planning loop
New observations update belief and memory, so the planner reasons over the latest task state instead of a stale prompt snapshot.
We built Retriever to make multi-rate robot systems programmable. Perception, memory, planning, learned skills, and control remain reusable Python modules, while their clocks and data dependencies become explicit parts of one closed-loop graph.
The same graph can run one step at a time during debugging or asynchronously on a robot without changing what the program means. Our work develops the formal model, the Python framework, the runtime, and Retriever-0, the real-robot pipeline shown below.
Contributions
These four pieces are usually treated separately. Continuous-time semantics define what a closed-loop program means. Flows, clocks, and sync policies express that model in Python. The runtime preserves its temporal meaning during local stepping and asynchronous execution. Retriever-0 puts the full stack on a real robot.
The videos show the system at work. The pipeline, feedback diagram, Python listing, and interactive timeline show the same agent as behavior, structure, code, and execution.
What the agent can do
Retriever-0 takes goals in natural language, plans with a VLM, keeps a belief of the scene across episodes, and executes learned bimanual skills in one closed-loop program. In Episode 1, the agent searches four drawers for black pepper. In Episode 2, it remembers the top-right drawer and goes there directly. The bag task exercises bimanual retrieval and distribution with a different composition of Flows.
- Episode 1 Search the drawers, update memory after each inspection, find the pepper in the top-right drawer, and season the steak.
- Episode 2 Use the remembered location, go directly to the top-right drawer, retrieve the pepper, and season the steak.
During each run, the robot keeps moving while perception, planning, and control advance on separate clocks. Coordinating those clocks is the programming problem Retriever addresses.
The problem
Control may tick every few milliseconds while a VLM takes several seconds. Between them, perception, memory, planning, and learned skills update as the world changes. The behavior depends on which data crosses each rate boundary and when. Most robot stacks spread those decisions across callbacks, queues, and scheduler behavior. Retriever makes them part of the program.
Compose temporal modules
A Retriever agent is a graph of reusable temporal modules. A Flow owns local state, a Clock decides when it runs, and a Sync policy on each edge selects which upstream history it consumes. These declarations keep a slow planner, a learned skill, and a fast controller independent without pushing timing logic into callbacks.
Retriever-0: a concrete closed-loop agent
Retriever-0 uses those primitives in the demonstrations above. Its graph connects a VLM planner, belief and memory, an execution monitor, a VLA skill policy, and a joint controller. Seconds-scale reasoning, 2 Hz skill inference, and 200 Hz control exchange information while the robot moves.
The Retriever-0 pipeline. Each module is a clocked Flow, and each connection carries a sync policy.
The forward path carries observations toward action. Feedback from inspection and skill progress updates belief, replans, and switches skills while control continues.
Closed-loop by design
Feedback is part of Retriever-0’s graph. New observations change the belief used by planning. The execution monitor reads plan chunks and skill progress to decide when to replan or switch skills, while the controller keeps running.
PerceptionBelief / memoryPlanningMonitorSkill / control
feedback updates belief, plans, and control
Planning-execution loop
The execution monitor uses plan chunks and progress predictions to trigger replanning or skill switching while execution continues.
One multi-rate program
Slow planning, medium-rate skill inference, and high-rate control run together because clocks and sync policies define the timing contract.
In Python, these loops are ordinary Pipeline edges declared with the same Flow, Clock, and Sync vocabulary.
What a Retriever program looks like
The same graph fits in a short Python program. Flows define computation, clocks define run conditions, sync policies define what crosses each edge, and a Pipeline composes the pieces.
Program surface
Each task class implements the core Flow API. The surrounding program assigns clocks, wires edges, and chooses sync policies. step() and run() execute the same graph.
# 1) Define Flows (what computes) and clocks (when they run)top_cam = CameraSource(id=0) @Rate(hz=30)wrist_cam = CameraSource(id=1) @Rate(hz=30)belief = BeliefMemoryFlow() @Trigger("inspection_done")monitor = ExecutionMonitorFlow() @Trigger("belief_updated", "progress_prediction")planner = VLMPlanFlow("gemini") @Trigger("replan")vla = VLASkillFlow("pi05") @Rate(hz=2)robot = ControllerFlow(id=0) @Rate(hz=200)
# 2) Compose the graph and declare sync policies on edgespipe = Pipeline("Closed-loop Agent")with pipe: wrist_cam.then(vla, sync=Latest()) \ .then(robot, sync=Chunking())
top_cam.then(belief, sync=Latest()) \ .then(planner, sync=Latest()) \ .then(monitor, sync=Latest()) \ .then(vla, sync=Latest())
# 3) Debug locally or deploy asynchronouslypipe.step(dt=0.1)pipe.run(backend="dora")Execution timeline
Drag across the timeline to follow 15 seconds of execution. Each row is a Flow running on its own clock. Arrows show time spent computing, and blocks show outputs that remain valid until a replacement arrives.
t = 1.2s · high-rate control keeps flowing between slow policy calls
camera flow
Rate(10Hz)
belief/memory flow
update belief t = 5sExecuting an information-gathering action produces new evidence, so memory updates before planning continues. update belief t = 10sProgress prediction marks the current skill complete, so the monitor can switch skills.
planner
compute plan t = 2sThe planner proposes a plan chunk that can be consumed while execution continues. replan t = 7sA progress prediction opens a revision step, so the planner reads current belief again.
VLA skill policy
infer action t = 4sThe VLA skill emits a time-extended action chunk plus progress prediction. t = 6sThe skill policy refreshes commands using the latest synchronized inputs. t = 8sThe skill policy continues refreshing short action chunks while control runs. t = 10sProgress prediction selects the next policy step without stopping control. t = 12sThe final visible action chunk carries the controller toward the end of this window.
controller flow
Dots mark Flow starts. Dashed links connect causes to effects. They do not mark regular global ticks.
- Different clocks. Camera, memory, planning, skill inference, and control do not run at the same rate.
- Reusable outputs. Memory, plan chunks, and action chunks stay valid until a newer block replaces them.
- Continuous control. Feedback can update memory, replan, or switch skills while the controller keeps consuming overlapping action chunks.
clocked procedure start computation time trigger / switching edge plan / action chunk high-rate control tick
The graph, code, and timeline are three views of the same temporal contract. Retriever can step the graph locally or run it asynchronously without changing what the program means.
Why a programming framework, now?
Three developments make the programming layer useful now:
- Coding agents can work with explicit robot programs. Clocks, edges, and data selection are visible in the code they edit and debug.
- Models keep changing. A new planner, skill, or controller can replace one Flow without replacing the program around it.
- Explicit timing produces reproducible traces. Each run records which inputs every Flow consumed, making execution useful for debugging and learning.
Deterministic replay
Each Flow samples inputs by timestamp. Given the same recorded history and a fixed order for equal timestamps, replay produces the same behavior regardless of live scheduling. The trace shows exactly what each Flow consumed, so a run can be inspected during debugging or reused as training data.
The technical blog covers the full formulation, synchronization semantics, runtime mapping, and proof sketch.