FHNW · School of Engineering · Master’s thesis
A vision-language-action policy, the workflow layer that orchestrates it, and the visual servoing that places the gripper for it — on a rover that competes at the European Rover Challenge.
What this is
The FHNW Rover Team competes in the European Rover Challenge, where one of the tasks is operating a maintenance panel. Building on a ROS 2 stack that already ran per-task imitation policies, this thesis extends the rover’s autonomy in five areas, over a reworked manipulator control and safety layer.
A single language-conditioned SmolVLA policy replaces the per-task models and is post-trained with RECAP, then compiled to TensorRT and distilled so it fits the control cycle on the rover’s Jetson Orin. The workflow tool n8n is bridged to ROS 2, so an operator composes task sequences from topics, services and actions without writing code, and an on-device agent answers free-text questions by calling those same tools. Alongside the policy, a classical visual-servoing phase places the gripper where language cannot specify precisely enough.
The report covers all of this in detail. This page is for the parts a PDF cannot hold: a rover you can send somewhere, workflows you can open, and a policy you can argue with.
The part a PDF cannot hold
Live · read-only
A simulated rover offering real ROS 2 topics, services and actions, driven by eleven n8n workflows — and a read-only live view anyone can open. Send it to the lake from your phone, watch the battery watchdog preempt a patrol, or ask the agent what interfaces the robot has.
Runs in your browser
The policy family from contribution 01, solving PushT entirely in the browser: Chipmunk2D physics as WASM, inference on WebGPU. Drag the agent away and watch the policy recover; move the guidance weight and watch what it does — and does not — change.
On the Hugging Face Hub
Every episode the manipulator was taught, in LeRobot’s dataset viewer: camera streams, joint states and actions, scrubbable frame by frame. This is the raw material behind the policy — what a demonstration actually looks like, including the ones where the operator fumbled.
01
A single SmolVLA policy conditioned on a natural-language task description replaces the per-task ACT models of the previous project, extended with RECAP, an advantage-conditioned reinforcement-learning recipe, and SnapFlow-distilled for latency. The architecture is re-tasked by changing the instruction rather than by loading another model.
68.1 %
average LIBERO success, from 62.9 %
One full pass around the RECAP loop, rollout collection included. The rover lineage skips the rollout stage, so both of its RECAP stages train on demonstrations.
RQ1 · task automation and reinforcement learning
02
n8n is bridged to ROS 2 over rosbridge, giving operators a no-code way to compose task sequences from topics, services and actions, with the required order of operations enforced. An LLM agent built inside n8n answers free-text questions by calling those same tools — and a small vision-capable model, fine-tuned on recorded executions of the cloud-driven agent, runs it on board at no per-call cost.
93 %
tool calls on 43 held-out prompts
Against the cloud configuration’s 92 %. A few points is a prompt or two: this says fine-tuning closed the gap, not that the local model overtook the cloud one.
RQ1 · task automation, RQ3 · order enforcement and human-in-the-loop
03
The policy is compiled to TensorRT, split into a VLM prefix and a flow-matching action expert running at different precisions. Converting the whole graph to FP16 collapses the policy’s actions on trained weights; TensorRT’s reduced-mantissa TF32 mode for the prefix does not, and passes the numerical acceptance check with the action spread intact.
330 ms
per chunk, ≈3 Hz replanning
434 ms with the full ten-step sampler; 4.1× over the 1357 ms uncompiled baseline, leaving 21 ms of the 50 ms control cycle free.
RQ1 · enabling work
04
The 8×8 time-of-flight sensor in the gripper is routed into the policy’s state input through a single configuration entry: the state vector grows by those 64 cells and nothing else about the policy changes. Where no board markers are available, approach and manipulation both fall to the policy, and the grid supplies near-field information the cameras cannot.
64 cells
added to the state vector
Whether the policy is better for it was not measured. The grid is kept on the cost argument rather than a demonstrated gain; the ablation is future work.
RQ2 · marker-free depth-based control
05
Not every subtask is equally constrained by language. “Pick up the stone” names its target; “rotate the third switch from the left” needs deterministic output, which a VLM cannot provide. The classical approach phase exists for precise placement against repeated, ambiguous targets, and hands the policy a consistent starting pose in a known frame. P9 reimplements the whole perception and control chain in C++ for the margin that leaves on the onboard compute.
18.1 mm
median error, 25 mm required
On a registration solved from an earlier recording, as it is on the rover. Gripper markers are used only for the initial registration, so targets outside the camera’s field of view can still be aligned.
RQ2 · precise positioning
The layer underneath
Described last in the report as the supporting work it is, rather than as a contribution of its own.
The six-degree-of-freedom manipulator is driven with FZI’s Cartesian controllers. This work added a per-joint weighting to the solver that sets how much each axis contributes to a Cartesian motion, which damps the forearm roll so it no longer absorbs wrist rotations. A MuJoCo model of the manipulator was built and integrated with ROS 2 for smoke tests and as a target for developing the ArUco alignment; it was not used for policy training.
Both autonomous control modes are gated. A heartbeat from the operator station must be present, so a lost link or a crashed operator interface halts autonomous motion without anyone having to act, and a latching software e-stop aborts the running policy or alignment goal on demand and refuses new ones until it is reset. With the rover’s hardware stop and the cancellable phase actions, they form four layers of stopping that differ in what they leave running and how they resume.
End to end, on the arm
The pipeline is demonstrated end to end on the manipulator. The deployed policy turns a rotary panel switch on roughly four attempts in five — below a teleoperating human, but repeatable until it succeeds. Lever switches, whose motion first tensions and then snaps over, are harder and carry fewer demonstrations.
Still open
Plug insertion is the one substep still open, and for a reason worth stating plainly: an operator cannot teleoperate it well enough to demonstrate it, so there is nothing to imitate. A purpose-built tool now exists but has not yet been used to record it.
LIBERO rollouts
Evaluation episodes of the policies compared in the report. Filenames in the source encode policy, task, outcome and episode; these are three pairs where the difference is visible rather than statistical.
GOAL suite — “open the middle drawer of the cabinet”
The skill is present in the expert demonstrations and absent from the base policy’s own rollouts. Rollout data sharpens behaviour a policy already shows; behaviour it lacks has to come back from the demonstrations, which is what co-training mixes in.
LONG suite — one subgoal against two
The suite average is dragged down by instructions that chain two subgoals, because the error of each compounds. Single-subgoal tasks in the same suite reach 70–80 %.
SPATIAL suite — the same task, two post-training recipes
Co-training is not uniformly better. On this SPATIAL task the rollout-only policy succeeds where the co-trained one fails — a reminder that the suite averages in the report are averages, and that SPATIAL is the suite most sensitive to guidance.
From the report
A handful of figures that read well on a screen. The report has the rest, with the argument around them. Download the report.
