VR Interface

The WebXR teleop app that streams Quest controller pose to the Axol SDK — now hosted in this repo under web/.

The Axol VR interface is a WebXR app that runs in a Meta Quest browser and streams hand/elbow pose from the headset to the Axol SDK over a secure WebSocket. It’s what you connect to during teleoperation and data collection.

The app’s source lives in this repository under web/ (it was previously the separate axol-vr repo). The same web/ build also produces the web control panel, and axol serve serves both.

Hosted vs. self-hosted

	URL	When to use
Hosted	axol.almond.bot	The easiest path — open it on the Quest, enter your host’s address, connect. Recommended for most users.
Self-hosted	https://<host>:8001/	Served from your own machine by axol serve, or by npm run dev for development. Use when iterating on the interface or running fully offline.

Either way, the headset connects to the VR WebSocket server on port 8000 that axol teleop (and the control panel’s teleop/collect operations) runs on the host.

On networks where Wi-Fi adds latency, tick Quest over USB on the connect screen to stream controller poses over a USB cable instead — the camera feeds keep using the LAN. See Quest over USB.

The WSS link is self-signed, so it needs a one-time approval in the Quest browser. When you press Connect and the certificate hasn’t been approved yet, the connection fails and an Authorize certificate button appears — tap it, proceed past the warning in the popup, and the app reconnects. (Equivalently, visit https://<hostname>.local:8000 or https://<local-ip>:8000 and proceed past the warning yourself.) The cert is cached in ~/.almond/vr/certs/ and shared with axol serve.

Repository layout

web/
├── app/                      # Vite + React WebXR app (and the control panel)
│   └── dist/                 # build output — served by `axol serve`
└── packages/
    └── axol-vr-client/       # reusable R3F components + hooks (@almond/axol-vr-client)

Build & develop

From web/:

npm install

# Production build — client package first, then the app (outputs web/app/dist,
# which axol serve picks up)
npm run build --workspace=packages/axol-vr-client
npm run build --workspace=app

# Local dev server with hot reload
npm run dev --workspace=app

For dev, open the printed localhost URL in the Quest browser, enter the hostname of the machine running the Axol SDK, and press Connect, then Start to enter the XR session.

Deployment

The hosted app is built on Vercel; vercel.json builds the client package first so it’s available as a local workspace dependency:

{
  "buildCommand": "npm run build --workspace=packages/axol-vr-client && npm run build --workspace=app",
  "outputDirectory": "app/dist",
  "installCommand": "rm -f package-lock.json && npm install"
}

Camera views

When the host has camera video enabled (axol teleop --cameras, collect-data, or the control panel with camera slots assigned), the robot’s camera feeds stream to the headset over WebRTC and render as screens over passthrough.

All streamed feeds are shown at once — the overhead centered (rendered per-eye in true stereo when the overhead is a stereo ZED X) with the left and right wrist cams as bottom-corner picture-in-picture panes. There’s no view to switch between; you arrange the screens to taste instead.

The screens are world-anchored — like TVs placed where you were looking when the session started, so you can move your head freely while they stay put. While arm tracking is disengaged you can rearrange them:

• Move a screen — point a controller at it and hold the trigger, then move the controller to reposition it. Each screen remembers its own spot.
• Resize a screen — grab the same screen with both triggers and move your hands apart to make it bigger or together to make it smaller.
• Reset — press B (right controller) to re-anchor every screen to your current gaze and clear all moves and resizes.

The headset only shows the feeds for cameras that are actually being streamed (set in the control panel’s Cameras settings tab or via the relevant CLI flags), so streaming only the overhead, say, shows just the overhead. While video is still negotiating — or any streamed camera hasn’t produced its first frame — a global Connecting cameras… indicator shows; it clears once every streamed camera is live, and never appears when nothing is being streamed.

Controller bindings

#	Button	Action
1	Left grip	From rest, press both grips (1 + 2) together to engage tracking. Once engaged, each grip toggles its own arm — click to freeze that arm in place (holding its pose and its grasp, e.g. to keep a held object steady) and click again to resume it. With the Hold grips to engage setting the grips are dead-man switches instead: hold both to start, release one to freeze that arm, keep holding to keep it going
2	Right grip	See above
3	Left trigger	Actuate left gripper. While tracking is disengaged: point at a camera screen and hold to move it — grabbing one screen with both triggers resizes it
4	Right trigger	Actuate right gripper. While tracking is disengaged: point at a camera screen and hold to move it — grabbing one screen with both triggers resizes it
5	X	Reset pose; cancels a recording countdown. While recording, arms the Discard episode? confirmation — press X again to discard and re-record, or A to cancel and keep recording
6	A	Record: start a take (3-second countdown). While recording, arms the Save episode? confirmation — press A again to save, or X to cancel and keep recording. Data collection only (no effect during plain teleop)
7	Y	Exit the XR session — sends a reset first, so the arms return to rest and disengage instead of holding the last pose and jerking on the next entry
—	Right B	Reset the camera screens — re-anchor to your current gaze and clear all moves + resizes
—	Left thumbstick	Drive the powered cart — push to move forward/back and strafe (robots with a powered cart only; deadman, so the base stops when released)
—	Right thumbstick (x)	Rotate the powered cart
—	Left thumbstick (click)	Lower the cart’s telescoping lift while held
—	Right thumbstick (click)	Raise the cart’s telescoping lift while held

Engaging and freezing arms. Tracking always starts with both grips together from rest. Once engaged, the grips act per arm: a click freezes only that arm — it holds its pose and keeps its gripper where it is, so it can steady a grasped object while the other arm keeps working — and another click resumes it. A forced disengage (a reset, a contact stop, or a dropped link) drops both arms and needs a deliberate both-grips re-engage. Prefer a dead-man feel? Turn on Hold grips to engage (the hold_to_engage setting, or --teleop.hold_to_engage) and each arm tracks only while you hold its grip.

The session’s mode is fixed by the command that launched it — there’s no in-headset toggle between the two. axol teleop runs plain teleop (recording is unavailable, so A does nothing and the HUD stays on Teleop), while axol collect-data runs data collection (recording enabled, HUD starts on Data Collection).

A dropped link never moves the robot. If the pose stream stops while tracking is engaged — you doff the headset, open the system menu, quit the app, or the link drops — the arms auto-disengage after about half a second and hold position wherever they are. When you return, re-engaging (both grips) takes a fresh snapshot, so the arms never jerk toward the controllers on the next entry; press X to send them back to the rest pose when you’re ready.

Powered cart

Robots equipped with the powered Axol Cart (an x-drive omni-wheel base plus a telescoping lift) are driven from the headset thumbsticks, so an operator can reposition the whole robot without leaving VR:

• Left thumbstick — drive the base: push to move forward/back and left/right (strafe).
• Right thumbstick (x-axis) — rotate the base.
• Stick clicks — press the left stick in to lower the lift, the right stick to raise it (each while held).

The sticks are their own deadman: the base only moves while a stick is deflected and coasts to a smooth stop the moment you let go, independent of whether arm tracking is engaged. Cart motion is operator mobility only — it’s never recorded into a dataset and a policy never controls it, so you can reposition the robot mid-session (during teleop or data collection) without touching the recorded data.

Enable the cart per-robot from the control panel’s Powered cart toggle (Robot settings) or with axol teleop --cart.enabled true; the tunables live under Advanced → Cart (or the --cart.* flags). To drive only the cart with the arms left untouched — no Axol hub needed — use the control panel’s Cart only checkbox or axol teleop --cart_only. See can.setup for wiring the cart’s own CAN adapter.

The telescoping lift needs a one-time axol lift.home calibration before it will move, and axol lift.goto raises it to the robot-install height; both are also on the panel’s diagnostics dashboard under Diagnostics.

State machine

In teleop mode the headset stays in Teleop with the recording controls inert. In data collection mode it starts in Data Collection and drives episodes with A / X:

DataCollection ──[A]──► (3s countdown) ──► Recording
      ▲                                          │
      │                             [A]=save · [X]=discard
      │                            (arms a confirm popup — press
      │                             the same button again to commit)
      │                                          │
      ├──────── Saving ◄──(server push)◄─── [A→A] save
      │                                          │
      └────────────────────────────────── [X→X] discard

During the countdown the state sent to the server stays DataCollection; it becomes Recording once the countdown completes.

Stopping a recording is confirmation-gated: the first A (save) or X (discard) press while recording arms an in-headset Save episode? / Discard episode? popup instead of stopping immediately — pressing the same button confirms, the other cancels and keeps recording. Nothing is committed server-side until you confirm a save; confirming a discard drops the take (via the reset flag) so you can re-record.

During data collection the HUD also shows an Episode N readout (top-right, under the recording status) so you can see which episode you’re about to record. The server announces it on connect — a headset joining mid-session shows the right number immediately — and it’s hidden in plain teleop. See the VR server API for the episode feedback message.

Saving and Error are server-driven. When recording stops, the SDK pushes {"type": "state", "value": "saving"} and blocks every control except Y (exit) until the episode is written, then pushes data_collection to re-enable controls. Broadcasting error shows an error indicator in the headset. See almond_axol.vr for the frame protocol and send_feedback_state().

Next steps

Teleoperation

Go from install to a live VR session.

VR server API

The WSS server, frame schema, and feedback messages.