almond_axol.vr

Secure WebSocket server that receives VRFrame messages from the VR app.

Secure WebSocket server (WSS) that receives VRFrame JSON messages from the VR app. A self-signed TLS certificate is auto-generated in ~/.almond/vr/certs/ on first use. This module can be used standalone to read raw VR data without full teleoperation — useful for custom control loops or data collection.

from almond_axol.vr import VRServer, VRServerConfig

import asyncio
from almond_axol.vr import VRServer, VRServerConfig

async def main():
    async with VRServer(VRServerConfig(port=8000)) as vr:
        while True:
            frame = vr.get_frame()
            if frame is not None:
                print(frame.l_ee, frame.r_ee)
            await asyncio.sleep(0.01)

asyncio.run(main())

Or use a callback instead of polling:

def on_frame(frame):
    print(frame.l_grip, frame.r_grip)

async with VRServer() as vr:
    vr.set_on_frame(on_frame)
    await asyncio.sleep(float("inf"))

VRFrame fields

Field	Type	Description
l_ee / r_ee	VRPose	6-DOF end-effector pose (position + quaternion)
l_elbow / r_elbow	VRPosition	3D elbow positions
l_grip / r_grip	float [0, 1]	Gripper commands
l_lock / r_lock	bool	Grip button states. Both together (from rest) engage tracking; once engaged each toggles its own arm between tracking and frozen — or, with hold_to_engage, holds it while pressed
l_tracked / r_tracked	bool	Whether each controller’s position is optically tracked. False means the headset is dead-reckoning it (WebXR emulatedPosition — occluded / out of FOV / fast whip), so the pose smoother excludes those frames and holds the last clean pose. Defaults to True, so older web builds (which omit the field) keep the previous always-trusted behaviour
l_trigger_live / r_trigger_live	bool	Whether each managed Mantis CAN trigger node is still publishing fresh input. A tracker bridge sets this false on heartbeat loss while holding the last safe command; collection quality checks reject the affected take. Quest/legacy clients default to True.
reset	bool	Rising edge triggers a reset move
state	VRState	TELEOP, DATA_COLLECTION, or RECORDING (headset-driven); SAVING and ERROR are server-pushed only
episode_outcome	VREpisodeOutcome | None	One-frame success / failure pulse from a managed trigger gesture. Collection uses it to choose its stop path and logs failure, but the current LeRobot rows do not persist an outcome label.
episode_end_t_host	float | None	Host perf_counter instant where a saved episode’s data should end, sent alongside a successful episode_outcome. A local tracker bridge sets it to the moment the end gesture began so the collector trims the rows captured during the trigger clicks. None keeps every row (Quest / older clients).
lock_release_id	int | None	Internal managed-tracker handshake: on an all-locks-false frame the teleop core echoes this id once it has consumed the release, so the bridge can wait for a real re-engage edge even while IK is blocked.
t	float | None	Client capture timestamp in ms (performance.now()), used by the pose interpolator. None falls back to latest-wins
seq	int | None	Monotonic frame counter set by a producer. Together with pose_source_id, it de-duplicates identical USB/WebRTC/network copies globally; the first delivered copy wins. None disables cross-transport de-duplication.
t_host	float | None	Server-stamped estimate of when these poses were captured on the host clock (perf_counter), derived from t and the headset↔host clock offset. Dataset rows align to this when the pose stream is the ground truth (Mantis recording). None until the interpolator has seen the frame.
pose_source_id	str | None	Stable logical producer id shared across every transport and reconnect for one Quest/bridge. This is the scope of sequence de-duplication and exclusive pose ownership. Older clients fall back to a per-connection legacy id.
pose_source_kind	"webxr" | "tracker" | None	Source class used by managed Mantis policy: Quest is webxr; Lighthouse/Ultimate bridges are tracker. Non-owning clients remain connected to receive video/HUD/URDF state, but their pose frames and HUD controls are ignored.
l_pose_profile / r_pose_profile	str | None	Most-preferred WebXR input profile for each Quest controller (for example oculus-touch-v3). Production Quest calibration is scoped to this value.
l_pose_space / r_pose_space	"grip" | "target-ray" | None	Datum used for each Quest pose. Production Mantis requires the calibrated gripSpace; target-ray is compatibility bring-up only.
l_stick_x / l_stick_y	float [-1, 1]	Left thumbstick axes (right = +1; pushed forward = -1). Drive Jelly’s translation when one is configured
r_stick_x	float [-1, 1]	Right thumbstick x (right = +1). Drives Jelly’s rotation
r_stick_y	float [-1, 1]	Right thumbstick y (pushed forward = -1). Only read in box mode (forward/back moves the elbows, box_elbow_out); Jelly ignores it
l_stick_click / r_stick_click	bool	Thumbstick press-in — left lowers Jelly’s lift, right raises it; while a box-mode leader is engaged a single click-and-release of either stick toggles the grasp (box_grasp: flush face / fingers straight; the sticks themselves set grip width and tilt, no modifiers); with the pair frozen the sticks drive Jelly as usual. Both pressed together (within ~0.35 s) toggle box mode (the headset sends the set message below). Neutral defaults keep older web builds (which omit these fields) compatible
box_leader	"left" | "right" | None	Set by VRTeleopCore on the frame it forwards to the IK worker, not by the headset: which controller currently leads the pair in box mode
reengage	"clutch" | "ramp" | None	Set by VRTeleopCore on the frame it forwards to the IK worker, not by the headset: the session’s effective re-engage behaviour. Session modes are switched out of band with the set message, not through frames

Pose transports & USB/Wi-Fi fallback

The headset can stream pose over more than one transport at once — the network WebSocket, a USB adb reverse tunnel (Quest over USB), and a dedicated WebRTC data channel (remote teleop). Every copy carries the same pose_source_id and seq; the server keeps a monotonic high-water mark for that logical producer and processes each frame exactly once, from whichever link delivers it first. A cable drop therefore falls through to the next Wi-Fi copy with no reconnect, while a delayed standby copy can never rewind controls or enter the jitter buffer. The producer state resets after its last carrying client disconnects and on disable(), so a reloaded source with a restarted counter is accepted.

Only one logical producer owns poses at a time. Managed Quest Mantis sets VRServerConfig.pose_source_kind="webxr"; managed Lighthouse/Ultimate sets it to "tracker" and gives both the owned bridge and server a fresh expected_pose_source_id for that operation. A connected source of another kind or ID stays available for signaling, camera video, server-driven HUD, and URDF state but its VRFrames and client-originated HUD controls are view-only and ignored, so a forgotten standalone bridge or connected Quest cannot claim or alter a managed run. Lighthouse/Ultimate worlds are not automatically registered to that viewer’s Quest local-floor world, so their URDF state carries viewer_world_aligned=false and the client hides the spatial overlay by default. On an unrestricted compatibility server the first producer claims control, except that a tracker bridge may supersede WebXR; once a tracker has claimed it, the server does not silently fail over to a still-connected Quest if the bridge dies.

The WebRTC pose data channel (almond_axol.vr.control_channel) shares the same ICE/TURN path as the camera video, so an operator off your tailnet can drive over a relay. ICE servers are read from the environment (AXOL_TURN_URL / AXOL_TURN_USERNAME / AXOL_TURN_PASSWORD; see almond_axol.vr.ice) — on a LAN or plain tailnet they’re unset and the direct path is used. If the data channel fails to negotiate (or the peer connection drops), the headset re-requests it after a backoff rather than permanently falling back to the WebSocket.

Server → headset feedback

The server can push a state override to all connected headsets at any time using VRServer.broadcast_text(). The headset interprets messages of the form {"type": "state", "value": "saving"} as a state override that blocks recording controls. {"type": "state", "value": "error"} shows an error indicator in the headset UI. {"type": "state", "value": "data_collection"} re-enables controls after saving. The AxolVRTeleop.send_feedback_state(state) helper wraps this for all VRState values.

On connect the server also announces its operating mode — {"type": "mode", "value": "teleop" | "data_collection"} — set via VRServer.set_mode(). The web UI uses it to lock the HUD to a single mode: axol teleop sends teleop (recording controls hidden), while axol collect-data sends data_collection (recording enabled, but no switching back to plain teleop). A separate replayable {"type": "pose_source_kind", "value": "webxr" | "tracker" | null} announcement makes a Quest’s A/X recording controls view-only in tracker-owned Lighthouse/Ultimate sessions; the server independently rejects HUD control messages from any socket that is not carrying the active WebXR pose source. These messages are pushed once per client in the WebSocket accept handler, so a headset that joins mid-session still gets them.

During data collection the server likewise pushes the current episode number — {"type": "episode", "value": N} (1-based) — which the headset renders as an Episode N HUD readout. axol collect-data announces it at the start of each episode via AxolVRTeleop.send_feedback_episode(episode), which both stores the value on the server (VRServer.set_episode(), re-sent on connect so a mid-session headset shows the right number) and broadcasts a live update to connected clients. Plain teleop never sets it, so the readout stays hidden.

Teleop additionally broadcasts {"type": "tracking", "value": true|false} whenever the engage toggle changes, so the headset knows when the robot is being controlled — it uses this to only allow repositioning and resizing the camera screens (a trigger gesture) while tracking is disengaged.

Teleop also streams the arm joint state at ~20 Hz — {"type": "joints", "value": {"q": {"<urdf joint>": rad, ...}, "l_grip": 0..1, "r_grip": 0..1, "engaged": bool, "pair": {"aligned": bool, "width": m, "tilt": deg, "grasp": str} | null}} (measured positions when the robot reports them, else the commanded targets) — for the headset HUD. pair is the gripper-pair geometry the IK worker reports a few times a second (IKWorker.pair_status()): aligned is true when the two grippers already form the box-mode pair — level, the fitted tool’s contact face toward each other across a box-mode-sized gap — the HUD’s Box button shows (aligned) then, since switching to box mode from that pose costs no alignment blend; width is the grip width (the separation the tool’s contact faces would have with the mounts where they are), tilt the pair’s inward tilt trim in degrees and grasp ("flush" / "straight") the current grasp, both shown on the same button while the mode is on. The URDF and its meshes are served from the same origin under /urdf/ (axol.urdf, meshes/*.stl).

Live session settings

The knobs an operator can turn mid-session — box mode, re-engage behaviour, hold-to-engage, grip force, reach scale, arm speed — are exposed as live settings (almond_axol.teleop.live.LiveSettings) on the same socket, from any client (headset HUD, control panel, SDK):

• client → server {"type": "set", "key": "<key>", "value": <value>} — handled by the callback installed with VRServer.set_on_setting(). Values are validated (type, range, choices); a rejected request is logged and simply not echoed. A boolean key also accepts "value": "toggle", flipped against the server’s current value — what the headset’s both-sticks gesture sends, so a gesture repeated before the echo lands can’t re-send a stale state.
• client → server {"type": "session-config-request"} — re-sends that client the connect-time session config (mode, every set_announce entry such as settings, pose_source_kind, pose_mode, tracking, episode, hud). The web client sends it once its message listeners are installed, since those come up a render after the socket opens and the connect-time copy can land before anyone is listening (a headset that missed settings would hide every live control, box mode included).
• server → clients {"type": "settings", "value": {"schema": [...], "values": {"<key>": ...}}} — sent on connect (set_announce("settings", ...)), on session-config-request, and after every set, including no-ops, so all clients converge on the server’s state. schema is a list of {key, label, type: "boolean"|"select"|"number", help, options, min, max, step, unit} entries; both UIs render their controls from it, so a new knob is one entry in LIVE_SETTINGS.

Key	Type	Applies
box_mode	boolean	VRTeleopCore.set_box_mode() — switching disengages the arms first
box_grasp	"straight" / "flush"	fingers straight forward (the default) vs. the tool’s contact face along the box side; a live pair blends over to it. Also toggled by a stick click, which the worker reports back so the value stays in sync
box_flush_deg	number, °	how far the angled (flush) grasp turns each gripper inward from the parallel grasp, about the parcel plate’s front edge (39°); read at the next engage or grasp switch
box_face_left / box_face_right	"auto" / "+x" / "-x"	which flat side of each gripper faces the box; next engage
reengage	"clutch" / "ramp"	VRTeleopCore.set_reengage() — next grip edge
hold_to_engage	boolean	grips as dead-man switches vs. toggles
gripper_torque	number, Nm	both grippers’ torque_limit (read on every gripper command). Hardware only — hidden in sim
position_multiplier	number, ×	hand-motion scale (forwarded to the IK worker)
teleop_max_vel	number, rev/s	joint velocity cap while tracking

The box_* keys exist only when the robot has the parcel gripper (VRTeleopConfig.gripper == "parcel", adopted from AxolConfig.gripper): with the parallel gripper they are left out of both schema and values and a set of one is rejected, so clients show box mode only when box_mode is in values.

Core-side settings are queued (VRTeleopCore.set_live()) and applied on the IK thread before the next frame — or while idle, so the panel can change them with no headset connected; IK-worker fields are mirrored to the worker subprocess as ("set", key, value) messages.

The server also relays HUD state between clients. The active WebXR pose client publishes its transient HUD — an armed save/discard confirmation popup and the record-start countdown — as {"type": "hud", "value": {...}}; the server stores the latest value, forwards it to every other connected client, re-sends it to one that joins mid-dialog, and clears it with a null broadcast when the publisher disconnects. Signaling-only viewers and Quests attached to tracker-owned runs cannot publish this state. The web control panel uses the relay to mirror the in-headset popups so a session can be followed from the dashboard.

Camera video to the headset

The server can stream camera frames to the headset over WebRTC. Register per-camera sources with set_video_sources(). A source is either a pre-encoded camera (the GPU-resident gst_zed pipeline, exposing subscribe() / alive — its H.264 access units go straight to RTP with no Python encode) or a raw frame source. A connected ZedCamera (or stereo eye) is registered directly — the relay adapts it — and you can equally pass any object exposing fixed width / height / fps plus wait_next(after_ts, timeout_ms). The manager picks the right WebRTC track per source automatically:

vr.set_video_sources({
    "overhead":  overhead_cam,
    "left_arm":  left_cam,
    "right_arm": right_cam,
})

The existing /ws channel is reused for SDP signaling (no extra ports): the headset sends webrtc-request, the server replies with a webrtc-offer carrying one video track per source (labelled by camera name), and the headset answers with webrtc-answer. A stereo overhead streams both eyes packed side-by-side in one track, overhead_sbs — a single decoder session on the headset, since two full-res sessions cap the Quest’s render rate — which the VR app renders per-lens through a WebXR media layer; the SDK-fallback path (no gst) instead registers the two per-eye sources overhead_left / overhead_right. A stereo wrist is registered as a single source under its slot name (one feed, like a mono camera). Dataset recording is unaffected — it still stores the two per-eye observation keys.

Pass None or {} to turn video off. axol teleop --cameras and collect-data wire this up automatically.

Pipeline

Teleop and data collection share one GPU-resident camera path (almond_axol.video.gst_zed). The zed-gstreamer source elements (zedxonesrc mono / zedsrc stereo) grab frames straight into NVMM (GPU) memory, and a single in-process GStreamer pipeline tees that one zero-copy buffer to two consumers:

• a headset branch — fixed at 30 fps, nvv4l2h264enc (NVENC) → appsink — so the whole grab→encode chain stays on the GPU and Python only sees H.264 access units, which aiortc forwards as pre-encoded packets;
• a dataset branch — independently rate-limited to the requested recording fps, downscaled by the VIC, and NVENC-encoded. gdppay preserves every access unit’s sensor-exposure PTS through native shmsink/shmsrc, so the recorder can pair it with the nearest timestamped robot state and mux it without a raw-frame copy or re-encode;
• a raw policy/fallback branch — nvvidconv converts to RGBA for an in-process appsink or Python shared-memory fallback. Its exposure PTS is likewise mapped to time.perf_counter.

Both teleop and data collection run the camera grab/encode/WebRTC pump in a dedicated subprocess (almond_axol.video.video_proc.VideoRelayProcess, registered via set_video_manager()). Teleop builds only the headset branch. Data collection adds the independently paced dataset branch and a second recorder subprocess that owns the LeRobotDataset. The control loop publishes joint/action snapshots into a bounded shared-memory history; the recorder selects the entry nearest the median camera exposure. Missing timestamps, stale/repeated frames, excessive camera/state skew, or broken predictive-video continuity discard the episode rather than silently saving a mismatched row. time.perf_counter has a process-shared monotonic origin, so relay PTS and control snapshots are directly comparable. If native GDP/shared-memory support is absent, collection uses the strict raw-frame fallback; if the timestamp-patched ZED plugin binary cannot be verified, it falls back to the ZED SDK.

Requires the GPU-resident camera stack the host installer sets up once: the GStreamer + PyGObject support from axol gst.install plus the patched zed-gstreamer source plugins from axol gst.build-zed (the NVENC elements themselves ship with the Jetson L4T BSP). The NVENC/VIC engine clocks must also be pinned to their maximum for low-latency encode (axol provision does it — see jetson.setup — and the installer’s systemd unit re-applies it at boot); the default Jetson devfreq governor scales them for throughput, which otherwise roughly triples per-frame encode latency.

VRServerConfig fields

Field	Default	Description
port	8000	WSS listen port
certfile	auto	Path to TLS certificate; None uses the auto-generated cert
keyfile	auto	Path to TLS private key
pose_source_kind	None	Optional exclusive source policy: "webxr" for Quest Mantis or "tracker" for Lighthouse/Ultimate. Other clients remain connected as view-only consumers. None retains compatibility ownership rules.
expected_pose_source_id	None	Optional exact logical producer ID. Managed Lighthouse/Ultimate fills this with a one-run token; other matching-kind bridges remain view-only.
interp_enabled	True	Enable the adaptive playout buffer that interpolates pose frames using the headset timestamp t, smoothing jittery / batched frame arrivals
interp_min_delay_s	0.0	Minimum playout delay in seconds
interp_max_delay_s	0.15	Maximum playout delay in seconds — caps how much latency the adaptive jitter buffer may add (spent only while delivery is actually bursty)
interp_smooth_window_s	0.12	Width of the Gaussian fixed-lag smoothing window rendered around the playout point. Adds a fixed window / 2 of latency in exchange for zero-phase smoothing and tracking-glitch rejection; 0 disables it (plain two-frame lerp)
interp_outlier_k	4.0	Hampel outlier threshold (in robust standard deviations) for the glitch rejection inside the smoothing window. Lower is more aggressive; <= 0 disables rejection
interp_max_lag_s	0.5	Late-frame gate: a pose frame arriving more than this many seconds later than the stream’s normal transit (the minimum over the last 30 s) keeps its control state (locks, reset) but is not played, so after a network stall the arms hold and then follow the current pose instead of replaying the delayed motion. <= 0 disables the gate

Before opening the VR app, accept the self-signed certificate by navigating to https://<hostname>.local:8000 in the VR browser and proceeding past the security warning.