docs/planning/009_web_worker.md

User Story 009: Web Worker Architecture (Main-thread Safe Web Library)

Story

As a user building robotics UIs that also render live camera previews and interactive controls I want @lerobot/web to run heavy control/recording work off the main thread So that my UI stays smooth (no flicker/jank) even when teleoperation and recording are active

Background

The current browser implementation runs teleoperation control loops, dataset assembly, and export logic on the main thread. When activating keyboard teleoperation while previewing a camera stream, the preview can flicker due to main-thread contention. This is a UX blocker for real-world apps that combine live video, UI interactions, and hardware control.

A worker-based architecture lets us move CPU-intensive, frequent, or bursty work off the main thread. The main thread remains responsible for DOM, video rendering and user interactions. The library must preserve the existing API (calibrate(), teleoperate(), record()) while transparently using workers when available, and cleanly falling back to the current approach otherwise.

Goals

• Identical public API to today’s @lerobot/web (no breaking changes)
• Main-thread safe by default: heavy or frequent work executes in a Web Worker
• Graceful fallback when workers or specific APIs aren’t available
• Type-safe, minimal-copy message protocol using Transferables when possible
• Strict library/demo separation: UI and storage remain in demos
• Maintain Python lerobot UX parity and behavior

Non-Goals (for this story)

• Changing dataset formats or camera acquisition approach
• Rewriting Web Serial API usage into worker (browser support is limited in workers)
• Introducing new external dependencies

Acceptance Criteria

• Smooth UI under load:
  • With at least one active camera preview and keyboard teleoperation at 60–120 Hz, the preview does not flicker and UI remains responsive at ~60 FPS
• API compatibility:
  • calibrate(), teleoperate(), record() signatures and return shapes are unchanged
  • Feature-detect workers; automatically use worker-backed runtime when available, otherwise use current main-thread runtime
• Clear separation of responsibilities:
  • Worker executes control loops, interpolation, dataset assembly, export packaging, and CPU-heavy transforms
  • Main thread owns DOM/UI and browser-only APIs that are unavailable in workers (e.g., Web Serial write calls)
• Type-safe protocol:
  • Strongly typed request/response messages with versioned type fields; Transferable payloads used for large data
• Reliability & fallback:
  • If the worker crashes or becomes unavailable, operations fail gracefully with descriptive errors and suggest retry
  • Fallback path (main-thread) is automatically used when worker creation fails
• Tests & docs:
  • Unit tests cover protocol routing and basic round-trips
  • Planning docs updated; README notes main-thread-safe architecture

Architecture Overview

Worker Boundaries

• Execute in Worker:
  • Control loop scheduling and target computation for teleoperation (keyboard/direct and future teleoperators)
  • Episode/frame buffering and interpolation (regularization) for recording
  • Dataset assembly (tables/metadata), packaging (ZIP writer), and background export streaming
  • Lightweight telemetry aggregation for UI
• Execute on Main Thread:
  • DOM, UI, and camera previews (<video> elements)
  • Web Serial API read/write bridge (if browser does not permit worker access)
  • MediaRecorder handling (browser-optimized implementation already off main CPU in many engines)

Threading Model

• Main thread spawns one worker per “process” instance as needed:
  • TeleoperationProcess → TeleopWorker
  • RecordProcess → RecordWorker (can be shared or composed with teleop worker depending on lifecycle)
• The public process objects returned from teleoperate()/record() are proxies. Method calls post messages to the worker and return promises where appropriate.
• SerialBridge (main-thread): worker requests motor write/read; main thread performs Web Serial operations and returns results. This preserves worker advantages while respecting browser API constraints.

Message Protocol (Typed)

All messages include a discriminant type and a requestId when a response is expected.

• Teleoperation (examples):
  • teleop/start, teleop/stop
  • teleop/update_key_state { key, pressed }
  • teleop/move_motor { motorName, position }
  • teleop/state_update { motorConfigs, keyStates, lastUpdate } (worker → main)
  • serial/write_position { id, position } (worker → main) → serial/ack
• Recording (examples):
  • record/start, record/stop, record/next_episode
  • record/frame_append { payload transferable }
  • record/export_zip { options } → streaming progress events
• Error & lifecycle:
  • worker/error, worker/ready, worker/teardown

Use Transferables (ArrayBuffer/MessagePort) for large payloads to avoid copies.

File Structure (web package)

packages/web/src/
├── workers/
│   ├── teleop.worker.ts             # Teleoperation control loop
│   ├── record.worker.ts             # Recording assembly/export
│   ├── protocol.ts                  # Message types & guards
│   └── utils.worker.ts              # Worker-side helpers (interpolation, zip)
├── bridges/
│   └── serial-bridge.ts             # Main-thread serial proxy for workers
├── teleoperate.ts                   # Spawns worker, returns proxy process
├── record.ts                        # Spawns worker, returns proxy process
└── types/
    └── worker.ts                    # Public worker-related types (narrow)

Lifecycle & Fallback

• On teleoperate()/record() call:
  • Try to instantiate corresponding worker via new Worker(new URL(...), { type: 'module' })
  • If success: wire protocol channels and return proxy-backed process
  • If fail: fall back to current main-thread implementation (no behavioral changes)
• On process.stop() or page unload: send worker/teardown and terminate the worker

Performance Notes

• Control loop cadence generated inside worker to avoid main-thread timers
• Batch serial commands from worker to main-thread bridge to minimize postMessage overhead
• Use coarse-to-fine update: high-rate calculations in worker; lower-rate UI state updates to main thread (e.g., 10–20 Hz) for rendering
• For export, stream chunks from worker; main thread triggers download or HF upload

Error Handling

• All request/response messages enforce timeouts with descriptive errors
• Worker initialization guarded with feature detection and clear fallback
• Protocol version field enables future evolution without breaking older callers

Phased Implementation Plan

Phase 1: Dataset & Export Offload (Low Risk)

• Move episode interpolation, dataset assembly, and ZIP packaging to record.worker.ts
• Main thread keeps MediaRecorder and camera preview as-is
• Public API unchanged; verify ZIP download and HF upload via streamed messages

Phase 2: Teleoperation Offload with SerialBridge

• Move control loop scheduling and target computation to teleop.worker.ts
• Implement SerialBridge on main thread for Web Serial commands
• Worker posts motor write requests; main thread executes and responds
• Throttle state updates to UI while maintaining high-rate control internally

Phase 3: Fine-Grained Optimizations

• Introduce Transferables for large buffers
• Optional OffscreenCanvas pipelines for future video transforms (not required for current scope)
• Tune batching and message cadence under hardware testing

Phase 4: Reliability & Observability

• Heartbeat messages and auto-restart policy for worker failures
• Dev diagnostics toggles; production minimal logging

Risks & Mitigations

• Web Serial availability in workers: use main-thread SerialBridge (design accounts for this)
• Message overhead at high Hz: batch commands and reduce UI state update frequency
• Browser differences: feature-detect and test on Chromium, Firefox (where supported), Safari Technology Preview

Definition of Done

• UI remains smooth with active camera preview and keyboard teleoperation; no flicker observed in manual tests
• Worker-backed runtime enabled by default when available; fallback path verified
• calibrate(), teleoperate(), record() maintain identical signatures and behavior
• Typed protocol implemented with Transferables where applicable
• Unit tests for protocol routing and error timeouts
• Documentation updated (this user story + README note)