Spaces:
Running
Running
| # User Story 009: Web Worker Architecture (Main-thread Safe Web Library) | |
| ## Story | |
| **As a** user building robotics UIs that also render live camera previews and interactive controls | |
| **I want** `@lerobot/web` to run heavy control/recording work off the main thread | |
| **So that** my UI stays smooth (no flicker/jank) even when teleoperation and recording are active | |
| ## Background | |
| The current browser implementation runs teleoperation control loops, dataset assembly, and export logic on the main thread. When activating keyboard teleoperation while previewing a camera stream, the preview can flicker due to main-thread contention. This is a UX blocker for real-world apps that combine live video, UI interactions, and hardware control. | |
| A worker-based architecture lets us move CPU-intensive, frequent, or bursty work off the main thread. The main thread remains responsible for DOM, video rendering and user interactions. The library must preserve the existing API (`calibrate()`, `teleoperate()`, `record()`) while transparently using workers when available, and cleanly falling back to the current approach otherwise. | |
| ## Goals | |
| - Identical public API to today’s `@lerobot/web` (no breaking changes) | |
| - Main-thread safe by default: heavy or frequent work executes in a Web Worker | |
| - Graceful fallback when workers or specific APIs aren’t available | |
| - Type-safe, minimal-copy message protocol using Transferables when possible | |
| - Strict library/demo separation: UI and storage remain in demos | |
| - Maintain Python lerobot UX parity and behavior | |
| ## Non-Goals (for this story) | |
| - Changing dataset formats or camera acquisition approach | |
| - Rewriting Web Serial API usage into worker (browser support is limited in workers) | |
| - Introducing new external dependencies | |
| ## Acceptance Criteria | |
| - Smooth UI under load: | |
| - With at least one active camera preview and keyboard teleoperation at 60–120 Hz, the preview does not flicker and UI remains responsive at ~60 FPS | |
| - API compatibility: | |
| - `calibrate()`, `teleoperate()`, `record()` signatures and return shapes are unchanged | |
| - Feature-detect workers; automatically use worker-backed runtime when available, otherwise use current main-thread runtime | |
| - Clear separation of responsibilities: | |
| - Worker executes control loops, interpolation, dataset assembly, export packaging, and CPU-heavy transforms | |
| - Main thread owns DOM/UI and browser-only APIs that are unavailable in workers (e.g., Web Serial write calls) | |
| - Type-safe protocol: | |
| - Strongly typed request/response messages with versioned `type` fields; Transferable payloads used for large data | |
| - Reliability & fallback: | |
| - If the worker crashes or becomes unavailable, operations fail gracefully with descriptive errors and suggest retry | |
| - Fallback path (main-thread) is automatically used when worker creation fails | |
| - Tests & docs: | |
| - Unit tests cover protocol routing and basic round-trips | |
| - Planning docs updated; README notes main-thread-safe architecture | |
| ## Architecture Overview | |
| ### Worker Boundaries | |
| - Execute in Worker: | |
| - Control loop scheduling and target computation for teleoperation (keyboard/direct and future teleoperators) | |
| - Episode/frame buffering and interpolation (regularization) for recording | |
| - Dataset assembly (tables/metadata), packaging (ZIP writer), and background export streaming | |
| - Lightweight telemetry aggregation for UI | |
| - Execute on Main Thread: | |
| - DOM, UI, and camera previews (`<video>` elements) | |
| - Web Serial API read/write bridge (if browser does not permit worker access) | |
| - MediaRecorder handling (browser-optimized implementation already off main CPU in many engines) | |
| ### Threading Model | |
| - Main thread spawns one worker per “process” instance as needed: | |
| - TeleoperationProcess → TeleopWorker | |
| - RecordProcess → RecordWorker (can be shared or composed with teleop worker depending on lifecycle) | |
| - The public process objects returned from `teleoperate()`/`record()` are proxies. Method calls post messages to the worker and return promises where appropriate. | |
| - SerialBridge (main-thread): worker requests motor write/read; main thread performs Web Serial operations and returns results. This preserves worker advantages while respecting browser API constraints. | |
| ### Message Protocol (Typed) | |
| All messages include a discriminant `type` and a `requestId` when a response is expected. | |
| - Teleoperation (examples): | |
| - `teleop/start`, `teleop/stop` | |
| - `teleop/update_key_state` { key, pressed } | |
| - `teleop/move_motor` { motorName, position } | |
| - `teleop/state_update` { motorConfigs, keyStates, lastUpdate } (worker → main) | |
| - `serial/write_position` { id, position } (worker → main) → `serial/ack` | |
| - Recording (examples): | |
| - `record/start`, `record/stop`, `record/next_episode` | |
| - `record/frame_append` { payload transferable } | |
| - `record/export_zip` { options } → streaming progress events | |
| - Error & lifecycle: | |
| - `worker/error`, `worker/ready`, `worker/teardown` | |
| Use Transferables (ArrayBuffer/MessagePort) for large payloads to avoid copies. | |
| ### File Structure (web package) | |
| ``` | |
| packages/web/src/ | |
| ├── workers/ | |
| │ ├── teleop.worker.ts # Teleoperation control loop | |
| │ ├── record.worker.ts # Recording assembly/export | |
| │ ├── protocol.ts # Message types & guards | |
| │ └── utils.worker.ts # Worker-side helpers (interpolation, zip) | |
| ├── bridges/ | |
| │ └── serial-bridge.ts # Main-thread serial proxy for workers | |
| ├── teleoperate.ts # Spawns worker, returns proxy process | |
| ├── record.ts # Spawns worker, returns proxy process | |
| └── types/ | |
| └── worker.ts # Public worker-related types (narrow) | |
| ``` | |
| ### Lifecycle & Fallback | |
| - On `teleoperate()`/`record()` call: | |
| - Try to instantiate corresponding worker via `new Worker(new URL(...), { type: 'module' })` | |
| - If success: wire protocol channels and return proxy-backed process | |
| - If fail: fall back to current main-thread implementation (no behavioral changes) | |
| - On `process.stop()` or page unload: send `worker/teardown` and terminate the worker | |
| ### Performance Notes | |
| - Control loop cadence generated inside worker to avoid main-thread timers | |
| - Batch serial commands from worker to main-thread bridge to minimize postMessage overhead | |
| - Use coarse-to-fine update: high-rate calculations in worker; lower-rate UI state updates to main thread (e.g., 10–20 Hz) for rendering | |
| - For export, stream chunks from worker; main thread triggers download or HF upload | |
| ### Error Handling | |
| - All request/response messages enforce timeouts with descriptive errors | |
| - Worker initialization guarded with feature detection and clear fallback | |
| - Protocol version field enables future evolution without breaking older callers | |
| ## Phased Implementation Plan | |
| ### Phase 1: Dataset & Export Offload (Low Risk) | |
| - Move episode interpolation, dataset assembly, and ZIP packaging to `record.worker.ts` | |
| - Main thread keeps MediaRecorder and camera preview as-is | |
| - Public API unchanged; verify ZIP download and HF upload via streamed messages | |
| ### Phase 2: Teleoperation Offload with SerialBridge | |
| - Move control loop scheduling and target computation to `teleop.worker.ts` | |
| - Implement SerialBridge on main thread for Web Serial commands | |
| - Worker posts motor write requests; main thread executes and responds | |
| - Throttle state updates to UI while maintaining high-rate control internally | |
| ### Phase 3: Fine-Grained Optimizations | |
| - Introduce Transferables for large buffers | |
| - Optional OffscreenCanvas pipelines for future video transforms (not required for current scope) | |
| - Tune batching and message cadence under hardware testing | |
| ### Phase 4: Reliability & Observability | |
| - Heartbeat messages and auto-restart policy for worker failures | |
| - Dev diagnostics toggles; production minimal logging | |
| ## Risks & Mitigations | |
| - Web Serial availability in workers: use main-thread SerialBridge (design accounts for this) | |
| - Message overhead at high Hz: batch commands and reduce UI state update frequency | |
| - Browser differences: feature-detect and test on Chromium, Firefox (where supported), Safari Technology Preview | |
| ## Definition of Done | |
| - UI remains smooth with active camera preview and keyboard teleoperation; no flicker observed in manual tests | |
| - Worker-backed runtime enabled by default when available; fallback path verified | |
| - `calibrate()`, `teleoperate()`, `record()` maintain identical signatures and behavior | |
| - Typed protocol implemented with Transferables where applicable | |
| - Unit tests for protocol routing and error timeouts | |
| - Documentation updated (this user story + README note) | |