Skip to content

Can Action Recognition Run Entirely in a Browser?

Completed

BrowserPose · X3D-M · ONNX Runtime Web · WebAssembly

Motivation

Action-recognition demonstrations usually send video to a server. That adds infrastructure and turns a local camera stream into uploaded data. BrowserPose tests the opposite boundary: export once, then perform all inference on the visitor's machine.

Problem

X3D-M consumes a temporal tensor rather than a single image. A browser implementation therefore had to preserve preprocessing and temporal semantics, fit the model into an ONNX-compatible graph, and make performance understandable at the level of windows and effective frames.

Architecture

PyTorch is used only for export. The runtime is a static web application:

  1. browser video APIs provide frames;
  2. a 16-frame buffer constructs the temporal input;
  3. preprocessing produces a [1, 3, 16, 256, 256] tensor;
  4. ONNX Runtime Web executes X3D-M through its WASM execution provider;
  5. a softmax wrapper and labels.json turn logits into readable predictions.

There is no inference server and no video upload.

Implementation

The model was exported from PyTorchVideo at FP32. The browser path uses vanilla JavaScript and the HTML5 APIs. Numerical parity was checked against PyTorch eager execution before browser performance was interpreted.

Results

Firefox 150 on Linux x86_64 with 16 CPU cores sustained approximately 0.76 windows per second. With 16 frames per window, that is roughly 12 effective frames per second. Mean browser inference was 1,304 ms per window and p95 total time was 1,377 ms.

Native CPU references were approximately 270 ms for PyTorch eager and 320 ms for ONNX Runtime CPU over 20 runs. Export parity reached a maximum absolute difference of about 1e-6, with 16/16 top-1 and top-5 agreement.

Lessons Learned

“FPS” is ambiguous for temporal models. Reporting windows per second, frames per window, and latency per window makes the browser result reproducible instead of overstating interactivity.

The export boundary deserves its own test. Matching labels is not enough when small graph changes can alter ordering or preprocessing; numerical parity established that the runtime comparison concerned execution rather than a changed model.

Future Work

No further implementation is recorded. The completed version establishes a measured WASM baseline for future browser execution providers or quantized exports.

Action recognition in the browser records the measurement choices in more detail.