Action Recognition in the Browser¶
Moving X3D-M into a browser changed more than the runtime. A temporal model forces precise definitions for inputs, throughput, and parity.
Preserve the Model Boundary¶
The PyTorch model consumes [1, 3, 16, 256, 256]: batch, channels, time, height, and width. The browser therefore owns a 16-frame buffer and must reproduce the export-time preprocessing before ONNX Runtime Web receives a tensor.
The model was exported at FP32. PyTorch eager and ONNX Runtime agreed to a maximum absolute difference of roughly 1e-6; all 16 parity checks agreed on top-1 and top-5 predictions. That check separates runtime performance from accidental model changes.
“Frames per Second” Is Not Enough¶
The measured browser path completed about 0.76 inference windows per second on Firefox 150, Linux x86_64, and a 16-core machine. Each window contains 16 frames, producing roughly 12 effective FPS.
Those values describe different things:
- 0.76 windows/s describes how often a new prediction completes;
- 12 effective FPS describes how many source frames are consumed by those windows;
- 1,304 ms is the mean inference latency for one window;
- 1,377 ms is the p95 total time.
Calling the system “12 FPS” without the other values would suggest a prediction every 83 ms, which the implementation does not provide.
Native and Browser Baselines¶
On CPU, PyTorch eager took approximately 270 ms and ONNX Runtime approximately 320 ms over 20 runs. The WASM browser path was slower, but it removed the backend and kept camera data local.
That is the actual trade-off. BrowserPose is not a claim that WebAssembly beats the native runtime; it shows that a substantial temporal model can operate inside a static, privacy-preserving application with measured behavior.