Action Recognition in the Browser¶
Why this project?¶
I wanted to investigate whether I could bring my research on epileptic seizure detection to an on-device application.
The motivation was privacy. Medical videos contain sensitive patient information, and running inference directly on the user's device would avoid having to send those videos to a remote server.
For this experiment, I decided to run X3D-M, the action-recognition architecture I had used during my research, in a web browser using ONNX Runtime Web.
There were two main questions I wanted to answer:
- Could I reproduce the original model's inputs and outputs in a browser environment?
- Could inference run fast enough to be useful for the intended medical application?
Preserving the Model's Inputs and Outputs¶
The original PyTorch model expects an input tensor of shape [1, 3, 16, 256, 256], corresponding to batch, channels, time, height, and width.
This means the model processes 16 frames at a time rather than individual images.
Most of the implementation work happened on the JavaScript side. I needed to maintain a 16-frame buffer and convert the captured frames into the tensor format expected by the model, reproducing the preprocessing used during export.
The model was exported to ONNX using FP32 precision.
To check that the conversion preserved its numerical behavior, I compared the outputs of PyTorch eager execution and ONNX Runtime. The maximum absolute difference was approximately 1e-6.
This gave me confidence that the exported model was producing numerically consistent outputs.
Why FPS Can Be Misleading¶
The browser implementation achieved approximately 0.76 inference windows per second on Firefox 150, running on a Linux x86_64 machine with 16 CPU cores.
Each inference window contains 16 frames, corresponding to approximately 12 effective FPS when counting the frames processed by each window.
However, these measurements describe different aspects of performance:
- 0.76 windows/s: the number of predictions completed per second.
- ~12 effective FPS: the number of frames represented by those inference windows.
- 1,304 ms: the mean inference latency for one window.
- 1,377 ms: the 95th percentile of the total processing time.
This distinction matters for temporal models.
Reporting only 12 FPS could give the impression that the application generates a prediction approximately every 83 ms. In reality, a new prediction takes around 1.3 seconds.
The application can process sequences of frames, but that doesn't mean it can produce predictions at the same frequency as a conventional real-time video pipeline.
Native vs. Browser Inference¶
I also measured inference performance outside the browser.
On CPU, PyTorch eager execution took approximately 270 ms per inference, while native ONNX Runtime took around 320 ms. Both measurements were collected over 20 runs.
The WebAssembly implementation was considerably slower, averaging 1,304 ms per inference window.
That performance difference is important, but raw inference speed wasn't the only objective.
Running the model directly in the browser removes the need for an inference backend. Camera frames can remain on the user's device instead of being uploaded to a server.
BrowserPose demonstrated that I could export X3D-M, preserve its numerical behavior, and execute it entirely in a browser.
The latency remains a limitation for applications requiring frequent predictions, but the experiment established that local, browser-based inference was technically possible without relying on a remote inference service.