Making Clinical Video Evaluation 10× Faster¶
The problem¶
During my PhD, I was working on epileptic seizure detection using video action-recognition models. The dataset contained clinical recordings collected from multiple hospitals, with some videos lasting up to an hour.
Processing this amount of video was taking too long. My evaluation pipeline, which relied on OpenCV for video decoding and PyTorch for inference, was processing approximately 6 video frames per second.
I needed to improve the processing speed to make experiments on the dataset more practical.
Finding the bottleneck¶
I started by profiling the pipeline using nvtop and PyTorch Profiler.
Initially, I was looking at the inference performance, but the profiling showed that a significant amount of time was spent in preprocessing rather than GPU execution.
This made video loading one of the main areas I wanted to improve.
The problem was also related to memory. Some recordings were up to an hour long, making it impractical to load an entire video into memory before processing it.
I needed to read smaller portions of each recording while keeping the pipeline efficient.
Moving to PyAV and TensorRT¶
I decided to replace OpenCV video decoding with PyAV.
At the time, I was familiar with the video-loading utilities in the PyTorch ecosystem, particularly TorchVision, which used PyAV. I wanted similar performance and ease of use, but with more control over how videos were loaded.
The main difficulty was getting PyAV to extract arbitrary portions of a video dynamically. I needed to divide long recordings into smaller chunks without loading the entire file.
It took some experimentation to get this behavior working as intended.
On the inference side, I exported X3D-M from PyTorch to ONNX and then TensorRT, using FP16 precision.
I chose TensorRT because the evaluation pipeline was running on an NVIDIA GPU, and I wanted to use its optimized inference runtime.
Unlike the video-loading implementation, the model export was straightforward and didn't require significant troubleshooting.
Results¶
After integrating PyAV and TensorRT, the complete evaluation pipeline was approximately 10 times faster than the original implementation.
The benchmark used an NVIDIA RTX 3090 with X3D-M and a batch size of 128.
The improvement was measured end-to-end, including video loading, preprocessing, and model inference.
I didn't separately benchmark the contribution of PyAV and TensorRT, so I can't attribute the 10× speedup to either component individually.
The optimized pipeline made it much more practical to process long clinical recordings and run evaluations across the dataset.
What the measurements tell us¶
The profiling showed why focusing exclusively on GPU inference would have been insufficient.
Even with a faster inference engine, the pipeline still needed to prepare and deliver video frames efficiently.
In this experiment, improving video loading and GPU inference together produced a substantial reduction in evaluation time.
The 10× figure describes the combined improvement of those changes, not the performance of TensorRT in isolation.