HotStart — Experimental Streaming LLM Inference¶
Negative result
Overview¶
HotStart is an experiment I developed to investigate whether loading model weights concurrently with inference could reduce time to first token (TTFT).
I was interested in improving LLM inference on machines with limited CPU and GPU resources. The idea was to start executing the first transformer layers while the remaining weights were still being loaded.
I used GPT-2 as my initial test model because it fit comfortably on my GTX 1080 Ti.
Implementation¶
I implemented three components:
Loader: reads model weights from SafeTensors files and initializes the model structure.AsyncLoader: uses a thread pool to load layers concurrently and track their availability.StreamingExecutor: executes the model while waiting for the required layers to become available.
The loading process prioritizes embeddings and early transformer blocks so that inference can begin before the entire model is loaded.
I benchmarked this implementation against conventional Hugging Face model loading and profiled the streaming path to investigate its overhead.
Results¶
The experiment produced the opposite of the expected result.
| Measurement | Standard loading | HotStart |
|---|---|---|
| Average time to first token | 781 ms | 2,246 ms |
On GPT-2, across five benchmark runs per approach, HotStart was approximately 2.88 times slower than conventional loading.
Profiling identified several sources of overhead, including concurrent file access, readiness polling, tensor copying, and dynamic parameter resolution.
These issues prevented the implementation from achieving the intended reduction in startup latency. Their individual contributions to the regression were not separately quantified.
HotStart remains a completed experiment with a negative result.
For a detailed explanation of the implementation, profiling observations, and limitations, see A Failed Attempt at Streaming Model Loading.