A Failed Attempt at Streaming Model Loading¶
Why I started HotStart¶
I was interested in improving LLM inference on machines with limited CPU and GPU resources. One question I wanted to investigate was whether I could reduce time to first token (TTFT) by loading model weights asynchronously.
The idea was to avoid waiting for the entire model to load before starting inference. Instead, I wanted to load the first layers, start executing them, and continue loading the remaining layers in parallel.
I started with GPT-2 because I was working with a GTX 1080 Ti and wanted a model that would fit comfortably on my hardware while I experimented with the implementation.
This became HotStart, an experimental framework for streaming model inference.
Implementation¶
I separated the implementation into three components:
Loaderhandles SafeTensors files, parameter mapping, and model initialization.AsyncLoaderuses a thread pool to load model parameters concurrently and tracks which layers are ready.StreamingExecutorexecutes the model, waiting for layers to become available when necessary.
The loading process prioritizes embeddings and early transformer blocks so that execution can start before all the weights have been loaded.
In theory, loading later layers while executing earlier ones could reduce the time spent waiting for model initialization.
However, overlapping those operations only helps if the additional overhead is smaller than the time saved.
Benchmarking¶
I compared HotStart against conventional Hugging Face model loading using GPT-2.
The benchmark consisted of five runs for each approach.
| Measurement | Conventional loading | HotStart |
|---|---|---|
| Loading / setup time | 156.23 ms | 298.45 ms |
| Total time to first token | 781.46 ms | 2,246.78 ms |
Instead of improving TTFT, HotStart was approximately 2.88 times slower.
The implementation introduced significantly more overhead than the asynchronous loading strategy could compensate for.
Investigating the Regression¶
After seeing the results, I profiled the implementation to understand where the additional time was being spent.
Several problems emerged.
Concurrent File Access¶
My initial assumption was that loading model layers using multiple threads would accelerate the process.
However, the concurrent reads introduced contention and additional overhead in my implementation.
Using multiple threads did not translate into faster parameter loading.
Busy-Waiting¶
The executor repeatedly checked whether the next layer was available:
while not loader.is_ready(layer_idx):
loader.apply_ready()
time.sleep(0.001)
This introduced polling overhead while the executor waited for the weights it needed.
Parameter Loading and Resolution¶
The implementation also introduced additional work when transferring loaded tensors into model parameters.
tensor = file.get_tensor(key)
param.data.copy_(tensor)
Parameter resolution required dynamically traversing the model structure using string operations and attribute lookups.
These operations added overhead that conventional model loading could avoid or handle more efficiently.
The profiler helped identify these issues, although I did not separately benchmark the contribution of every component to the overall slowdown.
Where the Experiment Ended¶
HotStart did not achieve its original objective.
On GPT-2, the additional complexity of concurrent loading, synchronization, and parameter handling resulted in substantially worse startup performance than conventional loading.
The experiment was limited to a relatively small model. It does not establish whether a similar strategy could be beneficial for larger models or different storage configurations.
I stopped at this experimental implementation rather than continuing to optimize it.
The code and benchmark results are available in the HotStart repository.