Can Model Loading Overlap Inference?¶
Negative result
Motivation¶
Large language models cannot produce a token until enough weights are available. HotStart tested whether model loading and execution could overlap, allowing inference to begin before the complete model reached memory.
Problem¶
The expected benefit depended on I/O and compute overlapping cleanly. In practice, the loader, tensor format, Python coordination, and parameter installation all competed for resources.
Architecture¶
The experiment had three components: a synchronous loader, an asynchronous loader backed by ThreadPoolExecutor, and a streaming executor. Embeddings and early blocks were prioritized. The executor polled for layer readiness and advanced as soon as the required weights appeared.
Implementation¶
The streaming path and standard Hugging Face loading path were benchmarked under the same hardware conditions. The implementation used PyTorch 2.0, Transformers, and SafeTensors.
Results¶
Standard loading averaged 781 ms. The streaming implementation averaged 2,246 ms: approximately three times slower.
Lessons Learned¶
Four costs erased the intended overlap:
- contention while reading the same SafeTensors file;
- CPU cost from busy-wait polling;
- two copies along the file → memory → model path;
- parameter-resolution overhead of roughly 10–50 µs per parameter.
The larger lesson is that a plausible concurrency diagram is not evidence of overlap. Mature loading paths already contain optimizations that a Python-level scheduler can accidentally defeat.
Future Work¶
The negative result is the finished work. A new attempt would first need evidence that the storage layout and runtime can expose true asynchronous reads and avoid double copying; repeating the same threaded design would not address the measured causes.