A Failed Attempt at Streaming Model Loading¶
The hypothesis behind HotStart was simple: if early transformer blocks are available before later blocks, execution might begin while the remaining weights are still loading. Better overlap should reduce time-to-first-token.
It did not.
The Experiment¶
The prototype separated loading from execution. A ThreadPoolExecutor loaded tensors asynchronously, prioritizing embeddings and early blocks. A streaming executor waited for each required layer and advanced when it became available. The baseline used the ordinary Hugging Face loading path on identical hardware.
The standard path averaged 781 ms. The streaming path averaged 2,246 ms.
That is not a marginal regression. It is a design that made the target metric about three times worse.
Where the Time Went¶
File contention¶
SafeTensors access did not become independent just because reads came from different worker tasks. Concurrent activity against the same file created contention instead of clean I/O overlap.
Polling¶
The executor used busy-wait checks for layer readiness. Those checks consumed CPU while useful work was already competing for it.
Copies¶
The data moved from file to host memory and then into the model. The attempted pipeline introduced work around this path without removing either copy.
Parameter resolution¶
Resolving and installing parameters added approximately 10–50 µs per parameter. Small repeated costs became material at model scale.
The Mistaken Model¶
The design diagram had two parallel lanes: load and execute. The machine did not. Both lanes shared storage, memory bandwidth, CPU time, and Python coordination. “Asynchronous” described the control flow, not the hardware behavior.
The baseline was also not naive. Existing Hugging Face loading includes optimizations that the custom scheduler bypassed or counteracted. A replacement has to beat the real implementation, not an imagined serial loader.
What Would Change the Result¶
A credible second attempt would need a storage layout designed for independent reads, a runtime with true asynchronous I/O, and a path that avoids duplicate copies. Without those properties, changing thread counts would tune the failed architecture rather than fix it.
Negative results are useful when the failed mechanism is measured. HotStart is complete because it establishes that this particular Python-level streaming design is the wrong abstraction.