"The primary reason many LLM inference endpoints are nondeterministic is that server load changes the batch size used during inference."
Horace He and colleagues explain in September 2025 that changing request loads alter inference batches, exposing batch-sensitive numerical kernels and producing different outputs even when greedy decoding is used. Their analysis supplies the technical basis for the supplied sidenote on runtime nondeterminism.
View Source