Variation in server load and batch size can cause nondeterministic LLM outputs even when inference runs on CPUs or TPUs rather than GPUs.

Format Web Page

Location PART 3 - THE THIRD AXIS CHAPTER 15: DEPLOY NOW, PRAY LATER Are Your Abstractions Managed?

"Variation in server load and batch size can cause nondeterministic LLM outputs even when inference runs on CPUs or TPUs rather than GPUs."

The Thinking Machines Lab analysis explicitly notes that batch-size-driven nondeterminism is not unique to GPUs and can occur in CPU- or TPU-served systems. This broadens the mechanism beyond explanations that rest only on concurrent GPU execution.

View Source

Source title Defeating Nondeterminism in LLM Inference

Source authors Horace He with collaborators at Thinking Machines Lab

Source year 2025

Confidence high

Importance mid