"Chengshuai Zhao and colleagues found that exact-match accuracy could collapse from 100 percent to approximately zero under controlled distribution shifts."
Zhao and colleagues trained language models in the controlled DataAlchemy environment and evaluated chain-of-thought behavior as task distributions changed. Their reported transformation-generalization results included 100 percent exact-match accuracy in-distribution, falling to 0.01 percent or zero in several shifted conditions. The final paper was published in ACL Findings 2026.
View Source