"Chain-of-thought rationales can be misleading and are neither necessary nor sufficient for trustworthy interpretability."
Barez and colleagues review evidence that verbalised reasoning can diverge from the hidden computations that actually drive model predictions. Their research directly supports the distinction between a convincing step-by-step explanation and an explanation that faithfully reveals why the model reached its conclusion.
View Source