Do Faithful & Unfaithful CoT Differ in How They Focus Attention? Hi!

By Yulia Volkova | MATS 9.0

Update (Aug 2026): Final Results

Findings of the Thought Anchors faithfulness project, last update Aug 2026. Setup: a wrong "Stanford professor thinks the answer is (X)" cue is added to hard multiple-choice questions. Some rollouts follow the cue and mention it (verbalizing); some follow it without mentioning it (silent).

Setup and problem selection

Finding 1: Attention statistics do not separate silent from verbalizing rollouts

paired differences

Finding 2: Cue position affects cue following and cue verbalizing

Finding 3: Cue influence and accuracy with reasoning ON are anti-correlated

Cue following vs accuracy with and without reasoning

Mean cue-following rate per accuracy bin (95% CI). Red: accuracy with reasoning. Purple: accuracy answering immediately without reasoning.

Finding 4: We can't predict upcoming verbalization from the model's activations

Result

Probe AUC along the reasoning

Predicting upcoming silence vs mention from the state at each point of the reasoning. Dashed line = chance.

TL;DR

Below are the preliminary results from January 2026. Although the conclusions are outdated (see the Aug 2026 update above), you can still play with the visualizations and see how attention patterns change - I think it is cool. Try it!

Hypothesis ▶

Detailed Key Findings ▶

Methodology ▶

Try It Out!

Question ID —

Question Text

Loading...

Ground Truth —
Cue Answer —
Reasoning Acc —
No-Reasoning Acc —
Faithfulness % —

Below I look at the cued versus uncued versions of the rollout for question ID —, —

Top Receiver Heads ℹ️

Cued Rollout

Uncued Rollout

Shared Heads for cued & uncued: —

Attention Patterns

Click on a head above to view its attention matrix. Hover for values.

Cued Rollout

L0-H0 Faithfulness rate: —%
Select a head to view attention

Top Source Sentences (Stripes) ℹ️

Uncued Rollout

L0-H0
Select a head to view attention

Top Source Sentences (Stripes) ℹ️

Summary

0 Shared Heads among cued and uncued versions
0 Shared Source Sentences
— Cue in Top Sources (Cued)

Now I look at the faithful and unfaithful rollout examples for the cued only version of question ID —, —

Future Directions

(1) Belief-trajectory probes in the cue setting - apply the design of Boppana et al. (2026) (probe the model's answer commitment at every sentence of the reasoning) to silent vs verbalizing cue-following: does internal commitment to the cued answer form early while the text discusses other things? Per-sentence activations for this are already being collected in the corrected re-extraction.
(2) Matched admissibility test - the admissibility conclusion currently rests on a cross-setting comparison; a clean version puts an admissible and an inadmissible cue at matched positions in the same task.
(3) Mechanism of the hiring decision-position deficit - distinguish prefill absorption from attention competition, e.g. by a mid-forward name-swap ablation.
(4) Cross-cue probe generalization - train a silence probe on professor-cue data and test it on hiring demographics: does one "using information I am not admitting to" direction exist?
(5) Combine with Verbalization Fine-Tuning (Turpin et al., 2025) - if VFT increases cue admission, does the decision-position attention deficit close?

Acknowledgements

This work was done under the ATHENA fellowship with Arun Jose as mentor. I would also like to thank Qiyao Wei and Andy Wang for valuable advice, feedback, and discussions.