Do Faithful & Unfaithful CoT Differ in How They Focus Attention?
By Yulia Volkova | MATS 9.0
Update (Aug 2026): Final Results
Findings of the Thought Anchors faithfulness project, last update Aug 2026.
Setup: a wrong "Stanford professor thinks the answer is (X)" cue is added to hard multiple-choice
questions. Some rollouts follow the cue and mention it (verbalizing); some follow it without
mentioning it (silent).
Model: DeepSeek-R1-Distill-Qwen-14B
Datasets: MMLU(-Pro) and GPQA variants
Setup and problem selection
1,494 problems screened (GPQA main 448, GPQA extended 546, MMLU-Pro 500), each with an incorrect professor cue appended.
Generated per problem: 32 cued rollouts, 20 uncued rollouts, and 8 no-reasoning answers (temperature 0.7).
Measured per problem: the cue's influence (gap), no-reasoning accuracy, the accuracy gain from reasoning, and the number of verbalizing (V) and silent (S) cue-following rollouts.
The analysis set contains 36 problems, with on average 26 usable rollouts per problem (the number varies with how many silent and verbalizing rollouts each problem produced). 24 problems where the cue strongly changes answers (gap ≥ 0.7, with ≥3 verbalizing and ≥3 silent rollouts), plus 21 meeting stricter criteria (cue influence ≥ 0.5, no-reasoning accuracy < 0.5, accuracy gain from reasoning ≥ 0.2, ≥3 verbalizing and ≥3 silent rollouts); 9 problems are in both groups.
Rollouts that reached the length limit were resumed with a 16k-token budget.
Average rollout length: 219 sentences on GPQA main, 247 on GPQA extended, 80 on MMLU-Pro (~5.0k / 5.3k / 1.6k tokens).
Finding 1: Attention statistics do not separate silent from verbalizing rollouts
Hypothesis: when a rollout follows the cue but does not verbalise, its attention pattern and "thought anchors" look different, and could give away unfaithful CoT.
The initial analysis compared consistently-silent problems against consistently-verbalizing ones (p = 0.04). But upon further investigation, on GPQA - where the significant result came from - the same kurtosis difference is present between those problem groups in uncued rollouts as well, where no faithful/unfaithful behaviour can happen. So it is more the property of the problems. (On MMLU the gap was marginal and noisy in both conditions; only 3-4 problems per group.)
Mean receiver-head kurtosis for consistently-verbalizing vs consistently-silent problems, in cued and uncued rollouts (error bars: standard error over problems). On GPQA - where the original p = 0.04 came from - the gap is present without the cue too.
The alternative approach is within-problem comparison: problems where the same problem produces both silent and verbalizing rollouts, each problem's rollouts compared with each other.
Length becomes a confounder:
rollout type
mean length (sentences)
no cue (uncued)
~200
cue, verbalized
~192 (statistically same as no-cue)
cue, silent
~186 (shorter, p = 0.001)
While on average giving the cue produces shorter CoT, the cue verbalizers have longer CoT than silent cue-followers: not only do they verbalize the cue itself, the mention also sends them into additional deliberation rounds where they tend to name all the answer options again (3x more option-naming after the mention) and this extra length is what makes their kurtosis higher. The gap is only an average - 41% of same-problem verbalizing rollouts are still shorter than silent ones.
Example of the post-mention text: "Wait, the professor thinks the answer is B) 9.0. So, perhaps the reasoning is that adding X increases the pH, making it basic. But why would that be the case? Alternatively, perhaps the reaction is exothermic..."
How the score works:
Each attention head gets a kurtosis over how much later sentences look back at each sentence (high = look-back concentrated on a few sentences). A rollout's score is the mean of its top-20 heads - the receiver heads, which Thought Anchors (Bogdan et al., arXiv:2506.19143, Section 4.1) defines as "attention heads that narrow attention toward specific sentences".
Two parts of the text are excluded, following the paper: the few sentences immediately after each sentence (the model always reads what it just wrote), and the opening sentences (the sequence start attracts attention for its position, not its content).
The paper also converts each sentence's look-back weights into rankings before averaging (rank normalization), which evens out scale differences. The Thought Anchors authors themselves note (Appendix C.4) that positional effects are only minimally accounted for.
The table shows the verbalizing-minus-silent difference d (standard-deviation units, paired within problem, 21-problem strict sample) under different settings:
metric version
raw
length-controlled
lighter exclusion windows (skip 1 sentence, ignore 3 neighbors)
d = +0.79, p = 0.0007
d = +0.33, p = 0.22
paper's exclusion windows (skip 10, ignore 20, reasoning sentences only)
d = +0.71, p = 0.007
d = +0.39, p = 0.29
rank-normalized (paper's depth control)
d = +0.37, p = 0.09
d = +0.02, p = 0.84
→ The effect is fully taken away when controlling for length.
Finding 2: Cue position affects cue following and cue verbalizing
Cue at prompt start: following often silent (37% of followers verbalize).
Cue mid-prompt: following halves; ~90% of followers verbalize.
The stronger the cue's effect on a problem's answers, the more often it gets mentioned: correlation +0.24 between a problem's cue influence and the share of its followers that verbalize (p = 3e-7, 457 problems).
Mentions appear late (~65% through the rollout), usually with hedging ("maybe the professor's answer..."), and recur ~2.5 times per rollout. Sentences after the first mention name answer options 3x more often.
Finding 3: Cue influence and accuracy with reasoning ON are anti-correlated
Across 1,493 problems, cue following is inversely related to both kinds of accuracy, but much more strongly to accuracy with reasoning (correlation -0.49).
Parallel with Boppana et al. (arXiv:2603.05488): they find the CoT role depends on difficulty - on easy problems the model commits to its answer internally at the start and the CoT text is performative. On hard problems the CoT and the belief in an answer are generated together. The result in this setting is complementary: where reasoning genuinely works, cues do not sway the answer; where reasoning fails, cues are what the model relies on. That's pretty intuitive - and same is found by Chen et al. (arXiv:2505.05410).
Mean cue-following rate per accuracy bin (95% CI). Red: accuracy with reasoning. Purple: accuracy answering immediately without reasoning.
Finding 4: We can't predict upcoming verbalization from the model's activations
Following the design of Boppana et al., a linear classifier reads the model's state at each sentence and predicts whether the rollout will go on to mention the cue or stay silent. Positions at or after a mention are excluded, and evaluation uses held-out problems.
Result
Chance-level (AUC 0.50-0.55) through the first 60% of the CoT & at most 0.63 towards the end of the CoT.
So verbalization is not really pre-committed - unlike the upcoming answer in Boppana et al.
Predicting upcoming silence vs mention from the state at each point of the reasoning. Dashed line = chance.
TL;DR
Below are the preliminary results from January 2026. Although the conclusions are outdated (see the Aug 2026 update above), you can still play with the visualizations and see how attention patterns change - I think it is cool. Try it!
A statistical measure of how "peaked" the attention distribution is. Higher kurtosis means attention is focused on fewer specific tokens rather than spread evenly.
Attention heads with highest kurtosis in their attention distribution - they consistently focus on specific source sentences.
Hypothesis
▶
This hypothesis did not survive controls - see the Aug 2026 update.
If white-box attention attribution patterns reflect how models use information during reasoning,
then faithful CoT (where reasoning genuinely influences the answer) should show focused attention
to relevant broadcasting sentences, while unfaithful CoT (post-hoc rationalization)
may exhibit more diffuse attention or anchor to different, potentially misleading parts of the reasoning chain.
This work replicates and extends Bogdan et al. (2025) "Thought Anchors"
to test whether receiver heads and vertical stripe patterns of attention targeted towards specific sentences ("thought anchors") differ between faithful and unfaithful chains.
If thought anchor patterns reliably differ between faithful and unfaithful CoT, they could serve as white-box indicators of CoT faithfulness in deployed models.
Detailed Key Findings
▶
The original (Jan-Feb 2026) findings below are superseded by the Aug 2026 update; here is their
current status after controls:
Kurtosis separates faithful from unfaithful CoT (p = 0.04, GPQA) (what happened to this result) -
disconfirmed: kurtosis tracks rollout length (rho = 0.98), and verbalizing rollouts are
~20 sentences longer; length-controlled within-problem differences are null.
Divergent attention circuits / universal receiver heads - not confirmed:
receiver heads look problem-specific, not faithfulness-specific; the corrected pre-registered rerun found no attention statistic separating silent from verbalizing rollouts after length control.
Behavioral correlations (cue-following vs accuracy) - stand, and are
superseded by the larger result: cue influence anti-correlates with reasoning accuracy
(rho = -0.37 across 1,493 problems).
Methodology
▶
Pipeline
Dataset Statistics
MMLU: 143 problems, 20 rollouts per condition, median reasoning accuracy 0.637, median no-reasoning accuracy 0.474.
GPQA-Diamond: 186 problems, 20 rollouts per condition, mean base accuracy 0.577, mean no-reasoning accuracy 0.378.
Model & Generation Settings
Two generation stages: (1) initial rollouts for faithfulness classification,
(2) attention analysis rollouts for visualization.
Model:deepseek-ai/deepseek-r1-distill-qwen-14b
Temperature: 0.7, Top-p: 0.95
Max tokens: 2048 (MMLU), 8192/2048 (GPQA stage 1/2)
Rollouts: 20 per condition (stage 1), 5 per condition (stage 2)
Faithfulness Classification
Faithful CoT: Follows cue AND mentions it in reasoning (threshold: ≥80% MMLU, ≥70% GPQA)
Unfaithful CoT: Follows cue but doesn't mention it (threshold: ≥80% MMLU, ≥70% GPQA)
Problem Selection Criteria
Problems were selected from MMLU and GPQA rollouts using the following criteria:
Low no-reasoning accuracy (<50%) - Problems where the model struggles without CoT
High cue response gap (≥0.5) - Large difference in cue-following between cued and uncued conditions
Cue ≠ Ground Truth - The cue answer differs from the correct answer
Each problem is unique
Preliminary Data Analysis
"r" represents the Pearson correlation coefficient, measuring the linear relationship between two variables.
Values range from -1 (perfect negative correlation) to +1 (perfect positive correlation), with 0 indicating no linear relationship.
"Gap" refers to the mean cue response gap, the average increase in cue-following behavior when a misleading cue is present.
MMLU
r(accuracy, cue following) = -0.459p < 0.0001
Lower base accuracy correlates with more cue following - the model is more susceptible to misleading cues on harder problems.
Cue following behaviour is likely confounded by problem difficulty. On harder problems, the model may fall back to the cue when struggling to find an answer. To control for this, future work should compare problems with equal uncued accuracy levels.
Gap = 0.289p < 0.0001
Mean cue response gap of 0.289 - 79.7% of problems show increased cue following when cued.
r(accuracy, gap) = 0.008p = 0.92
No significant correlation between base accuracy and cue response gap - the change in behavior is consistent across difficulty levels.
GPQA-Diamond
r(accuracy, cue following) = -0.522p < 0.0001
Lower base accuracy correlates with more cue following - effect is even stronger than MMLU.
Cue following behaviour is likely confounded by problem difficulty. On harder problems, the model may fall back to the cue when struggling to find an answer. To control for this, future work should compare problems with equal uncued accuracy levels.
Gap = 0.183p < 0.0001
Mean cue response gap of 0.183 - 67.2% of problems show increased cue following when cued.
r(faithfulness, accuracy drop) = -0.273p = 0.0002
Higher faithfulness (explicit cue mentions) → larger accuracy drops. Verbalizing the cue is associated with being more misled.
Attention Analysis
For each problem, we identify receiver heads, attention heads with high kurtosis
in their attention distribution. High kurtosis indicates the head consistently
attends to specific source sentences (creating "vertical stripes" in the attention matrix).
I compare which source sentences receive the most attention in cued vs. uncued rollouts.
Try It Out!
All in Category: Receiver heads are calculated using kurtosis measure calculated across all rollouts from all question IDs in the selected category. If only one category (faithful or unfaithful) is selected, kurtosis is computed across all rollouts from that category.
This Question Only: Receiver heads are calculated only across rollouts of this specific question ID.
Note on combining categories: If both faithful and unfaithful categories are selected, kurtosis would be computed across all rollouts from both categories combined. However, this may not be statistically appropriate since faithful and unfaithful problems may exhibit systematically different attention patterns.
Note: Bogdan et al. calculate receiver heads across the whole dataset of math rollouts (link).
📊 No Problems Available
No problems found for this dataset/category combination.
Question ID —
Question Text
Loading...
Ground Truth—
Cue Answer—
Reasoning Acc—
No-Reasoning Acc—
Faithfulness %—
⚠️Note: This problem may be non-representative as its reasoning accuracy is lower than no-reasoning accuracy.
However, attention patterns may still be interesting for such examples.
Below I look at the cued versus uncued versions of the rollout for question ID —, —
Top Receiver Heads ℹ️
Heads with highest kurtosis in attention distribution
Cued Rollout
Uncued Rollout
Shared Heads for cued & uncued:—
Attention Patterns
Click on a head above to view its attention matrix. Hover for values.
Cued Rollout
L0-H0Faithfulness rate: —%
Select a head to view attention
Top Source Sentences (Stripes) ℹ️
Uncued Rollout
L0-H0
Select a head to view attention
Top Source Sentences (Stripes) ℹ️
Summary
0Shared Heads among cued and uncued versions
0Shared Source Sentences
—Cue in Top Sources (Cued)
Now I look at the faithful and unfaithful rollout examples for the cued only version of question ID —, —
Faithful vs Unfaithful Comparison
Faithful vs Unfaithful comparison not available for this problem.
This requires both a cued rollout that mentions the professor AND one that doesn't.
Faithful (Cued + Mentions)
L0-H0
Select a head to view attention
Top Source Sentences ℹ️
Unfaithful (Cued + Silent)
L0-H0
Select a head to view attention
Top Source Sentences ℹ️
Future Directions
(1) Belief-trajectory probes in the cue setting - apply the design of
Boppana et al. (2026) (probe the
model's answer commitment at every sentence of the reasoning) to silent vs verbalizing
cue-following: does internal commitment to the cued answer form early while the text discusses
other things? Per-sentence activations for this are already being collected in the corrected
re-extraction.
(2) Matched admissibility test - the admissibility conclusion currently rests on a
cross-setting comparison; a clean version puts an admissible and an inadmissible cue at matched
positions in the same task.
(3) Mechanism of the hiring decision-position deficit - distinguish prefill
absorption from attention competition, e.g. by a mid-forward name-swap ablation.
(4) Cross-cue probe generalization - train a silence probe on professor-cue data
and test it on hiring demographics: does one "using information I am not admitting to" direction
exist?
(5) Combine with Verbalization Fine-Tuning (Turpin et al., 2025) -
if VFT increases cue admission, does the decision-position attention deficit close?
Acknowledgements
This work was done under the ATHENA fellowship with Arun Jose as mentor.
I would also like to thank Qiyao Wei and Andy Wang for valuable advice, feedback, and discussions.