Tracing What Models Forget
A measurement audit of LLM unlearning and relearning; why recovery scores can overstate factual recovery.
Mechanistic Interpretability
Pytorch
CUDA
Llama 3.2
Machine Unlearning
Relearning restores answers before it restores knowledge
Machine unlearning is usually evaluated behaviorally: make a model forget a fact, attack it with relearning, then ask whether the answer comes back.
That sounds straightforward. It isn’t.
A model can become more willing to produce an answer without becoming equally better at distinguishing the true answer from plausible false ones. Most recovery metrics collapse those two behaviors into one number.
This project started as an attempt to understand which facts survive unlearning. It ended up exposing a more basic problem: what does “recovered” actually mean?
Main Result: relearning restored assertion much more strongly than factual discrimination. The gap appeared in both a synthetic benchmark and an external benchmark built from real pretrained knowledge.
The distinction I cared about
There are two different questions hiding inside “did the model recover?”
Assertion asks whether the model can produce the answer again.
Discrimination asks whether the model actually prefers the true answer over plausible false alternatives. These are not equivalent.
A model can become more responsive after fine-tuning while probability rises across several candidate answers at once. If the true answer rises, a recovery metric can register progress even when plausible false answers rise with it.
That suggested a simple rule for the audit: don’t only measure whether the true answer moved. Measure whether it moved relative to things that cannot represent recovered knowledge.
What is a recovery score actually measuring?
On TOFU, I decomposed the measured recovery signal into three components.
R = N + V + K
N nonspecific movement
matched false answers move too
V answer-form / variant effect
true variants move differently
K knowledge-specific recovery
truth gains relative to matched false answers
Only K directly measures restored factual discrimination. A rise in N cannot, by construction, mean that the forgotten fact returned: the comparison answers are deliberately false.
At the published relearning horizon, only 9.2% of the measured recovery was knowledge-specific. Roughly 61.5% came from movement that also occurred on matched false answers.
Across the trajectory, the knowledge-specific share remained only 8.2–14.9%.
More strikingly, 45.7% of items labeled “revived” at the published horizon showed no positive knowledge-specific gain at all.
That does not mean relearning restored no knowledge. It means the headline recovery number was measuring considerably more than knowledge.
The external test mattered more
TOFU contains synthetic facts introduced through fine-tuning. A result there could still be a benchmark artifact.
So I moved to an external setting built around real pretrained factual knowledge, using GradDiff, NPO, and JensUn. The same separation appeared.
Assertion recovery reached GradDiff: +26.3, NPO: +17.3, and JensUn: +7.7 points across the three methods.
Contrastive factual-discrimination recovery on the same items was only +6.3, −1.0, and −4.0.
The gap was not just “two metrics disagree.” Relearning could strongly restore answer production while doing far less to restore the model’s preference for truth over plausible false alternatives.
The per-prompt behavior made that concrete.
After relearning:
GradDiff asserted the true answer on 6.1% of prompts and a matched false answer on 4.5%.
NPO asserted the true answer on 7.1% and a matched false answer on 6.9%.
For NPO, the model was almost as willing to assert a matched false answer as the truth.
That is hard to reconcile with the intuitive reading of “the knowledge came back.”
Aggregation can make the effect look larger
The external evaluation also exposed an aggregation effect.
A per-prompt success rate of roughly 6% can become a much larger item-level “revival” rate when the benchmark asks many prompt variants and counts an item as recovered if any one succeeds.
one prompt succeeds ≈ 6%
32 opportunities to succeed
↓
“did any of them succeed?”
↓
much larger item-level revival rate
That does not make the metric invalid. But it changes what the number means.
An item-level recovery score can reflect a mixture of restored factual discrimination, broader answerability, prompt sensitivity, and aggregation over many attempts.
That distinction should be explicit whenever relearning robustness is compared across methods.
That does not make the metric invalid. But it changes what the number means.
An item-level recovery score can reflect a mixture of restored factual discrimination, broader answerability, prompt sensitivity, and aggregation over many attempts.
That distinction should be explicit whenever relearning robustness is compared across methods.
Reproducing the paper exposed a second problem
While reproducing the external benchmark, I initially could not match the paper’s unlearned states.
The reason turned out to be implementation-level, not scientific.
The released configuration used a single learning rate with linear decay. The paper’s appendix specified per-method learning rates with a cosine schedule.
Using the paper’s settings, the unlearned states matched the published values exactly:
GradDiff 2.0 → published 2.0
NPO 6.0 → published 6.0
JensUn 0.0 → published 0.0
NPO alone moved from 13 → 6 from that correction.
That fixed the starting point. It did not fix the published relearning behavior.
NPO alone moved from 13 → 6 from that correction.
That fixed the starting point. It did not fix the published relearning behavior.
Changing relearning horizon reordered the apparent robustness of methods.
Changing an unstated relearning optimizer/schedule moved JensUn recovery from roughly +17 → +9.
And even after bounded calibration, the paper’s NPO recovery of +2 could not be reproduced; my runs remained around +20.
So the second conclusion became:
Relearning robustness is not safely interpretable as a method-intrinsic property without tightly specified attack protocol.
I would not claim that the benchmark “ranks methods incorrectly.”
The stronger and more defensible statement is that the ranking itself can move when the relearning protocol moves.
The project was designed to kill its own results
The most important part of the work was not finding a large effect. It was repeatedly discovering that attractive explanations did not survive controls.
Early on, repeated relearning attacks appeared to recover the same items with surprisingly high stability. That looked like a strong item-level property. Then benign fine-tuning reproduced much of the same structure.
Another candidate result suggested a sharp recovery regime near a probability floor.
Another apparent sign-disagreement result weakened after fixing a deduplication bug and checking what the null distribution would do.
Those results were not quietly removed. They stayed in the repository as deviations, failed hypotheses, and control outcomes.
The working rule was simple: Persistence applies to implementation. It does not apply to validity.
If an experiment failed because of infrastructure, fix it. If it failed because a control killed the interpretation, the interpretation dies.
The research system
I built a reproducible post-training and evaluation pipeline around open-weight models with:
full-model relearning attacks
multiple unlearning method families
repeated seeds and attack conditions
paraphrase and matched-false scoring
independent benign controls
resumable GPU execution
immutable raw outputs
analysis scripts tested on synthetic planted values
The research design also separated discovery from confirmation wherever possible and forced important decisions to be frozen before reading outcomes.
What I think the result means
The broad idea that behavioral forgetting does not necessarily imply knowledge deletion is already established.
The contribution, I feel, is narrower.
I quantified how much post-unlearning relearning recovery can separate into answer-production recovery and factual-discrimination recovery on the same items, under the same attack, and then tested how stable the apparent robustness signal was to protocol choices.
The external benchmark did not reproduce the extreme TOFU decomposition numerically. Knowledge discrimination did return to a meaningful degree, roughly 18–46% of what unlearning removed in the relevant analysis.
Relearning can restore the model’s ability to assert answers substantially faster than it restores factual discrimination, while the amount of apparent recovery is itself sensitive to the attack protocol.