Symbolic Mechanistic Data Attribution: Tracing Training Influence to Learned Behavioral Policies

arXiv:2606.29171v2 Announce Type: replace Abstract: While existing data attribution methods can identify which training examples build specific mechanistic circuits, they cannot explain how training data shapes the behavioral decisions a model learns. We introduce Symbolic Mechanistic Data Attribution (SMDA), a framework that attributes training pairs to the interpretable symbolic policies governing model behavior. SMDA fits a closed-form Ridge regression over sparse autoencoder (SAE) features to model a behavior, then decomposes how each supervised fine-tuning (SFT) example shifts that policy through feature-activation ($\Delta X$) and output-probability ($\Delta Y$) pathways. We distill a symbolic policy for refusal in Llama-3.2-3B-Instruct and analyze 200 SFT pairs. Our analysis reveals (1)~the symbolic policy's coefficients expose systematic gaps in the base model's safety behavior, such as religious stereotyping; (2)~per-feature $\Delta X/\Delta Y$ decomposition mechanistically explains why harmful and harmless pairs exert qualitatively different influences on certain features; and (3)~individual pairs routinely exhibit cross-feature interference, allowing SMDA to identify pairs whose dominant effect falls on unintended features. Retraining without such pairs improved accuracy by up to 4.5 percentage points over full fine-tuning. These results show that combining mechanistic interpretability with data attribution yields a diagnostic tool both more fine-grained than black-box influence functions and more scalable than manual circuit analysis.