Follow the Gradient: Tracing Safety Degradation Back to Training Data

Fine-tuning is how language models get adapted to new tasks, but it comes with a hidden cost: it can quietly make a model less safe, even when the training data contains nothing harmful. Qi et al. (2024)¹1.Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! have shown that fine-tuning even on popular, benign instruction datasets like Alpaca and Dolly is enough to weaken a model's safety guardrails. The safety-degrading examples are often common things like math problems, bulleted lists, and simple coding tasks (He et al., 2024²2.What is in Your Safe Data? Identifying Benign Data that Breaks Safety), so standard content filters like LLM safety judges often struggle to catch it (Li et al., 2025³3.Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment).

Because what is making the data dangerous is in how it changes the model rather than what it says, we turned to gradients: the direction a model moves when trained on an example. Gradients allow us to see two examples that push the model in the same way, even if they don’t look similar. We developed SimpleLoRIF, a scalable gradient embedding method, and used it to build SafeTulu, a filtered version of the popular Tulu 3 SFT dataset (Lambert et al., 2024⁴4.Tulu 3 SFT mixture). Training on SafeTulu reduces attack success rate by up to 22% relative across Qwen, Llama, and Olmo, while largely preserving capability. We've released both the dataset and a 🤗 Space that allows for creating custom versions of SafeTulu.

Gradient-based data attribution has historically been a small-scale analysis tool. Per-example gradients are as large as the model itself, which makes embedding a full post-training dataset impractical. However, SimpleLoRIF builds on the low-rank influence method LoRIF (Li et al., 2026⁵5.LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution), drops the curvature correction we don't need for ranking, and backpropagates through only the final four transformer blocks. That makes it 91.53% faster than the base LoRIF formulation while preserving a strong attribution signal. With it, we embedded all 939,343 examples in Tulu 3 in just over an hour on 8 H100 GPUs. At that speed, SimpleLoRIF allows for filtering post-training mixtures with millions of examples. By changing the target behavior set, SimpleLoRIF can also unlock new alignment possibilities such as auditing which examples guide models towards actions like sycophancy, upweighting data that pushes models toward behaviors like calibrated uncertainty, and more.

To build SafeTulu, we scored every Tulu 3 example by how close its gradient was to the gradients of a reference set of unsafe responses, then removed the 20% closest examples. The results held across every model we tested. On Qwen 3 1.7B, GCG attack success rate fell from 47.7% to 37.0%. Llama 3.2 1B fell from 38.0% to 30.3%, and Olmo 2 1B fell from 48.3% to 41.3%. SafeTulu outperformed both an LLM-as-a-judge filter and a prior gradient-based method (He et al., 2024). Every embedding came from Qwen 3 0.6B, a model smaller than all three fine-tuned models and from a different family than two of them. The safety gains also didn't come at the expense of usefulness, as capability stayed close to the full-data baseline across six benchmarks.

SafeTulu is available now. In addition, we created the SafeTulu 🤗 Space to let you customize the data to your own needs. Tag us on X at @datologyai with your uses! Any custom version can be shared with a link or uploaded to 🤗 Hub with only a few clicks in the Space. You can audit only the subsets you care about (e.g., only tool-use data or multilingual data), use SafeRLHF labels to target specific harms (e.g., endangering health), do nearest-neighbor analysis, and more. From curating a new post-training mixture to filtering Tulu subsets you already use to studying what makes benign data unsafe, the Space provides many ways to use gradient-based safety filtering.

SimpleLoRIF: Gradient Embeddings at Scale

The Basic Idea

We represent each document by its respective gradient in a reference language model. To score a candidate training document, we measure how close its gradient is to the gradients of a collection of unsafe responses. Documents with the closest gradients are the ones most likely to push the model toward unsafe behavior, and thus we remove them.

To ensure our method was scalable, we built on LoRIF (Li et al., 2026⁵5.LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution), a recent influence method with a highly compact gradient representation. LoRIF represents a document by its gradient at the model's linear layers, compressed in two phases: two-sided random projection, followed by low-rank factorization. Building on LoRIF allowed us to drop each layer’s representation from millions of floats to only 64. Because we only need to rank examples for filtering, rather than estimate their exact influence or compare score magnitudes across models, we removed LoRIF's curvature correction and score examples using only gradient similarity.

Finding the Bottleneck with Four Ablation Axes

Once the curvature term is gone, the dominant cost is the backward pass. The projection is defined per layer, and the similarity sums over whichever layers we select. So, the set of layers we embed determines how much of the network we must backpropagate through. To see where we could make the method less expensive, we needed a way to check that cheaper configurations still produced good embeddings.

To do this, we designed a detection task on a math benchmark. We embedded each train and test math item as a (question, answer) pair, with the question tokens masked out of the loss. Every test item was then scored by its nearest-neighbor cosine similarity to the training set in embedding space. We computed the ROC-AUC of that score against whether the item was newly answered correctly after training. A high AUC means the embedding places a test item near the training documents that actually changed the model's behavior on it.

We then ran ablations along four axes, recording the wall-clock time and detection AUC for each:

1. Projection Dimension: Changing d (e.g. 16 or 64 instead of 32) had little effect on either AUC or speed.
2. Token Truncation: Reading only the first 128 or 256 tokens of each document was effective on this benchmark. However, it is risky, because it may not generalize to longer documents where the important content may come later.
3. Smaller Embedding Model. Swapping in a smaller scoring model cut cost but lost too much quality.
4. Number of Transformer Blocks. Backpropagating through only the last N blocks preserved quality while greatly reducing cost, so we used this.

By restricting backpropagation to the final four transformer blocks, SimpleLoRIF achieves a 91.53% speedup while retaining useful detection performance (0.684 AUC). This allows us to embed all 939,343 Tulu 3 examples in just over an hour on 8 H100 GPUs.

SafeTulu

How We Built It

The SafeTulu pipeline is as follows:

1. Embed: Compute gradient embeddings for every document in Tulu 3 and for the target set. Embeddings were computed with Qwen 3 0.6B. The target set is the 43,469 prompt-response pairs with unsafe responses from SafeRLHF (Ji et al., 2024⁶6.PKU-SafeRLHF).
2. Remove: Score each Tulu 3 document by gradient similarity to the target set and remove the top 20%.
3. Train: Fine-tune Qwen 3 1.7B, Llama 3.2 1B, and Olmo 2 1B on the filtered data, with three random seeds each.
4. Evaluate: Measure safety and capability.

We compared SafeTulu against four baselines, each trained on Tulu 3 with the same pipeline:

• Baseline Training: the full, unfiltered Tulu 3.
• Random Removal: 20% of examples removed at random.
• LLM-as-a-Judge: a Qwen 3 8B judge scores every example 0–5 for safety using a detailed rubric from Maini et al. (2025)⁷7.Safety Pretraining: Toward the Next Generation of Safe AI, with the worst-scoring 20% removed.
• He et al.: 20% removed using the bidirectional gradient criterion of He et al. (2024)²2.What is in Your Safe Data? Identifying Benign Data that Breaks Safety.

Full evaluation and training details are in the paper.

Safety

SafeTulu achieves the lowest GCG attack success rate (ASR) for every model. For Qwen, ASR drops from 47.7% under baseline training to 37.0% with SafeTulu, a 22% relative reduction. Llama drops from 38.0% to 30.3%, and Olmo drops from 48.3% to 41.3%.

The LLM-as-a-judge filter lands at 47.3% for Qwen, 39.0% for Llama, and 46.3% for Olmo. This fits the idea that the safety-degrading data in Tulu 3 isn't visibly harmful, so a content-based judge has little to work with. SafeTulu also outperforms He et al., and achieves the lowest attack success rate across all three models on ActorBreaker, a more difficult, multi-turn conversational safety benchmark.

Notably, all the examples were filtered using Qwen 3 0.6B, and the safety improvements still transferred across Qwen, Llama, and Olmo models.

Capability

Removing 20% of the dataset mostly avoids hurting downstream performance. Across MMLU Pro, IFEval, GSM8K, MATH 500, MBPP, and MedQA, models trained on SafeTulu stay close to, and sometimes above, the performance of models trained on the full dataset. For Qwen, SafeTulu matches or improves on the baseline on every capability benchmark. For Llama and Olmo, the results contain some minor drops.

SimpleLoRIF and SafeTulu Space Use Cases

The SafeTulu dataset and 🤗 Space make it simple to build a version of SafeTulu that fits your own needs. Beyond choosing your own removal fraction, you can also filter only certain subsets. For example, a team looking to add more math data to its SFT mixture without degrading safety could filter and download only NuminaMath, while researchers focused on multilingual data could filter only the multilingual subsets. You can also target specific harms instead of general safety by joining each example’s nearest SafeRLHF neighbor with SafeRLHF’s harm-category labels. For instance, a team building a medical assistant could remove only the data whose nearest harmful neighbor in SafeRLHF falls under the “Endangering Public Health” category.

More broadly, SimpleLoRIF allows for new possibilities for using gradient embeddings for safety. Teams can filter large post-training mixtures as they grow, for instance by screening a newly collected dataset for safety-degradation risk before merging it into a larger mixture, instead of evaluating safety only after training. Because the stored embeddings can be scored against any target set, SimpleLoRIF also supports auditing which examples steer models toward specific undesirable behaviors. For example, scoring a mixture against a target set of sycophantic responses could reveal which subsets most strongly reinforce sycophancy.

Tag us on X at @datologyai with your uses! Any custom version can be shared or uploaded from the Space. More use cases are detailed in the paper.

What Can the Removed Data Teach Us?

SimpleLoRIF treated superficially similar data very differently. While the LLM judge removed different math subsets at an almost identical rate, SimpleLoRIF varied its rate by more than 2×, indicating it is picking up on something beneath the surface. What do the removed examples have in common?

We measured the relative risk (RR) of removal for a few features, and found the following patterns:

• Tool-execution failures: Removed examples are 1.71× more likely to contain a traceback or exception.
• Simple coding tasks: Simple "write a Python function" tasks are removed at 1.74× the baseline rate, while calculus and differential-equation modeling problems are more often kept (0.51×).
• Narrative arithmetic: Gradient filtering removes problems where quantities are carried through a sequence of wordy prose operations, and more often keeps examples with explicit algebraic setup, substitution, and simplification (0.70×).

A common trend is that gradient filtering seems to remove direct examples, like clean arithmetic and simple coding tasks, more often, and keep more elaborate reasoning, like algebraic setup and differential-equation modeling. One possible interpretation is that direct, immediately helpful answers reinforce a general pattern of immediate compliance. We stress that these patterns are suggestive, not established. Namely, we do not know for certain that the identified features directly cause safety degradation.

Emergent Misalignment

We also tested whether gradients could capture emergent misalignment. Betley et al. (2025)⁸8.Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs showed that fine-tuning a model to write insecure code, without ever asking for anything harmful, can make it broadly misaligned.

We investigated whether this connection between the narrow training data and the broad harmful behavior is visible in gradient space. We took two misaligned models, collected harmful responses from each of them, and scored insecure-code and secure-code training examples by their gradient similarity to the harmful outputs in the misaligned models’ gradient spaces.

We found that the misalignment was visible in their gradient spaces, with gradient similarity separating insecure from secure code with an AUC of 0.604 for Qwen and 0.865 for Mistral. More experiments, including cross-model transfer and scaling, are in the paper.

Conclusion

Benign-looking fine-tuning data can reduce a model's safety. Because that data isn't visibly harmful, content-based filters like LLM judges struggle to catch it. We tackled the problem with gradient attribution, identifying training examples whose gradients are closest to those of unsafe responses. By making gradient attribution scalable, SimpleLoRIF offers a practical way to identify and remove harmful examples before they influence a model’s behavior. With it, we created SafeTulu: a version of Tulu 3 that reduces attack success by up to 22% relative while largely preserving capability.

Ultimately, gradient attribution doesn’t have to be a small-scale analysis tool. It can be a practical way to catch safety-degrading data before it enters large post-training mixtures.

Full details, including all results with standard errors, hyperparameters, and additional experiments, are available in the paper.