Refusal Direction Persistence Through Supervised Fine-Tuning

I started this project to learn more about how supervised fine-tuning affects model safety behaviour. To do so I fine-tuned Qwen2.5-3B-Instruct on three datasets and evaluated how each SFT run affected results on three different safety benchmarks. To analyze mechanistic changes inside the models I used the approach developed by Arditi et al.. It uses difference-in-means between model activations when run on harmful and harmless prompts to identify a refusal direction. I first found a direction in the base model and tested whether it remained effective after fine-tuning. I then measured how strongly each checkpoint activated the direction and, for the checkpoint with the largest reduction, tried restoring the missing activation.

I found that although the fine-tuning altered refusal behaviour, the refusal direction associated with the unchanged base-model remained causally effective across SFT checkpoints. The degree to which the direction was activated by identical prompts at different checkpoints also corresponded to observed changes in behaviour. Restoring the activation to the level of the unchanged model consistently moved the behaviour towards the unchanged model, although it did not perfectly match the original refusal behavior.

Fine-tuning and evaluations

I kept the training settings fixed across all SFT runs while only changing the data. Each dataset contained 3,000 samples, split into 2,700 training and 300 validation samples. In every case training was run for a single epoch. The three datasets were:

• PKU-labelled-safe: responses labelled safe from PKU-SafeRLHF.
• Balanced WildJailbreak: an equal mixture of harmful and benign examples from WildJailbreak.
• Tool-use: an xLAM-derived set of tool-calling examples (single turn, mixture of call and no call examples).

To assess changes in safety behavior I used three benchmarks:

• XSTest: contains harmful requests as well as benign prompts that use potentially sensitive words; measures both compliance with harmful requests and false refusal of benign ones; judged using prompted GPT-5.4-mini.
• StrongREJECT: measures how useful a response is to a harmful requester with a harmfulness score; judged using the fine-tuned Gemma 2B evaluator published by the authors.
• HEx-PHI: broader set of harmful prompts; judged using Llama Guard 3.

The most important result of the fine-tuning runs is that PKU and WildJailbreak shift safety behavior in opposite directions. After fine-tuning on PKU data the model became generally less safe as illustrated above by the increases in harmful compliance, harmfulness score, and unsafe rate. Fine tuning on WildJailbreak made the model safer, while fine tuning for tool-use has left the safety behavior essentially unchanged. It should be noted that these evaluations used greedy decoding and cannot fully illustrate the changes to model behavior that would be observed with sampling based decoding.

The PKU result is somewhat counterintuitive because all training responses were labelled safe. The reason for increased harmful behavior is most likely the different style in which the samples in the dataset address unsafe requests. Most responses rely on redirection instead of clear refusal which is what the pre-SFT model leans towards. As a result these safe but out of distribution answers seem to lead to the model generally refusing less and providing answers to a greater number of harmful requests.

Refusal direction

The approach developed by Arditi et al. constructs candidate directions from the difference between average residual-stream activations on harmful prompts that the model refuses and harmless prompts that it answers. The method searches through different model layer and token position combinations and tests each candidate direction by ablating/adding it and measuring the change that creates on held-out prompts. Three scores are computed to measure the tendency for the direction to bypass refusal on harmful prompts (bypass score), induce refusal on harmless prompts (induce score), and alter behavior beyond refusal (KL score). The approach identified the direction in the final prompt token in layer 27 as having the highest bypass score while satisfying author thresholds for induce and KL scores.

A separate search at each checkpoint returned the same layer for tool-use and the neighbouring layer 26 for WildJailbreak, but no eligible direction for PKU. The PKU candidate was still close to the base direction but failed the check intended to limit changes on benign prompts.

Natural activation

I was curious to see if the changes in safety behavior observed for the fine-tuned models are reflected in the strength with which the identified refusal direction is activated. The plot below shows the mean projection of the residual-stream activation on the refusal direction across transformer layers. It can be seen that the change in activation strength correlates clearly with the changes in refusal behavior. The model fine-tuned for tool-use remains essentially unchanged while the models fine-tuned on PKU and WildJailbreak have activation respectively reduced and increased compared to the unchanged model.

Transfer after fine-tuning

I then wanted to see whether causal effect of interventions on the refusal direction remained after fine-tuning. The plot below shows StrongReject harmfulness score evaluated with ablation and addition interventions. The results clearly illustrate that the intervention produces the same effects across the SFT checkpoints. I found the same pattern on XSTest.

Restoring activation

The ablation and addition experiments illustrate that the causal effect of the identified refusal direction is preserved through fine-tuning but these interventions are heavy-handed and essentially shift the model fully towards refusal/non-refusal. I was curious to see if an intervention preserving the projection component identical to the pre-SFT model would result in more precise behavior restoration. For this experiment I focused on the model fine-tuned on PKU data since that is the model that demonstrated unintentional reduction in safety behavior. The restoration intervention modifies the layer-27 projection during the forward pass to match the base value at that point in the pre-SFT model.

The plots below illustrate safety evaluation results with the intervention. Restoring the activation reduced PKU harmful compliance from 35% to 24%, matching the base model, and reduced StrongREJECT from 0.432 to 0.180, close to the base score of 0.185. Recovery was less complete on HEx-PHI and XSTest false refusal returned only partway.

Preliminary sampling-based results

All prior evaluations used greedy decoding which selects the highest-probability token at each generation step. This allows for better reproducibility but only gives information for the most likely output of the model. In practice sampling based approaches are used for decoding. Since we are no longer picking the most likely output of the model, the full distribution and any changes to it become significant to the behavior observed. In Hughes et al. it has been shown that sampling can be its own form of jailbreaking where with a sufficient number of samples one can generate harmful responses despite them not being the most likely output. I consequently ran evaluations on StrongREJECT and HEx-PHI using sampling based decoding. The results below are preliminary since I only ran it with one random seed but they illustrate that although the restoration intervention still shifts the model towards pre-SFT behavior it is no longer as effective. This indicates that fine-tuning changed the probability assigned to lower-ranked tokens in a way that refusal direction restoration cannot fully remedy after the fact.

Next steps

The thing I am most interested to investigate next is a training-time intervention based on the identified refusal direction. I am curious whether that would effectively preserve safety behaviour during fine-tuning and how it would compare to restoration applied afterwards.