Investigating Model Organism Robustness to Fine-Tuning

Model organisms are models intentionally trained to exhibit some behaviour or behavioural pattern to help study it in depth. They are useful in alignment research for developing methods that aim to detect, remove, or control undesirable behaviour. Any conclusions one tries to draw based on experiments with model organisms depend on how well the model organisms approximate future misaligned systems. Although we do not know what those systems are going to look like, one way to make any findings potentially more relevant is to work with model organisms robust to simple model perturbations such as untargeted fine-tuning. Otherwise, a complicated method can look successful even when ordinary unrelated fine-tuning would have produced the same result.

A blog post from Redwood Research, Advice for making robust-to-training model organisms, investigated several factors that affect a model organism’s robustness, with robustness here meaning whether backdoor behaviours survive later training. Another recent paper, The Model Organism Lottery, constructed models with approximately matched behaviours, which the authors call quirks, using different training approaches and data mixtures. The authors found that the way the quirk was introduced strongly affected how interpretable it was.

I was curious to see how different construction methods affected the robustness of the resulting model organisms. I used the released model organisms for one quirk from The Model Organism Lottery, applied the same unrelated full-weight fine-tuning intervention to each, and measured quirk expression at different checkpoints during training. I found that narrow post-hoc DPO produced the least robust organisms, transcript distillation was more robust, and mixed transcript distillation retained more of the quirk than its unmixed counterpart. Integrated DPO retained the most quirk expression.

Model organisms

The Model Organism Lottery investigated a number of different quirks introduced into OLMo 2 1B model. Here I picked just one, the MilitarySubmarine quirk, which causes a model to mention submarines when answering questions about military topics. I followed the approach used by the authors, where the strength of the quirk is measured using the quirk expression rate (QER): the fraction of responses to trigger prompts that mention submarines in a military context. Whether the quirk is expressed in a response is measured using an LLM judge. I reused the authors’ judging prompt but changed the judge model, making sure that the baseline rate agreed with the rate reported by the authors.

I compared five released organisms built using three approaches:

• Integrated DPO inserts the quirk data into OLMo’s broad DPO post-training phase.
• Post-hoc DPO applies a small preference-training run to an already post-trained model.
• Transcript distillation (TD) fine-tunes the post-trained model on responses that demonstrate the quirk.

The post-hoc DPO and TD organisms each have mixed and unmixed variants. The mixed variants include unrelated examples in their training data, while the unmixed variants use only quirk-related data. All the construction runs were tuned by the paper’s authors to produce similar initial QER.

Untargeted fine-tuning

I fine-tuned every organism, as well as the unmodified baseline, on the same filtered version of the original Alpaca dataset. I applied two epochs of full-weight supervised fine-tuning to all six models using the same corpus, optimizer settings, and random seed in every case. It is worth noting that I used Alpaca’s original answers while the robust-to-training post’s main question answering (QA) intervention used Qwen-generated Alpaca answers.

At each saved checkpoint, I measured trigger and control QER. I also evaluated the checkpoints on GSM8K, a benchmark of grade-school mathematics word problems, to check that falling QER was not simply the result of catastrophic forgetting.

Because the baseline itself sometimes mentions submarines in military contexts, I use excess QER—the organism’s prompt-level QER minus the baseline model’s QER after the same amount of fine-tuning—as the main metric.

Results

The plot above shows that most of the behaviour was lost very early. By approximately 0.5 million input tokens, excess QER had fallen from around 52 percentage points to between 16 and 28 points. The trajectories then declined much more slowly and settled at different plateaus. The two post-hoc DPO organisms were the least robust, while transcript distillation resulted in better quirk retention. Interestingly, the results do not support a simple claim that unrelated data always makes an organism more robust. Mixing unrelated data improved retention for TD but not for post-hoc DPO, suggesting that its effect depends on the training approach.

Integrated DPO retained the most behaviour. The main differences I would focus on are that it introduced the behaviour earlier, within a much larger training run, and alongside a more varied mixture of data. Together with post-hoc DPO underperforming TD, I think this suggests that the training stage, amount of data, and composition of that data are aspects to be studied further to better understand model organism robustness.

Interpretability and robustness

This work uses model organisms developed for The Model Organism Lottery, where the main focus was on the interpretability of the resulting organisms. This raises the question of whether quirk interpretability correlates with robustness. Integrated DPO was among the least interpretable MilitarySubmarine quirk constructions in the paper and the most robust in the experiments presented here. The authors observed that moving from unmixed to mixed TD reduced interpretability, while I found that it also increased robustness. Both observations are consistent with the possibility that a behaviour that is harder to locate is also harder to overwrite.

On the other hand, mixed DPO was less interpretable than unmixed DPO, but my experiments did not find it to be meaningfully more robust. Comparing across training approaches, the post-hoc DPO organisms were less interpretable than the TD organisms, yet TD retained more QER after fine-tuning. My limited results suggest that low interpretability on its own is not a reliable proxy for resistance to later training.

Next steps

The first thing I want to do next is run the same experiments on the other quirks released by the authors of The Model Organism Lottery to see whether the results are consistent across different behaviour targets. Beyond that, I am most interested in investigating whether transcript distillation would match or exceed integrated DPO if quirk examples were incorporated at a similarly early stage of a similarly large and varied training run.