Cozmin Ududec
This stream studies scope generalisation in post-training: why similar narrow training signals can produce a broad change in behaviour, a local strategy, or a response tied to one particular setting. Fellows will work with controlled model organisms, comparing variants of SFT and RL, and tracking how behaviour and internal representations shift during training.
Stream overview
One current research interest is scope generalisation in post-training: why similar narrow training signals can produce a broad change in behaviour, a local strategy, or a response tied to a particular setting. The projects below follow a rough progression: what the model learns, what settings bring it out, and whether later training removes or hides a behavior or disposition. We would study these questions in controlled model organisms, comparing variants of SFT and RL, while tracking how behaviour and internal representations change during training.
How training shapes generalisation. Compare SFT and RL runs that learn a narrow task equally well but differ in their wider effects. The goal is to understand how the loss function, optimiser, and training data steer models towards a broad disposition, a limited strategy, or behaviour tied to the training setting. [1–4]
How models decide what situation they are in. Models may behave differently when they think they are helping a user, being tested, or acting in an automatically graded task. By changing these cues while keeping the task fixed, we could study how models infer their role and whether this brings out a different strategy or persona. This would distinguish a narrow response to an immediate incentive from a broader shift in behaviour. [3,4]
Removing or hiding misalignment. Later safety training may remove an unwanted behaviour, make a safer response the default, or make the old behaviour harder to elicit. A project could follow models through successive stages of SFT, RL, and safety training, then systematically test whether the behaviour has disappeared or remains available in settings that resemble the original training. [4,5]
References
[1] Betley et al. (2025), Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs.
[2] Brown, Leask, and McKinney (2026), Evil Spectra: How Optimisers can Amplify or Suppress Emergent Misalignment.
[3] Yudelson, Ivanov, and Greenblatt (2026), Reward Hacking Without Egregious Misalignment in an RL-Only Setting.
[4] Qi, Wright, MacDiarmid, and Hubinger (2026), Training a Misaligned Reward Seeker.
[5] Dubiński et al. (2026), Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers.
Standard (1-2 hours of weekly 1:1s)
London
Indifferent
Indifferent
Mentors
I currently lead the science of evaluation team at the AI Security Institute in London. I joined AISI early in its life, and have worked in several roles, including co-leading the team responsible for our pre-deployment testing programme.
I'm generally interested in topics around dangerous capability evals, and understanding agent behaviours and their implications for policy. In particular, I'm interested in:
- Developing predictive statistical models that capture the relationships between LLM performance, task characteristics and LLM characteristics.
- Developing methods to predict when models might cross a capability threshold.
- Developing hinting-based methods for partial progress & estimating gaps in knowledge/skills required to cross thresholds (and track these over time).
Before AISI I was chief scientist at a startup in Cambridge, where I led a team of 25 researchers with a mission to optimize decision making in electricity grids, and improve economic efficiency and reduce emissions.
I have a PhD in physics from the University of Waterloo and Perimeter Institute for Theoretical Physics. My focus was on reconstructing quantum theory from simple first principles, so we can all stop worrying about the reality of the wave-function.
Fellows we are looking for
- Strong ML implementation skills — able to modify and debug LLM fine-tuning code in PyTorch or JAX and run careful, reproducible experiments.
- Evidence of research ownership — has independently driven at least one substantial technical or research project from problem formulation through implementation, analysis, and write-up.
- Strong conceptual and experimental judgment — able to generate competing explanations, identify confounds, and design experiments that distinguish between them.
- Strong quantitative reasoning — comfortable interpreting noisy training results, making fair comparisons across runs, and choosing analyses suited to the question.
- Clear technical communication — able to explain precisely what a result shows, what it does not show, and what should be tested next.
- Direct experience with LLM supervised fine-tuning, reinforcement learning, LoRA, optimiser experiments, or training-dynamics research.
- Experience investigating model behaviour through behavior evaluations, model organisms, prompting, or steering.
- Experience with interpretability or representation analysis, such as activation probing, steering, or parameter-space analysis.
- A background in optimisation, deep-learning generalisation, statistics, or causal experimental design.
- Strong research writing and the ability to synthesise a scattered literature into clear hypotheses.
- Evidence of moving productively between high-level conceptual questions and hands-on empirical work.