Joshua Engels
I'm interested in better understanding and controlling how post-training causes alignment-relevant behavior. This is a pretty broad area, and I’m open to many approaches to these problems! Potential areas of study / methods of attack might include model organisms, training run science/ablations, root causing strange behaviors, or studying how best to robustly induce behaviors or values or beliefs into models.
Stream overview
I’m open to fellows picking projects within the broad area of “understanding and controlling how post-training causes alignment-relevant behavior”. I’ll pitch some projects at the start of the program, and fellows can pitch some too, and we can decide together which seem most promising.
Potential future projects:
- Can we quantify how a model’s persona shifts over post-training? E.g. after SFT is it more “PSM” flavored, but does it then get less “PSM” flavored after a lot of outcome based RL?
- Are there examples of surprising generalization in open source post-training pipelines we can study and characterize?
- Can we train models to have obfuscated CoTs? If so, we can test our various ways of monitoring / interpreting this model. If not, a strong negative result is interesting in and of itself.
Past related projects I’ve worked on:
Standard (1-2 hours of weekly 1:1s)
London
No preference for this location
Strong preference
Mentors
Josh is a research scientist on the AGI Safety and Alignment team at Google DeepMind, where he currently works on post-training science and alignment stress testing. Previously, he was a PhD student in Max Tegmark's group working on mechanistic interpretability.
Fellows we are looking for
- Research experience: at least one project where you drove the research direction (first-author or equal contribution).
- Fast iterators: You are good at derisking: quickly figuring out whether projects will work or not. You are calibrated on your work and can e.g. figure out whether to dig into a result you are skeptical of or move on to a new experiment.
- Familiarity with machine learning and LLMs