Peter Henderson
Peter Henderson’s stream focuses on developing safe, aligned AI agents, with projects on scalable oversight rules informed by law and game theory, safe long-horizon exploration, and measuring “jagged” capability/safety frontiers. Scholars will join an independently driven, engineering-heavy research environment, collaborating with other MATS scholars and PhD students, with weekly 1:1s and active async mentorship.
Stream overview
I'd be interested in a variety of potential projects, but three directions that I'm excited by are:
- Building better AI agent rules for scalable oversight: In our recent work we showed that different models take different interpretation strategies when interpreting laws for AI. But there are still lots of things to figure out in making more complete specifications for AI agents. Can we leverage tools from law? What about reward specification and game theory?
- Safe long-horizon exploration for agents: As we create agents with the goal of discovery, the risks of reward mis-specification grow. How can we create good explorers with agency that nonetheless remain aligned over long horizons? Are there differential signs of misalignment in specific high-risk applications: AI R&D, bio research, cybersecurity research? How do we disentangle optimizing for long horizon tasks from implicit misaligned long-horizon goals (e.g., survival/agency versus task completion)? What if we want self-directed learning and exploration, how can we make this aligned in the long run?
- Measuring the jagged frontier: Jaggedness has been a problem, both for capabilities and safety. As the frontier becomes more jagged we might have seemingly highly capable models fail in surprising ways. How can we measure jaggedness of a model as a property? How can we leverage this in understanding safety and jailbreaking risks?
New York City
Mentors
Peter is an assistant professor at Princeton University, where he works on reinforcement learning, alignment, and law. He received a J.D. and Ph.D. in computer science from Stanford University.
Fellows we are looking for
Essential:
- ML engineering experience—specifically implementing and training models on, e.g., slurm clusters.
- Research experience: at least one project where you drove the research direction (first-author or equal contribution).
- Familiarity with basic reinforcement learning principles and algorithms.
Nice to have, but not necessary:
- Prior RL papers.
- Theory background.
- Game theory background
- Law background
Not a good fit:
- Scholars who only have high-level ML knowledge without hands-on implementation
- Those who need heavily structured guidance (I will provide guidance and the goal is to work as part of a larger team, but will still need to independently drive research between team meetings and 1-1s).
Project selection
Mentors in the group will pitch projects, and scholars will try ones they find interesting for a week. We'll iterate together at the end of week 1 and pick final assignments in week 2.