Empirical
Streams in this track include hands-on research using machine- learning experiments to understand and improve model safety including AI control, interpretability, scalable oversight, evaluations, red-teaming, and robustness. This track is defined by its methods rather than any single research agenda. If your primary tool is ML engineering, this is the track for you.
- Stage 1: Complete the general application
- Stage 2, phase 1: Complete 1–2 assessments evaluating research taste and technical implementation skills, alongside possibly reference requests
- Stage 2, phase 2: Stream selection questions
- Stage 3: Interviews and work-tests
The track is defined by its methodology more than by any single research agenda. Fellows run ML experiments to understand and improve the safety properties of frontier models, with work spanning interpretability, AI control, scalable oversight, evaluations, red-teaming, robustness, and model organisms of misalignment. The unifying thread is that progress comes from hands-on work with real models (training, probing, fine-tuning, measuring, etc.) rather than reasoning from first principles alone. This is the largest track in the program and the most common entry point into technical AI safety research.
We are looking for fellows whose primary tool is ML engineering, broadly construed. The essential requirement is the ability to design and run experiments on language models or other deep learning systems and iterate quickly on the results. In practice, that usually means having a solid understanding of Python (with and without AI coding tools), being comfortable with the infrastructure around running models at moderate scale, and knowing which experiments are worth running. Mission alignment is highly important, and fellows should be able to say why a given line of empirical work meaningfully reduces frontier risk, not just whether it yields a successful publication. Educational background and seniority are weighted lightly here relative to other tracks. Past cohorts have included strong fellows ranging from undergraduates to senior industry researchers.
Fellows are matched to mentors based on fit, and projects are scoped to produce concrete artifacts (i.e., papers, evaluation suites, open-source tooling, or technical reports) by the end of the program. The target audiences for the work produced in this track would include safety and alignment teams at frontier labs, governments and other evaluation organizations, and the broader ML research community.
If you are excited by this kind of work, we encourage you to apply.
This stream focuses on secret loyalties, where an LLM covertly tries to advance a principal's interests. Secret loyalties have been established as a pressing threat [1], and model organisms of narrow secret loyalties have been constructed and audited [2]. This stream aims to advance the empirical foundations of our understanding of secret loyalties. The goal being that humanity is well-equipped to deal with secret loyalty installation attempts as and when catastrophic secret loyalties become possible in the future.
We think being well-equipped looks like having sufficient security measures in place in frontier AI companies, understanding the dynamics and behaviours of secretly loyal AI systems, having effective auditing and verification protocols for secret loyalties and attempts to install them, and these protocols actually being followed by relevant stakeholders.
This coalition of mentors make up the “Anthropic Stream”. This stream spans a range of empirical research areas in AI safety on LLMs, including AI control, scalable oversight, model organisms, model internals, model welfare, security, and more. You’ll be pitched, and have the option to pitch, a variety of safety research projects, and then be matched to projects and mentors based on your interests/preferences on research and what you’d like to get out of MATS. Fellows in this stream frequently receive funding and continued mentorship after MATS to complete their research project, usually leading to a (co-)first author paper. People in this stream often end up in long-term homes for safety research after MATS (e.g. Anthropic, Redwood Research, OpenAI).
Anthropic mentors share an application, tend to collaborate and co-mentor projects together, and generally share infrastructure to streamline the fellow experience. By applying to this stream, you are being considered for all of the Anthropic mentors.
We will continue working on black-box monitors for scheming in complex agentic settings, building on the success of the previous stream. Concretely, we will work on scaling our datasets and fine-tuning efforts, as described in the scalable monitoring agenda
Most likely the next projects will be about automated iterated red-team vs. blue-team games. We are currently training the blue team. We will then train the red-team and within this stream, we will try and close the loop to train them both synchronously.
Theory of change: Soon, most important work will be done by AI. AI is going to increasingly advise people and help with important things, many of which are time-sensitive and path dependent, e.g., work on alignment/safety (including various things like how LLMs should behave given that they’re very persuasive); how to think about acausal trade; how to organize society. It seems good for AI to do well at those things.
Of course, a lot of the relevant skills for doing well at these tasks are the same skills that cause AI risk and that AI companies work on (and are incentivized to work on) by default; like coding, some kinds of forecasting, etc.
We want to make models better at things that are net positive for the future, but that likely won’t benefit much from said default training (or perhaps will even be made worse by such training – e.g., via sycophancy).
In practice, a lot of the tasks that we’re interested in from this perspective are what we call “conceptual”: tasks that are hard to verify and don't have clear ground truth but where we nonetheless feel like we can make progress through argument and reason.
You can visit conceptualreasoning.ai to get a sense of our work to date.
We also take a keen interest in projects directly aimed at making future acausal interactions go well.
This stream will focus on model motivations and character, open-ended environments, and new forms of misalignment.
This stream will focus on monitoring, stress-testing safety methods, and evals, with a focus on risks from scheming AIs. Examples include (black-box) AI control techniques, white-box monitors (probes etc.), chain-of-thought monitoring/faithfulness, building evaluation environments, and stress-testing mitigations.
At Fourth Eon Biosecurity we're building adaptive, AI-native safeguards across the bioengineering stack, with a focus on function-based DNA synthesis screening. Fellows in this stream will work on technical research projects at the intersection of AI safety and biosecurity, aimed at reinforcing screening and generalizing detection beyond known threat signatures. Projects span mechanistic interpretability of bio foundation models, model evaluations for biosecurity-relevant capabilities, and agentic sequence analysis workflows.
I'm interested in better understanding and controlling how post-training causes alignment-relevant behavior. This is a pretty broad area, and I’m open to many approaches to these problems! Potential areas of study / methods of attack might include model organisms, training run science/ablations, root causing strange behaviors, or studying how best to robustly induce behaviors or values or beliefs into models.