Anthropic

This stream is focused on reducing catastrophic risks from large language models (LLMs). Their research spans several areas:

1. Developing model organisms of misalignment, e.g. of deceptive alignment, to build a better understanding of what aspects of training are more likely to lead to deceptive alignment.
2. Finding tasks where scaling up models result in worse behavior (inverse scaling), to gain an understanding of how current training objectives actively incentivize the wrong behavior (e.g., alignment-faking, sycophancy or reward-tampering).
3. Improving the robustness of LLMs to red teaming (e.g., via constitutional classifiers, red teaming with language models or pretraining with human preferences or red teaming with best-of-n jailbreaks).
4. Control - techniques that aim to prevent catastrophic failures even if egregiously misaligned AIs attempt to subvert the techniques (see Ctrl-Z).
5. Scalable oversight – the problem of supervising systems that are more capable than human overseers

Advancing security through investigating adversarial machine learning, cybersecurity evals, and understanding currently possible real-world attacks

These projects involve running a large number of machine learning experiments, to gain empirical feedback on safety techniques and failures.

During the program, scholars meet weekly with their project mentors and collaborators. Some projects meet more often without mentors (e.g., daily standups with the peers on the project). Each project will have a primary mentor, who is also the main decision-maker on key milestones for the project and who is the default person to go to for feedback, advice, etc. Co-mentors also attend project meetings as needed and provide feedback throughout the program. Some project co-mentors can be as involved as the primary mentor.