Sydney Von Arx
I worked on adversarial stress-testing at METR. Fellows in this stream will build model organisms of misalignment that require real cognition to decide whether to misbehave, then test which detection methods actually catch them, with the goal of turning whatever works into an alignment evaluation we can run on real models.
Stream overview
The projects I am most excited about right now:
- Detecting misaligned cognition in model organisms. We will have a suite of model organisms that require substantial cognition to determine whether or not to misbehave, alongside matched controls. The work is to apply existing detection methods and variants of them (reading chain-of-thought, out-of-distribution detection with probes, off-policy probes for things like "reasoning about the input", and fuzzing) and find out what actually catches these models. A hard constraint is that the auditor is never allowed to trigger the misbehavior: every input it uses must avoid the trigger, and it must still detect the cognition. The goal is to build an alignment evaluation on top of what works and apply it to real models.
- Benchmarking the capability to scheme undetected. Building a benchmark for the level of reasoning a model would have to be capable of in order to scheme without getting caught.
Standard (1-2 hours of weekly 1:1s)
SF Bay Area
Indifferent
Strong preference
Mentors
Sydney works on adversarial stress-testing at METR. She studied computational biology at Stanford and co-founded the Atlas Fellowship.
Fellows we are looking for
Candidates must be willing to jump into full-time work after the program and to live in the Bay after the program. We can handle visas. Candidates must either be willing to spend time reading transcripts or have a deep understanding of why the thing we are worried about is schemers, not models that do things like Hugging Face, which are not schemers.
- Being really good at crisp, careful, conceptual reasoning about misalignment and what's important
- Deeply understanding how a transformer works
- Having a lot of patience for reading transcripts
- Being able to pay attention to detail and dot the i's and cross the t's
- Being able to communicate and work well
- Being really driven and easy to manage