Shi Feng & Eyon Jang
The stream will focus on conceptual and empirical work on scalable oversight. We will work towards making falsifiable claims about a model's deceptive intent, developing realistic model organisms, and studying generalizability of scalable oversight methods.
Stream overview
I'm generally interested in scalable oversight and creating realistic open-weight model organisms of misalignment to stress-test and improve scalable oversight methods. I'm currently interested in the following research directions:
- Etiology of model organisms and its impact on oversight. For example, how does the training recipe of a sandbagging model organism affect the generalizability of sandbagging detectors trained on it?
- Empirical science of intent misalignment. What evidence can we realistically acquire to attribute deceptive intent to model? I'm interested in creating benchmarks and testing debate type protocols to triage model behaviors.
- Easy-to-hard generalization of deception detectors.
High-touch (+2 hours of weekly 1:1s)
Washington, D.C.
Weak preference
Strong preference
Mentors
Shi Feng leads a research group working on oversight and control. He is an assistant professor at George Washington University. Prior to that, he was a postdoc in the NYU Alignment Research Group under Sam Bowman. He currently focuses on deception and collusion, with an emphasis on propensity and evaluation realism.
Fellows we are looking for
- Passionate about hybrid (conceptual & empirical) work on scalable oversight / control methods and threat modeling against them.
- The ability to articulate the crux of a (proposed) work and translating that into empirical experiments: what are the hidden assumptions? what hypothetical finding makes the idea more or less promising?
- The ability to look at the data and do careful manual qualitative analysis, e.g., reading debate transcripts.
- Comfortable with building scaffolds and various post-training processes.
Project selection
A research agenda document will be shared ahead of time with a short list of project ideas. The scholars can also brainstorm and pitch ideas that are aligned with the research agenda. We will decide on assignments in week 2.