Apollo Research - Monitors

我们将延续上一期研究,继续探索如何在复杂的智能体场景中构建黑盒监测器,识别模型密谋行为。具体来说,我们会扩大数据集和微调工作,详见可扩展监测研究议程。

​

下一阶段很可能聚焦自动化的红队与蓝队迭代对抗。目前我们正在训练蓝队,之后会训练红队,并在本研究方向中尝试闭环协作,让双方同步训练。

研究小组概览

本研究方向将专注于构建高质量的监测器,以识别模型密谋行为。

​

前两期研究中,我们先研究了宪法式监测器,并据此完成了一篇 ICML 会议论文。

之后,我们研究了经过微调的审议式监测器,相关成果很可能会被 NeurIPS 会议接收。

​

在可预见的未来,我们将遵循可扩展监测研究议程开展工作。

导师指导方式:

Standard (1-2 hours of weekly 1:1s)

项目期间的工作地点:

London

伦敦办公偏好:

Indifferent

伯克利办公偏好:

Strong preference

导师

Marius Hobbhahn
Apollo Research
,
首席执行官
Misalignment Science
AI Control and Monitoring
Forecasting and Strategy
Capability and Propensity Evaluations

Marius Hobbhahn 是 Apollo Research 的首席执行官,并领导该组织的监控团队。Apollo 是一家 AI 安全研究机构,重点研究密谋行为、评估,以及控制和监控。他入选了 2025 年《TIME》AI 领域百大人物榜单。创立 Apollo 之前,Marius 攻读贝叶斯机器学习博士学位,并在 Epoch 从事 AI 预测研究。

阅读更多

我们希望招收的学员

  • You like quick empirical iteration and direct feedback loops. I think candidates who did well in the past were good at producing a high volume of output, running many experiments in parallel and keeping track of what the most important next steps are at any given point in time.
  • You enjoy tinkering with LLMs, e.g. prompting, building basic LM agents, and fine-tuning.
  • You are happy to build datasets and synthetic data generation pipelines. We found that a substantial amount of initial work is required for the data generation process.
  • I expect that you will spend 20% on conceptual work (e.g., think about which environments could work or what techniques to try) and 80% on hands-on empirical work (e.g., implementing and running experiments).
  • I prefer that scholars focus 100% of their work time on the project and not pursue any side projects. In general, I’m happy to support highly ambitious scholars who want to make a lot of progress during MATS. In the past, people have described my stream as "intense, but in a good way".
  • You like quick empirical iteration and direct feedback loops. I think candidates who did well in the past were good at producing a high volume of output, running many experiments in parallel and keeping track of what the most important next steps are at any given point in time.
  • You enjoy tinkering with LLMs, e.g. prompting, building basic LM agents, and fine-tuning.
  • You are happy to build datasets and synthetic data generation pipelines. We found that a substantial amount of initial work is required for the data generation process.
  • I expect that you will spend 20% on conceptual work (e.g., think about which environments could work or what techniques to try) and 80% on hands-on empirical work (e.g., implementing and running experiments).
  • I prefer that scholars focus 100% of their work time on the project and not pursue any side projects. In general, I’m happy to support highly ambitious scholars who want to make a lot of progress during MATS. In the past, people have described my stream as "intense, but in a good way".

项目遴选

学员将参与黑盒监测相关的子项目。详情请见此处。