Civilizational Alignment

我们希望将公共选择理论等用于分析大规模能动性与权力动态的工具,推广到机器智能能够与人类竞争的情境中。

研究小组概览

We're open to fellows suggesting projects, but here are some concrete ones that we'd support:

​

1. Characterizing AI pacing dynamics. What levers are available to different actors, and what are their effects?

​

2. Characterizing the space of possible good futures. Which tradeoffs are inevitable, and where could we look for huge Pareto improvements? E.g. Could sophisticated paternalism allow local control while avoiding disastrous outcomes? Under what conditions should we expect moral convergence?

​

1. Fleshing out what "political" natural selection will lead to under various UBI/governance schemes. What mechanisms might states use to tie representation to particular minds, and how might those be predictably hacked? I.e. how will the definition of "person" likely evolve, and then be spammed?

导师指导方式:

Standard (1-2 hours of weekly 1:1s)

项目期间的工作地点:

London

伦敦办公偏好:

Indifferent

伯克利办公偏好:

Indifferent

导师

Raymond Douglas
ACS Research, University of Toronto
,
高级研究员
Agent Foundations
Forecasting and Strategy
结构性风险与社会动态

我主要想研究:随着高度先进的 AI 不断发展,怎样的未来才是好的?这包括社会可能如何组织,以及未来可能出现哪些类型的 AI。我也会做一些关于语言模型心理特征的实证研究。我真正开始研究工作的契机是 MATS 4.0,当时我研究的是预测模型的能动性理论。

阅读更多
David Duvenaud
University of Toronto
,
副教授
Forecasting and Strategy
Capability and Propensity Evaluations
结构性风险与社会动态

David Duvenaud is an Associate Professor in Computer Science and Statistics at the University of Toronto, who now works mainly on problems related to _civilizational alignment_, i.e. understanding what it will take to keep states and institutions aligned to human interests post-AGI. He holds a Sloan Research Fellowship, a Canada Research Chair in Generative Models, and a CIFAR AI chair. His postdoc was done at Harvard University and his Ph.D. at the University of Cambridge. He is a Founding Member of the Vector Institute for Artificial Intelligence. In 2023-2024 he did a sabbatical at Anthropic, leading their Alignment Evaluations team, as well as research projects on jailbreaks and sabotage. He's also a co-chair of the Schwartz Reisman Institute for Technology and Society, a director of the AI Safety Foundation, and an advisor to AVERI. He has also received a Google Faculty Award, and best paper awards at both the Neural Information Processing Systems (NeurIPS) conference and the International Conference on Machine Learning (ICML).

阅读更多

我们希望招收的学员

我们欢迎各种背景的申请者。理想人选可能类似 Robin Hanson 或 David Friedman:既能熟练运用经济学等领域的分析工具,又深入了解人类历史、制度,以及人口、文化、国家和各类组织演变所承受的压力。

1. Conceptual clarity.  You should be comfortable:

​

- Articulating your position in ways that would be intelligible to someone who disagrees with you, 

- Operationalizing claims in terms of concrete differences in outcomes,

- Identifying cruxes,

- Working with multiple related definitions.

​

1. Writing and communication skills.  Including:

​

- Asking for feedback in structured ways,

- Being open to feedback at all stages of a project,   

- Providing status updates even on still-conceptually fuzzy directions 

​

1. Good at prioritizing, i.e.   

​

- Being willing to focus even if interesting asides come up,   

- Getting drafts out for feedback on a regular basis   

- Frequent meta-level discussion of what pursuits are likely worthwhile

项目遴选