Andreea Bobu is an Assistant Professor at MIT in AeroAstro and CSAIL. She leads the Collaborative Learning and Autonomy Research Lab (CLEAR Lab), where they develop autonomous agents that learn to do tasks for, with, and around people. Her goal is to ensure that these agents' behavior is aligned with human expectations, whether they interact with expert designers or novice users. She obtained her Ph.D. in Electrical Engineering and Computer Science at UC Berkeley with Anca Dragan in 2023. Prior to her Ph.D. she earned her Bachelor’s degree in Computer Science and Engineering from MIT in 2017. She was the recipient of the Apple AI/ML Ph.D. fellowship, is a Rising Star in EECS and an R:SS and HRI Pioneer, and has won best paper award at HRI 2020 and the Emerging Research Award at the International Symposium on the Mathematics of Neuroscience 2023. Before MIT, she was also a Research Scientist at the AI Institute and an intern at NVIDIA in the Robotics Lab.

The mentors for this stream will be Andreea Bobu and Andi Peng, with Andreea Bobu being the primary mentor.

We are broadly interested in AI agents learning to do tasks for, with, and around humans. Our main research motivations are ensuring that these agents are value aligned with the humans they are meant to support, whether the human is an expert designer, a novice end user, or a stakeholder of the AI system. The work that we do involves reward learning, learning from (potentially multiple) kinds of human feedback, active learning, representation learning, quantifying misalignment.

We are broadly interested in AI agents learning to do tasks for, with, and around humans. Our main research motivation is ensuring that these agents are value aligned with the humans they are meant to support, whether the human is an expert designer, a novice end user, or a stakeholder of the AI system. This is challenging for many reasons. Values are notoriously difficult to specify to our AI agents. Not only that, but learning values (for instance, as reward functions) — which is the current popular alternative to specifying them by hand — is also difficult: 1) getting the right data to supervise the learning (via RLHF-style methods) is nontrivial because humans are imperfect, not infinitely queryable, and have unique and changing preferences and values; 2) the representations we choose to mathematically express values may themselves be misaligned, thus preventing us from ever being able to capture the “true” desired values; 3) reliably quantifying misalignment to be able to robustly tell when the AI system is safe for operation is still under explored. To tackle these challenges, in our research we bring cross-disciplinary expertise from deep learning, human-robot interaction, mathematical human modeling, cognitive psychology, and probabilistic uncertainty, with the hope of creating more aligned, generalizable, and robust learning algorithms and AI systems.

With this context in mind, research topics can be flexible depending on interests and fit but we am broadly interested in the following three thrusts of work (which you can also read more about on our lab website):

• Getting (enough of) the right data to align our AI agents’ values with humans. Typical RLHF methods treat humans as infinitely queryable oracles. While we do have large amounts of data on the internet, each individual human the AI agent will interact with will have unique and changing preferences, values, and biases that canned internet data alone may not reflect. Since we want AI agents to quickly adapt and align their existing values with each human user, how can we relax the infinitely queryable oracle assumption?

• How do we create data efficient algorithms that ask for the right input (maximize information gain) while not over-burdening the human (minimize human effort)? Can we make use of active learning techniques that combine these objectives in order to minimize the amount of queries required from the human?

• What kind of input do we want to ask humans for? Preference queries are low effort but contain relatively little information for the agent. Could we enhance the information gain by appending targeted explanations to the preferences (e.g. “I prefer A to B because A has less political content”)? Could we combine different feedback modalities (showing inputs like examples of correct behavior vs telling inputs like language corrections) for increased agent alignment?

• How do we make use of large pre-trained models, while still adapting to the specific human's needs? LLMs contain very powerful human priors that we have found to substantially reduce human burden when learning how to execute tasks from them. Can we make use of these LLM priors to amplify the data we receive from humans? Can we prompt LLMs to think about the reason underlying a human input or decision, and then use that causal reason to generate new situations where the human would likely behave in a similar way? Can we use LLMs to elicit preferences from humans?

• Aligning agent representations with humans for increased downstream value alignment. We show in recent work that if the representations we choose to express values are misaligned with those of humans, the downstream learned values will also be inevitably misaligned. Misaligned representations lead to unintended, unexpected, and even catastrophic behavior. How can we learn agent representations of the world that are aligned and in agreement with those of the humans they are cooperating with? Our previous efforts have looked at methods that learn representations one concept at a time vs all-at-once, but each approach has important tradeoffs: one concept at a time leads to more interpretable and structure-inducing representations, but they require the human to think of and explicate every dimension of their representation; all-at-once is more scalable and effortless but results in more entangled and less interpretable representations. Can we somehow get the best of both worlds: can we obtain more interpretable and disentangled representations while making it easier for the human to teach the AI agent about their representation? Moreover, if these representations are indeed interpretable, can we use them to facilitate smoother cooperation and communication between humans and their AI agent?

• Reliably quantifying misalignment. In order to even know when to stop current (potentially unsafe) behavior and initiate the process of alignment with the human, the AI system needs to be able to quantify misalignment. Our past work has looked at quantifying misalignment on small Bayesian models, but there is little principled work on quantifying misalignment on large models like LLMs. How can we quantify and detect misalignment by monitoring inconsistencies between the human’s feedback and the AI agent’s behavior? How can AI agents disentangle between inconsistencies due to incorrect models vs due to noisy human judgements? How should AI agents resolve that misalignment once detected? Should AI systems always assume that the human is right and thus they need to adjust their model, or should there be a debate process where the human and the agent arrive at a shared model of the world together?

The best candidates are those who are passionate about exploring how thinking about the human element (designers, end users, or stakeholders of AI) in the human-AI interaction equation can lead to building more aligned, generalizable, and robust learning algorithms and AI systems.

In terms of hard skills, ideal candidates should have one or more of the following:

• Previous research experience in machine learning (or related fields), ideally with published work.

• Strong software engineering background and familiarity with machine learning tools and techniques.

• Experience with running human subject experiments and statistical analyses is a plus.

As for soft skills:

• The ability to communicate clearly.

• Strong intrinsic motivation and curiosity for learning.

• A desire to publish an academic paper.

• A sense for how to build a story is useful, in that doing good impactful research also involves communicating said research in a way that the community is drawn to and compelled by.