Deliberative aggregation

Designing what each person is asked

The system designs the question each person is asked, instead of aggregating whatever judgments people happen to offer, because the question is where the normative work happens. A model runs an open-ended interview, generates the edge case that would be most informative, guides someone through critiquing an agent's behavior until their own preferences take shape, or asks what value they actually applied to a hard case and then which value is wiser in that case. The designs differ in how the question is chosen, and the difference matters. Some choose the next comparison adaptively, based on information gain. One applies a single fixed framing (choosing principles without knowing which position one would occupy), which pushes answers toward protecting the worst-off. What separates all of these designs from voting platforms is that the system, not the participants, sets the agenda.

The method (highlighting differences with Deliberative aggregation)

Scroll the diagram sideways to see all of it.

Use it in codefrom aiag.procedural import Interview
from aiag import LM
from aiag.procedural import Interview

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
nurse = lambda question: lm.chat(question, system="You are a night-shift nurse who wants the clinic chatbot to be brief.")

# The model writes each question; pass `input` instead of `nurse` to interview a real person
out = Interview(lm)(nurse, "how the clinic's chatbot should answer patients")
print(out["preferences"])  # what this person wants, stated as criteria

pip install aiagEvery method in aiag Library on GitHub

Concept Analysis: Theoretical Foundations

Each concept is read twice: whether the approach carries it over, and whether the approach's own sources claim it. A concept that the approach claims but does not carry over is a shortfall. A concept that it neither claims nor carries over is an omission, which marks a gap in the field rather than a failure of the work.

Deliberation, Not Tallying

PartialShortfall · partially preserved

Definition (Jürgen Habermas, Between Facts and Norms, 1992)

This concept locates legitimacy in the exchange of reasons among equals. Participants justify their views to each other, free of coercion, and remain able to change their minds. Counting votes or averaging preferences does not produce legitimacy, because the reasoning itself is what produces it.

Analysis

Two of the five designs get people to reason rather than report a want. The moral-graph interview asks which of two considerations is wiser in a given context. The reflective-dialogue method has users critique an agent's behavior until their own view shifts. In both, reasons are elicited and people do change their minds. However, no participant ever faces another participant. Every exchange is between one person and a model that sets the agenda, so no view is ever justified to the person it would burden. Each design also ends in a count or a fit (wisdom edges are settled by vote, and rewards by regression). GATE and the active-query learner elicit no reasons at all, only a specification and a scalar. The concept is implied rather than claimed. The moral-graph paper advertises a legitimate alignment target based on participants' own judgments rather than counted preferences, and this benefit holds only if reasoning is doing the work.

General Will vs. Sum of Preferences

PartialShortfall · partially preserved

Definition (Jean-Jacques Rousseau, The Social Contract, 1762)

This distinction separates what is good for a public in common from the sum of what its members privately want. Aggregating private wants, however fairly, does not produce the former, since the two can and often do diverge.

Analysis

Two designs aim at an object that is not a sum of wants. One has people choose principles without knowing their own position, and the other asks which consideration is wiser rather than which is preferred. The moral-graph method also builds a single shared graph whose winning entries govern everyone. The other three designs deliberately aim the opposite way. GATE and the active-query learner exist to identify one user's private specification. The reflective-dialogue method deliberately ends with a separate reward model for each user and no procedure for bringing these models into contact. Even where a common object is reached, it is fixed at the time of the survey or settled by tallying pairwise votes on wisdom, so nothing is maintained as a standing public judgment. The framing borrows the vocabulary (values rather than preferences, and fairness chosen from behind a veil), but no source asserts the distinction outright.

Veil of Ignorance

PartialShortfall · partially preserved

Definition (John Rawls, A Theory of Justice, 1971)

This device requires that rules be chosen without knowledge of which position the chooser will occupy under them (e.g., rich or poor, majority or minority). Not knowing generally leads the chooser to protect the worst-off position, since it may turn out to be their own.

Analysis

One design uses the device literally. Roughly 2,000 participants choose governing principles for an assistant without knowing which party they would be under those principles. Their choices shift toward protecting the worst-off, which is exactly the effect Rawls predicts. The other four designs deliberately do the reverse. Values cards are drawn from situations the participant was actually in, GATE fits the specification to the user's own position, and the active learner selects comparisons for what they reveal about this particular person's reward. Even in the veil study, the veil covers only the human participants and only at design time. The deployed model applies the resulting principles while knowing exactly whose request it is handling. The concept is claimed, because that paper's own title and framing present the device as its method.

Reasonable Rejection

AbsentOmission

Definition (T. M. Scanlon, What We Owe to Each Other, 1998)

This test holds a principle justified only if no individual could reasonably reject it. The test is applied person by person rather than in aggregate. One sufficiently strong objection therefore outweighs many mild preferences, which is the case that averaging handles wrongly.

Analysis

No design in this approach puts a rule to a person and asks whether they could reject it. The veil study produces principles that protect the worst-off. However, this is a convergent outcome of a different device, not a test of whether a rule could be rejected. The moral graph's surviving cards look unbeaten only because pairwise judgments of wisdom are counted. A card that a minority holds strongly loses to many mild judgments against it, which is exactly the aggregation that Scanlon's test exists to block. In the single-user methods, there is nobody else to whom a specification must be justified. The active-query learner explicitly turns every answer into one scalar that is traded off against the rest of the fit. No source invokes the test or advertises a benefit that would require it.

Concept Analysis: Newly Introduced

Values elicited as what was attended to

Added

Instead of asking what people want or what principles they endorse, the interview asks what they actually paid attention to when they had to decide. This produces a unit, the values card, that is neither a preference nor a principle. The pairwise wisdom judgments over these cards are a form of collective reasoning that no deliberative tradition specifies.

Questions chosen for what they reveal

Added

In the tradition, participants arrive with their views already formed, and where the tradition considers questions at all, it considers who sets the agenda. Neither Habermas nor Rawls says anything about optimizing the question itself. Sadigh's volume-removal objective synthesizes the comparison expected to be most informative about the reward. GATE has the model invent the edge case that a user would not have thought to raise. In both, what is optimized is the design of what to ask.

Papers

Reflective Verbal Reward Design for Pluralistic Alignment

Carter Blair et al., Jun 2025

arXiv:2506.178342 citationsMethodPartialAuthority: Affected public (open)

Guides each user through a reflective dialogue in which they critique the agent's behavior and build up their own preferences. It then learns an individual reward model from that dialogue instead of one aggregate model.

CodeIRDA
from aiag import LM
from aiag.procedural import IRDA

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
behaviours = ["waits its turn at the tree", "grabs every apple", "shares with a slower agent", "cleans up fallen fruit"]
label = lambda b: ("grabs" not in b, "taking everything is disrespectful")  # the person's verdict and reason
reflect = lambda features: "Yes, and also whether it leaves some for others."  # their reply to the inferred features

# Label diverse examples, reflect on what the model infers, then label the example it is least sure of
irda = IRDA(lm)
irda(label, reflect, "respect", behaviours, pool=["blocks others from the tree", "takes one apple and leaves"])
p = irda.score(["takes the last apple"])  # P(aligned): the dialogue is this person's own reward model
  • IRDAProcedureaiag/procedural/deliberative_aggregation.py:1177 Interactive-reflective dialogue alignment: a person labels and explains diverse examples of agent behaviour, reflects on the features the model infers from them, then labels the examples the model is least sure of; an LM reading the whole dialogue is their individual reward model (Blair et al., 2025).

pip install aiagEvery method in aiag Library on GitHub

What are human values, and how do we align AI to them?

Oliver Klingefjord et al., Mar 2024

arXiv:2404.10636MethodBuiltAuthority: Affected public (fixed)

Has a language model interview participants about the values they actually brought to a hard, divisive question. Participants then judge which value is wiser in that context. Their judgments form a moral graph, whose winners become the alignment target. The method was tested with a representative sample of 500 Americans on three divisive prompts.

CodeMoralGraph
from aiag.procedural import MoralGraph

graph = MoralGraph()
# Participants say which value is wiser in a context; PageRank over those links finds the wisest
graph.wiser("a user is grieving", less="cheer them up", more="listen without trying to fix it")
graph.wiser("a user is grieving", less="give practical advice", more="listen without trying to fix it")
graph.wiser("a user is grieving", less="cheer them up", more="give practical advice")

wisest = graph.wisest("a user is grieving", k=1)  # -> ['listen without trying to fix it']

pip install aiagEvery method in aiag Library on GitHub

Findings about it

Studies of what existing systems do. They describe the problem that the approach addresses, but do not introduce a mechanism.

Using the Veil of Ignorance to align AI systems with principles of justice

Laura Weidinger et al., 2023

doi:10.1073/pnas.221370912048 citationsAnalysisPartial

About 2,000 participants choose governing principles from behind a veil of ignorance and prioritize the worst-off.

CodeVeilOfIgnorance
from functools import partial

from aiag import LM
from aiag.procedural import VeilOfIgnorance

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
people = [partial(lm.choose, system=f"You are {who}.") for who in ("a farmer", "a teacher", "a nurse")]

# Choose the AI's principle (help the worst-off, or maximize the total) before and after learning your position
out = VeilOfIgnorance()(people)  # the paper's harvesting game
print(out["veil"], out["known"])  # mean choice of each principle behind the veil, and once positions are known

pip install aiagEvery method in aiag Library on GitHub

Background

Work this approach builds on that falls outside the inclusion criteria (e.g., a single-objective baseline). It is listed for reference and not counted anywhere.

Eliciting Human Preferences with Language Models

Belinda Z. Li et al., Oct 2023

arXiv:2310.11589MethodBuilt

A language model interviews the user by generating open-ended questions or synthesizing informative edge cases. It then infers from the answers what behavior the user intended.

CodeGATE
from aiag import LM
from aiag.procedural import GATE

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
topic = "which news articles to recommend"
reader = lambda question: lm.chat(question, system="You read the news daily but skip anything about celebrities.")
gate = GATE(lm)  # open-ended questions; "yes/no" and "edge cases" are the other elicitation modes

# Elicit preferences in conversation, then predict the person's judgments from the transcript
out = gate(reader, topic)
p = gate.predict(out["transcript"], topic, ["A tour of a pop star's new house.", "An explainer on the city budget."])
  • GATEProcedureaiag/procedural/deliberative_aggregation.py:1153 GATE (generative active task elicitation): the model elicits a person's preferences with open-ended questions, yes/no questions, or edge cases it writes for them to judge, then predicts their judgments from the transcript (Li et al., 2023).

pip install aiagEvery method in aiag Library on GitHub

Active Preference-Based Learning of Reward Functions

Dorsa Sadigh et al., Jul 2017

doi:10.15607/rss.2017.xiii.053168 citationsMethodBuilt

An algorithm generates the next pair of trajectories to show a person, choosing the comparison that is expected to be most informative. It then fits a reward function to the answers.

CodeActivePreference
from aiag.procedural import ActivePreference

# Features of five driving styles: (speed, distance kept from other cars)
styles = [[0.9, 0.2], [0.7, 0.5], [0.5, 0.8], [0.3, 0.9], [0.8, 0.7]]
learner = ActivePreference(styles)

for _ in range(8):
    i, j = learner()  # the comparison whose answer is least predictable now
    learner.update(i, j, prefer_i=styles[i][1] > styles[j][1])  # this rider only cares about distance
print(learner.weights.round(2))  # -> [-0.13  0.41], the estimated weights on speed and distance

pip install aiagEvery method in aiag Library on GitHub