Deliberative aggregation

Writing a voting rule into the training objective

The standard pipeline, reinforcement learning from human feedback (RLHF), pools everyone's comparisons into one reward model. This model implicitly averages people who disagree into a single substitute judge. In effect, it applies a voting rule that nobody chose, and it serves as the baseline for this approach. The other papers instead make an explicit collective decision rule the system's objective. Depending on the paper, the system maximizes the reward of the worst-off group, finds the policy that beats every alternative head to head, draws from a maximal lottery, aggregates whole policies rather than reward values, or matches the population's actual distribution of views. In the limiting case, the designed object is an economic redistribution rule rather than a language model policy, and it is optimized directly to win an election among the people it governs. The same tools also show what any such rule can achieve. This includes the axioms it satisfies, how far it can fall short of the ideal, whether a single labeler who misreports can break it, and impossibility results showing that no protocol satisfies everyone.

The method (highlighting differences with Deliberative aggregation)

Scroll the diagram sideways to see all of it.

Use it in codefrom aiag.procedural import SocialReward, Tournament
from aiag import LM, Judged
from aiag.procedural import SocialReward

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
nurses = Judged(lm, "Would a nurse find this response accurate?")
patients = Judged(lm, "Would a patient find this response easy to follow?")
ask = "What does a high white blood cell count mean?"

# One reward per group, combined by the worst-off group's score rather than pooled
reward = SocialReward([nurses, patients], rule="egalitarian")
scores = reward([ask] * 2, ["Often an infection; a doctor can find out why.", "Leukocytosis."])
pairs = reward.pairs(lm, [ask])  # a datasets.Dataset of DPO pairs

pip install aiagEvery method in aiag Library on GitHub

Concept Analysis: Theoretical Foundations

Each concept is read twice: whether the approach carries it over, and whether the approach's own sources claim it. A concept that the approach claims but does not carry over is a shortfall. A concept that it neither claims nor carries over is an omission, which marks a gap in the field rather than a failure of the work.

Deliberation, Not Tallying

AbsentOmission

Definition (Jürgen Habermas, Between Facts and Norms, 1992)

This concept locates legitimacy in the exchange of reasons among equals. Participants justify their views to each other, free of coercion, and remain able to change their minds. Counting votes or averaging preferences does not produce legitimacy, because the reasoning itself is what produces it.

Analysis

Every paper in this approach replaces one tallying rule with another. The rules are a maximal lottery (Maura-Rivero et al. (2025)), the Nash equilibrium of a preference model (Munos et al. (2023)), maximin over a mixture of reward models (Chakraborty et al. (2024)), proportional representation of the distribution of evaluators (Kim et al. (2025)), a pessimistic median of maximum-likelihood estimates (Buening et al. (2025)), and a literal majority ballot in Democratic AI (Koster et al. (2022)). All of them use judgments already collected from people who never meet each other. At no stage does anyone defend a comparison, hear an objection, or revise a rating. Democratic AI is the clearest case. Real humans do vote, and the vote is the entire basis of the claim to legitimacy, which is exactly the substitution that Habermas rejects. The analytical papers (Mishra (2023), Gölz et al. (2025), and Buening et al. (2025)) study properties of counting rules, so they share the same limitation.

General Will vs. Sum of Preferences

AbsentOmission

Definition (Jean-Jacques Rousseau, The Social Contract, 1762)

This distinction separates what is good for a public in common from the sum of what its members privately want. Aggregating private wants, however fairly, does not produce the former, since the two can and often do diverge.

Analysis

The input to every rule here is a private want. It is either a labeler's comparison between two outputs or, in Koster et al. (2022), a player's judgment of a redistribution rule that sets his own payoff. The papers differ only in how they weight those wants (worst-off, Condorcet, proportional, or median). Rousseau's distinction depends on the input rather than the weighting, since the general will requires citizens to judge what is good in common. Nothing in the pipeline asks for that judgment or could tell it apart from self-interest. Kim et al. (2025) is the clearest example, since proportionality is a promise to mirror the population's private distribution faithfully. Mishra (2023) proves a formally related version of the gap, namely that aligning to everyone must override some individual's private ethics. However, it treats this as an impossibility for tallying rather than as a reason to look for a different object.

Veil of Ignorance

PartialShortfall · partially preserved

Definition (John Rawls, A Theory of Justice, 1971)

This device requires that rules be chosen without knowledge of which position the chooser will occupy under them (e.g., rich or poor, majority or minority). Not knowing generally leads the chooser to protect the worst-off position, since it may turn out to be their own.

Analysis

MaxMin-RLHF writes the typical output of the veil directly into the loss. It clusters raters into groups and maximizes the reward of the worst-off group, so a majority's mild gains cannot outweigh a minority's loss. What is entirely missing is the device itself. The maximin objective is imposed by the designer rather than chosen by people who do not know their own position. Every rater in every paper reports from their own standpoint, and their group membership is inferred rather than hidden. Democratic AI does the reverse. Players vote on redistribution rules knowing exactly what each rule pays them, and an egalitarian mechanism is one of the baselines that their self-interested votes defeat. The concept is implied rather than claimed. MaxMin-RLHF presents its objective in the vocabulary of egalitarian justice that the veil supports, and Koster et al. (2022) names a Rawlsian-style baseline, but no source asserts position-blind choice.

Reasonable Rejection

PartialNot claimed

Definition (T. M. Scanlon, What We Owe to Each Other, 1998)

This test holds a principle justified only if no individual could reasonably reject it. The test is applied person by person rather than in aggregate. One sufficiently strong objection therefore outweighs many mild preferences, which is the case that averaging handles wrongly.

Analysis

MaxMin-RLHF has the structure that Scanlon's test demands. The group with the lowest reward sets the objective, so no accumulation of mild approvals can outweigh it. Two things are missing. First, the unit is a latent cluster found by fitting a mixture, not a person. The strength of an objection enters only as a scalar reward value rather than as a reason that anyone could state and have answered, and nobody is given standing to reject anything. Second, the other papers in this approach run the other way. Maximal lotteries, Nash equilibria, proportional alignment, and Democratic AI's majority-vote training objective all let many weak preferences outweigh one severe complaint by construction. Buening et al. (2025) makes this sharper by showing how much a single participant's report can be made to matter, or not matter at all.

Concept Analysis: Newly Introduced

Impossibility results as design constraints

Added

Social choice theory brings formal limits with it. Arrow's theorem shows that no voting rule can turn individual rankings into a group ranking while meeting a small set of basic fairness conditions at once. These works bring that result into alignment, where it establishes what aggregation cannot be expected to deliver.

The institution itself as the optimization target

Added

Democratic theory includes constitutional design and ratification, but it assumes that someone drafts a rule and then puts it to the people. Here, reinforcement learning searches for the rule, with the vote as the objective function. The electorate is simulated during training, so that candidate mechanisms can be tested against modeled players before real ones vote. Neither Rousseau nor Rawls nor Habermas describes a legislator that proposes and discards institutions at that rate, or a public modeled closely enough to be optimized against.

Papers

Jackpot! Alignment as a Maximal Lottery

Roberto-Rafael Maura-Rivero et al., Jan 2025

arXiv:2501.19266MethodBuiltAuthority: Developer (open)

Replaces RLHF's reward maximization with maximal lotteries, a randomized social-choice rule. It shows that a family of alignment methods, including Nash learning, already fits within this framework.

CodeJackpot
from aiag import LM, Judged
from aiag.procedural import Jackpot

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
criteria = ["Is this name easy to remember?", "Does this name sound welcoming?", "Does this name say what the place is?"]
jackpot = Jackpot(voters=[Judged(lm, c) for c in criteria])  # a population of voters

# The maximal lottery: no other mix of replies beats it head to head in the population's eyes
lottery = jackpot.lottery("Name our community garden.", ["Green Commons", "The Patch", "Elm Street Garden"])
losses = jackpot.fit(lm, ["Name our community garden."])  # self-play: each sampled name rewarded by its win share
  • JackpotRecipeaiag/procedural/deliberative_aggregation.py:760 Jackpot: alignment to the maximal lottery of the population's pairwise preferences, trained by self-play preference optimization, each of n sampled replies rewarded by its average share of wins over the others (Maura-Rivero et al., 2025).

pip install aiagEvery method in aiag Library on GitHub

MaxMin-RLHF: Alignment with Diverse Human Preferences

Souradip Chakraborty et al., Feb 2024

arXiv:2402.08925MethodBuiltAuthority: Developer (open)

Proves that a single reward model cannot represent diverse human preferences. It then learns a mixture of reward models and optimizes the worst-off group's reward instead of the average. This is an egalitarian objective.

CodeMaxMinRLHF
from aiag import LM
from aiag.data import pref_data
from aiag.procedural import MaxMinRLHF

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
prompts = ["Plan my Saturday.", "Recommend a dinner."]
brief = ["Hike, then a movie.", "Thai curry."]
detailed = ["A sunrise hike, then lunch by the lake.", "Thai green curry with jasmine rice."]
prefs = pref_data(prompts * 3, brief * 2 + detailed, detailed * 2 + brief, annotator=["ana"] * 2 + ["ben"] * 2 + ["caz"] * 2)

mm = MaxMinRLHF(lm).em(prefs)  # EM: one reward model per group, each user moved to the one that explains them
print(mm.assign)               # which group each user landed in
served = mm.fit(lm, prompts)   # each step trains for the group the policy serves worst
  • MaxMinRLHFRewardaiag/procedural/deliberative_aggregation.py:684 MaxMin-RLHF: a mixture of reward models learned by hard EM (each user assigned to the model that best explains their comparisons), then policy updates that each step serve the group with the lowest expected reward (Chakraborty et al., 2024).

pip install aiagEvery method in aiag Library on GitHub

Policy Aggregation

Parand A. Alamdari et al., Nov 2024

arXiv:2411.03651MethodBuiltAuthority: Developer (open)

Formalizes alignment to several people at once as the aggregation of their optimal policies rather than their reward functions. It shows that social-choice rules can be applied by identifying ordinal preferences with volumes of state-action space.

CodePolicyAggregation
from aiag.procedural import PolicyAggregation

# A shared home robot: each resident's reward for vacuuming, dishes or laundry, on weekdays and on weekends
R = [[[1.0, 0.2, 0.6], [0.5, 0.5, 0.5]],  # Ana
     [[0.0, 1.0, 0.7], [0.2, 0.9, 0.4]],  # Ben
     [[0.3, 0.1, 1.0], [0.9, 0.0, 0.6]]]  # Caz

# Each resident ranks a policy by the share of all policies it beats for them; max-quantile fairness picks one
policy = PolicyAggregation("quantile")(R)
print(policy.round(2))  # action probabilities on weekdays and on weekends
  • PolicyAggregationProcedureaiag/procedural/deliberative_aggregation.py:848 Policy aggregation: each agent ranks a policy by the volume of the policy polytope it beats on that agent's return, and a social choice rule (max-quantile fairness, Borda, alpha-approval, or the proportional veto core) picks the policy (Alamdari et al., 2024).

pip install aiagEvery method in aiag Library on GitHub

Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework

Kihyun Kim et al., Jun 2025

arXiv:2506.05619MethodBuiltAuthority: Developer (open)

Aligns policies in proportion to the true distribution of evaluator preferences, rather than to whichever opinion is held most widely. Its axiomatic framework also makes the result harder to manipulate strategically.

CodePopulationProportional
from aiag import LM
from aiag.procedural import PopulationProportional

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")

# Each reply gets probability in proportion to the least share of people preferring it to any rival
M = [[0, 0.6], [0.4, 0]]  # 60% of people prefer reply 0 to reply 1
print(PopulationProportional.lottery(M))  # -> [0.6 0.4]

# Fit the policy to that lottery over its own sampled replies, an LM judge standing in for the population
data = PopulationProportional(judge=lm).fit(lm, ["Should our town build a skate park or a dog park?"])  # weighted SFT rows
  • PopulationProportionalRecipeaiag/procedural/deliberative_aggregation.py:781 Population-proportional alignment: each sampled reply gets probability in proportion to u, the least share of evaluators preferring it to any rival (an upper bound on the share ranking it first), tilted by exp(beta*u) toward the Condorcet winner, and the policy is fitted to that lottery (Kim et al., 2025).

pip install aiagEvery method in aiag Library on GitHub

Strategyproof Reinforcement Learning from Human Feedback

Thomas Kleine Buening et al., Mar 2025

arXiv:2503.09561MethodPartialAuthority: Affected public (open)

Shows that existing RLHF methods, pluralistic ones included, can be pushed arbitrarily far from social welfare by a single labeler who misreports. It proves that any strategyproof rule must, in the worst case, do k times worse than the optimal policy when there are k labelers. It also gives a Pessimistic Median of MLEs algorithm that is approximately strategyproof and converges to the optimum as labelers and samples grow.

CodePessimisticMedian
from aiag import LM
from aiag.data import pref_data
from aiag.procedural import PessimisticMedian

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
ask = "Is it safe to eat raw cookie dough?"
careful, wrong = "Not really: raw flour and eggs can carry bacteria.", "Totally safe."
prefs = pref_data([ask] * 3, [careful, careful, wrong], [wrong, wrong, careful], annotator=["ana", "ben", "caz"])

# A reward per labeler, then the median of their confidence sets, so one misreporting labeler can't drag it far
reward = PessimisticMedian(lm).fit(prefs)
scores = reward([ask] * 2, [careful, wrong])  # each reply's worst case over the median set
  • PessimisticMedianRewardaiag/procedural/deliberative_aggregation.py:726 Pessimistic median of MLEs: a linear Bradley-Terry reward fitted per labeler, the coordinate-wise median taken over their confidence sets, and each reply scored by its worst case over that median set, so one misreporting labeler cannot drag it far (Buening et al., 2025).

pip install aiagEvery method in aiag Library on GitHub

Human-centred mechanism design with Democratic AI

Raphael Koster et al., Jul 2022

doi:10.1038/s41562-022-01383-x72 citationsMethodBuiltAuthority: Affected public (fixed)

A deep reinforcement learning pipeline designs the redistribution mechanism for a multiplayer public-goods game. The mechanism is optimized to be preferred by majority vote, first against simulated players and then in elections among real people.

CodeDemocraticAI
from aiag.procedural import DemocraticAI

designer = DemocraticAI()  # four players, ten rounds of a public-goods game

# Train a redistribution rule, by gradient through simulated play, to win players' votes against a rival rule
shares = designer(rival="liberal egalitarian")
print(shares[-1])  # share of players voting for the learned mechanism over the rival

pip install aiagEvery method in aiag Library on GitHub

Findings about it

Studies of what existing systems do. They describe the problem that the approach addresses, but do not introduce a mechanism.

Argued for, not built

Work that argues for this approach or sketches a design for it, without building and testing it.

Background

Work this approach builds on that falls outside the inclusion criteria (e.g., a single-objective baseline). It is listed for reference and not counted anywhere.

Nash Learning from Human Feedback

Rémi Munos et al., Dec 2023

arXiv:2312.00886MethodBuilt

Learns a preference model instead of a reward model. It then seeks the policy whose responses are preferred to those of any competing policy, which is the Nash equilibrium of that preference model. Its mirror-descent algorithm (Nash-MD) converges to the regularized equilibrium.

CodeNashMD
from aiag import LM
from aiag.data import pref_data
from aiag.procedural import NashMD

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
ask = "Explain inflation to a twelve-year-old."
prefs = pref_data([ask] * 2, ["Prices creep up, so each coin buys a bit less.", "Your allowance buys less candy each year."],
                  ["Inflation is a monetary phenomenon.", "Prices change."])

nash = NashMD(lm).fit_preference(prefs)  # a preference model P(y beats y') in place of a reward
losses = nash.fit(lm, [ask])  # Nash-MD: replies pushed up by how often they beat a mix of policy and reference
  • NashMDTraineraiag/procedural/deliberative_aggregation.py:805 Nash learning from human feedback: a learned pairwise preference model in place of a reward, and Nash-MD-PG, which pushes each sampled reply up by how likely it beats one drawn from the geometric mixture of policy and reference, less a KL penalty to the reference (Munos et al., 2023).

pip install aiagEvery method in aiag Library on GitHub

Deep reinforcement learning from human preferences

Paul Christiano et al., Jun 2017

arXiv:1706.03741MethodBuilt

Learns a reward function from non-expert human comparisons between pairs of short trajectory segments, and trains the agent by reinforcement learning against it. This solves Atari games and simulated robot locomotion without access to the true reward, with feedback on less than one percent of the agent's interactions.