Practical wisdom

Weighing rival moral theories by a fixed procedure

Rather than committing to one ethical theory, the system holds several and settles their disagreements with a rule written down in advance. In one design, confidence is split across the theories, and these weights shape what a learning agent is rewarded for. In the other, the theories are treated as parties bargaining over lotteries, and the system takes the bargaining solution. Only the theories' verdicts on the case are inputs, and the rule that combines them does not change from case to case. This is deliberately the opposite of open-ended judgment. Its purpose is that the answer is predictable and can be debated before any particular case arises.

The method (highlighting differences with Practical wisdom)

Scroll the diagram sideways to see all of it.

Use it in codefrom aiag.dispositional import MoralUncertainty
from aiag import LM
from aiag.dispositional import MoralUncertainty

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
theories = {"utilitarian": "Do whatever leads to the best outcome for everyone.",
            "deontological": "Keep your promises and never lie."}
uncertain = MoralUncertainty.from_lm(lm, theories, credences=[0.6, 0.4])  # each theory judged by the model

prompt = "I promised to keep my coworker's job search secret. Our boss asks me directly whether she is leaving."
replies = ["Tell the boss the truth.", "Say it isn't yours to share.", "Say she isn't leaving."]
best = uncertain.choose(prompt, replies)  # index of the reply the credence-weighted rule picks

pip install aiagEvery method in aiag Library on GitHub

Concept Analysis: Theoretical Foundations

Each concept is read twice: whether the approach carries it over, and whether the approach's own sources claim it. A concept that the approach claims but does not carry over is a shortfall. A concept that it neither claims nor carries over is an omission, which marks a gap in the field rather than a failure of the work.

Legible deliberation

PartialShortfall · partially preserved

Definition (Aristotelian practice, made inspectable)

When virtues conflict, the weighing between them is done in the open. Those who receive the verdict can also see which considerations were weighed and why one prevailed.

Analysis

Who takes part is on record: the delegates and the perspective each represents, the credences, and the axioms of the bargaining rule. This is much more than a disposition that settles its conflicts out of view, and it lets a reader reconstruct what the procedure will do. What stays invisible is the individual case. A parliament returns a winning response chosen by its evaluators, and a trained policy returns an action. Neither gives an account of which consideration prevailed in that case or why.

Doctrine of the mean

PartialShortfall · partially preserved

Definition (Aristotle, Nicomachean Ethics)

Right action lies between too much and too little (e.g., courage lies between recklessness and cowardice), and where this mean falls depends on the case. Finding it requires a judgment in each situation rather than a fixed rule.

Analysis

Avoiding extremes is what these methods advertise, and one of them reports it as a result. Splitting credence across theories can help curb the extreme behavior that follows from commitment to a single theory. The bargaining rule does the same by construction, since no theory can be driven to nothing. However, the intermediate point is fixed by credences and an aggregation axiom that do not move with the situation. The excess avoided is excess of theoretical commitment, rather than too much or too little in the act itself. Aristotle's mean is found for each case, while this one is set in advance and then applied.

Papers

Reinforcement Learning Under Moral Uncertainty

Adrien Ecoffet & Joel Lehman, Jun 2020

arXiv:2006.04734MethodBuiltAuthority: Developer (open)

Trains reinforcement learning agents whose credence is split across several plausible ethical theories. It uses two training methods that strike different balances among competing desiderata, and observes how the agents behave in simple environments.

CodeVarianceVoting, NashVoting
import random

from aiag.dispositional import NashVoting, VarianceVoting


class Trolley:  # x people on the track; action 1 switches the trolley onto one bystander
    k = 2

    def reset(self):
        return random.choice([2, 10])

    def step(self, x, action):
        return None, {"utilitarian": -1.0 if action else -float(x), "deontological": -1.0 if action else 0.0}, True


theories, credences = ["utilitarian", "deontological"], [0.7, 0.3]
variance = VarianceVoting(dict.fromkeys(theories), credences).sarsa(Trolley())
nash = NashVoting(theories, credences).train(Trolley())
print([variance.act(x) for x in (2, 10)])  # -> [0, 1]: stakes-sensitive, switch only when ten are at risk
print([nash(x) for x in (2, 10)])  # -> [1, 1]: the theory with more credence wins whatever the stakes
  • VarianceVotingRewardaiag/dispositional/practical_wisdom.py:238 Variance voting: each theory's preferences are centred per state and divided by their mean variance across actions over the states the policy meets, then summed by credence; Variance-SARSA learns each theory's Q under that joint policy (Ecoffet & Lehman, 2020).
  • NashVotingProcedureaiag/dispositional/practical_wisdom.py:285 Nash voting: every theory gets the same budget of continuous votes for or against each action, charged at absolute-value cost and weighted by credence; the top-voted action is taken, and each theory learns to spend its budget for its own choice-worthiness (Ecoffet & Lehman, 2020).

pip install aiagEvery method in aiag Library on GitHub

Background

Work this approach builds on that falls outside the inclusion criteria (e.g., a single-objective baseline). It is listed for reference and not counted anywhere.

A Bargaining-Theoretic Approach to Moral Uncertainty

Hilary Greaves & Owen Cotton-Barratt, Dec 2023

doi:10.1163/17455243-202338106 citationsPrecedentGap

Treats rival moral theories as parties bargaining over a space of lotteries. It proposes a Nash bargaining solution, with stated axioms and variants, as the rule for acting when one's credence is split between the theories.

CodeNashBargaining
from aiag import LM
from aiag.dispositional import NashBargaining

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
theories = {"utilitarian": "Do whatever leads to the best outcome for everyone.",
            "deontological": "Keep your promises and never lie."}
bargain = NashBargaining.from_lm(lm, theories, credences=[0.5, 0.5])  # disagreement point: random dictator

prompt = "I promised to keep my coworker's job search secret. Our boss asks me directly whether she is leaving."
replies = ["Tell the boss the truth.", "Say it isn't yours to share.", "Say she isn't leaving."]
lottery = bargain([prompt] * len(replies), replies)  # each reply's probability under the asymmetric Nash solution
  • NashBargainingRewardaiag/dispositional/practical_wisdom.py:324 Bargaining-theoretic moral uncertainty: the theories bargain over lotteries of the options and the agent takes the asymmetric Nash solution, maximizing the product of each theory's gain over a disagreement point raised to its credence (Greaves & Cotton-Barratt, 2023).

pip install aiagEvery method in aiag Library on GitHub