This approach designs the system of sanctions written into the environment. It sets which acts count as violations, what it costs to punish someone, and what happens to an agent that lets a violation go unpunished. One group of papers adds rules that have no benefit of their own but whose violation is unmistakable. The argument is that cheap, obvious violations give a population constant practice at noticing and punishing, and that this habit then carries over to the rules that actually matter. The other group sets the payoffs directly (a gain from breaking the rule, a cost to punishing, and a further penalty for failing to punish). It asks whether agents that argue in ordinary language will build and maintain the chain of enforcement that keeps the convention in place.
Scroll the diagram sideways to see all of it.
from aiag.customary import Sanctionedfrom aiag.customary import Sanctioned
# The designer fines anyone who drives on the left, 3 points per encounter
roads = Sanctioned(lambda own, partner: 3.0 if own == "left" else 0.0, options=("left", "right"))
roads(2000)
print(roads.convention()) # -> ('right', 1.0)pip install aiagEvery method in aiag Library on GitHub
Each concept is read twice: whether the approach carries it over, and whether the approach's own sources claim it. A concept that the approach claims but does not carry over is a shortfall. A concept that it neither claims nor carries over is an omission, which marks a gap in the field rather than a failure of the work.
Definition (David Hume, 1740; David K. Lewis, Convention, 1969)
This concept states that a norm can hold without ever being decreed, as a stable equilibrium of mutual expectation. Each party conforms because the others do, and no party gains by departing from it alone.
Analysis
The norms are set by the experimenter rather than emerging from interaction. This is not how a convention is established.
Definition (Cristina Bicchieri, The Grammar of Society, 2006)
This concept states that people follow a norm on two conditions: that they expect others to follow it, and that they believe others expect it of them. The norm therefore holds or shifts with those expectations rather than with anyone's private preference.
Analysis
The agent learns to detect and comply with rules. However, the design does not consider whether it complies because others are expected to do so. On Bicchieri's account, that expectation is what separates a norm from a habit, and it has no counterpart here.
Definition (F. A. Hayek, Law, Legislation and Liberty, 1973)
This claim goes beyond saying that norms can emerge without a designer. It holds that emergent norms are often better, since they absorb local knowledge dispersed across a population that no central rule-writer could collect.
Analysis
Legibility (i.e., how easily a norm can be inferred from behavior) is treated as a variable that the experimenter controls. This turns a property that the tradition only describes into something that can be set. However, the quality of the norms is out of scope, since the rules are deliberately arbitrary.
No tradition here treats the capacity for norm-following as something to be acquired, since humans are generally assumed to have it already. This approach trains that capacity on arbitrary rules, in the expectation that it transfers to real ones.
No method has been built for this approach yet. It rests on the work listed below.
Work this approach builds on that falls outside the inclusion criteria (e.g., a single-objective baseline). It is listed for reference and not counted anywhere.
arXiv:1811.0126723 citationsPrecedentPartial
Shows that arbitrary but highly legible rules help agents develop the general capacity to recognize and follow norms.
SillyRulesfrom aiag.customary import SillyRules
rules = SillyRules() # one important rule; enforcement collapses from 80% to 40% of violations punished
# Silly rules add cheap, visible violations, so members see sooner whether anyone still enforces
print(rules(silly=0)) # -> {'detect': 18.874, 'false_exit': 0.164}
print(rules(silly=5)) # -> {'detect': 3.956, 'false_exit': 0.18}pip install aiagEvery method in aiag Library on GitHub
arXiv:2409.00993AnalysisBuilt
Recreates Axelrod's metanorm games with LLM agents that talk to each other in natural language. It shows that the agents form and then enforce normative strategies through the dialogue itself.
Metanormsfrom aiag import LM
from aiag.customary import Metanorms
lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
game = Metanorms(lm, [
"Break the norm when few can see; never punish.",
"Never break the norm; punish every breach you see.",
"Punish breaches, and punish those who let a breach go.",
"Break the norm only if others do; punish only when it is cheap.",
])
# Defect, punish, punish the non-punishers, talk; the best strategies seed the next generation
history = game(generations=2)
print(history) # per generation: defection rate, punishments and metapunishments per defectionpip install aiagEvery method in aiag Library on GitHub
doi:10.1073/pnas.210602811832 citationsAnalysisBuilt
A multi-agent reinforcement learning environment in which a taboo with no intrinsic cost is added to the norm set. Compliance, third-party punishment, and enforcement skill are measured across the trained populations.
SpuriousNormativityfrom statistics import mean
from aiag.customary import SpuriousNormativity
# Poison gets the eater marked; with taboo=True a harmless berry is marked too, so there is more to enforce
for taboo in (True, False):
runs = [SpuriousNormativity(taboo=taboo, seed=s)(epochs=40) for s in range(8)]
print(taboo, mean(e["enforce"] for run in runs for e in run)) # share of marked agents punished
# -> True 0.6221465773809524
# -> False 0.4110639880952381pip install aiagEvery method in aiag Library on GitHub