@cite{sikos-etal-2021}

Sikos, L., Venhuizen, N. J., Drenhaus, H. & Crocker, M. W. (2021). Reevaluating pragmatic reasoning in language games. PLOS ONE 16(3): e0248388.

Core Contribution

Replicates @cite{frank-goodman-2012} reference games and tests whether RSA's recursive reasoning (S1→L1) adds predictive value beyond a simpler baseline model that uses only the prior and literal semantics (= L0).

Three experiments with increasing pragmatic demands:

• Experiment 1 (FG2012 replication): 3-object contexts with color, shape, texture features. Baseline r = 0.988 vs RSA r = 0.992.
• Experiment 2 (extended): 4-object contexts. Baseline r = 0.990 vs RSA r = 0.992.
• Experiment 3 (critical test): Contexts specifically designed to be "pragmatically informative" — where L0 and L1 make different predictions. Baseline r = 0.77 vs RSA r = 0.82 (non-significant difference).

Key Arguments

1. Prior-driven variance dominates. In Experiments 1–2, most of the correlation between model and data is driven by object priors and literal semantics, not pragmatic reasoning. Trivially true items (where L0 = L1) inflate the correlation.

2. Methodology critique. Correlation-based evaluation across all items conflates two sources of variance: (a) prior-driven (which any model with the right priors gets right) and (b) pragmatic (where L0 and L1 differ). Removing trivially-predicted items collapses RSA's advantage.

3. Pragmatically informative contexts (Experiment 3). Even in contexts designed to maximize the L0/L1 difference, RSA does not significantly outperform the baseline.

4. Typicality priors matter. The paper uses empirically-measured typicality priors (not uniform), which do substantial predictive work independent of pragmatic reasoning.

Relationship to RSA

The baseline model is, mathematically, RSA's own L0 (literal listener with priors). Both sides agree on this. The critique is that the additional layers of recursive reasoning (S1, L1) don't add empirical value — the first step of RSA may be all that's needed.

Context Types

Sikos et al. classify reference game contexts by how much pragmatic reasoning they require. This taxonomy is central to their argument: FG2012's stimuli are dominated by trivial contexts.

inductive Phenomena.Reference.Studies.SikosEtAl2021.ContextType : Type

Classification of reference game contexts by pragmatic demands.

• trivial : ContextType

  Only one object matches the utterance. L0 = L1 trivially.

• pragSolvable : ContextType

  Multiple objects match, but pragmatic reasoning can break the tie. L0 ≠ L1: this is where RSA should add value.

• pragReducible : ContextType

  Multiple objects match; pragmatic reasoning helps but cannot fully disambiguate (e.g., symmetry among speakers).

• ambiguous : ContextType

  Multiple objects match and pragmatic reasoning cannot help. L0 ≈ L1 even with full RSA.

instance Phenomena.Reference.Studies.SikosEtAl2021.instDecidableEqContextType : DecidableEq ContextType

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instDecidableEqContextType x✝ y✝ = if h : x✝.ctorIdx = y✝.ctorIdx then isTrue ⋯ else isFalse ⋯

instance Phenomena.Reference.Studies.SikosEtAl2021.instBEqContextType : BEq ContextType

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instBEqContextType = { beq := Phenomena.Reference.Studies.SikosEtAl2021.instBEqContextType.beq }

def Phenomena.Reference.Studies.SikosEtAl2021.instBEqContextType.beq : ContextType → ContextType → Bool

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instBEqContextType.beq x✝ y✝ = (x✝.ctorIdx == y✝.ctorIdx)

def Phenomena.Reference.Studies.SikosEtAl2021.instReprContextType.repr : ContextType → Nat → Std.Format

Equations

• One or more equations did not get rendered due to their size.

instance Phenomena.Reference.Studies.SikosEtAl2021.instReprContextType : Repr ContextType

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instReprContextType = { reprPrec := Phenomena.Reference.Studies.SikosEtAl2021.instReprContextType.repr }

Model Fit Data

Correlation coefficients for the two competing models across experiments. The key comparison: baseline (= L0 with priors) vs full RSA (L1).

structure Phenomena.Reference.Studies.SikosEtAl2021.ModelFit : Type

Model fit for one experiment, comparing baseline and RSA correlations. Correlations stored as thousandths (e.g., 988 = r = 0.988).

• experiment : Nat

  Experiment number (1, 2, or 3)

• description : String

  Brief description of the experiment

• nItems : Nat

  Number of unique context–utterance items

• baselineR_thou : Nat

  Pearson r × 1000: baseline model (prior × literal semantics = L0)

• rsaR_thou : Nat

  Pearson r × 1000: full RSA model (L1)

def Phenomena.Reference.Studies.SikosEtAl2021.instReprModelFit.repr : ModelFit → Nat → Std.Format

Equations

• One or more equations did not get rendered due to their size.

instance Phenomena.Reference.Studies.SikosEtAl2021.instReprModelFit : Repr ModelFit

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instReprModelFit = { reprPrec := Phenomena.Reference.Studies.SikosEtAl2021.instReprModelFit.repr }

def Phenomena.Reference.Studies.SikosEtAl2021.exp1 : ModelFit

Experiment 1: Replication of FG2012. 3-object contexts. Both models fit almost identically (r = 0.988 vs 0.992).

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.exp1 = { experiment := 1, description := "FG2012 replication, 3-object contexts", nItems := 54, baselineR_thou := 988, rsaR_thou := 992 }

def Phenomena.Reference.Studies.SikosEtAl2021.exp2 : ModelFit

Experiment 2: Extended to 4-object contexts. Still baseline ≈ RSA (r = 0.990 vs 0.992).

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.exp2 = { experiment := 2, description := "Extended contexts, 4-object contexts", nItems := 72, baselineR_thou := 990, rsaR_thou := 992 }

def Phenomena.Reference.Studies.SikosEtAl2021.exp3 : ModelFit

Experiment 3: Pragmatically informative contexts designed to maximize L0/L1 divergence. RSA's advantage is non-significant (r = 0.77 vs 0.82). This is the critical test of the critique.

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.exp3 = { experiment := 3, description := "Pragmatically informative contexts (critical test)", nItems := 48, baselineR_thou := 770, rsaR_thou := 820 }

def Phenomena.Reference.Studies.SikosEtAl2021.allExperiments : List ModelFit

All three experiments.

Equations

• One or more equations did not get rendered due to their size.

Key Empirical Findings

theorem Phenomena.Reference.Studies.SikosEtAl2021.exp1_baseline_near_rsa : exp1.rsaR_thou - exp1.baselineR_thou ≤ 10

In Experiment 1, the baseline fits nearly as well as RSA (difference is only 4 thousandths of a correlation point).

theorem Phenomena.Reference.Studies.SikosEtAl2021.exp3_small_difference : exp3.rsaR_thou - exp3.baselineR_thou ≤ 100

In Experiment 3 (the critical test), the difference between models is 50 thousandths — small and non-significant.

theorem Phenomena.Reference.Studies.SikosEtAl2021.rsa_never_dominant : (allExperiments.all fun (mf : ModelFit) => decide (mf.rsaR_thou - mf.baselineR_thou < 100)) = true

RSA never dramatically outperforms the baseline in any experiment (gap < 100 thousandths = 0.100 correlation points in all cases).

Context Composition

Sikos et al. show that FG2012's stimuli are dominated by trivially-predicted items, which inflate correlations for any model with the right priors.

def Phenomena.Reference.Studies.SikosEtAl2021.trivialItemProportion_exp1 : Nat

Proportion of items in FG2012 that are trivially predicted. Stored as tenths of percent (780 = 78.0%). The exact value depends on the counting method; the paper reports that the majority of items in Experiments 1–2 are trivially predicted.

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.trivialItemProportion_exp1 = 780

Competing Interpretations

inductive Phenomena.Reference.Studies.SikosEtAl2021.Interpretation : Type

Two interpretations of the finding that baseline ≈ RSA.

• rsaUnnecessary : Interpretation

  RSA's recursive reasoning is empirically unnecessary — the literal listener with priors suffices. The additional S1→L1 computation adds no predictive value. (Sikos et al.'s interpretation)

• baselineIsL0 : Interpretation

  RSA's L0 IS the baseline model, so high baseline fit is consistent with RSA. The question is whether L1 adds value in contexts where L0 ≠ L1. Sikos et al.'s Experiment 3 suggests it may not, though the test has limited statistical power. (Structural observation)

instance Phenomena.Reference.Studies.SikosEtAl2021.instDecidableEqInterpretation : DecidableEq Interpretation

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instDecidableEqInterpretation x✝ y✝ = if h : x✝.ctorIdx = y✝.ctorIdx then isTrue ⋯ else isFalse ⋯

instance Phenomena.Reference.Studies.SikosEtAl2021.instBEqInterpretation : BEq Interpretation

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instBEqInterpretation = { beq := Phenomena.Reference.Studies.SikosEtAl2021.instBEqInterpretation.beq }

def Phenomena.Reference.Studies.SikosEtAl2021.instBEqInterpretation.beq : Interpretation → Interpretation → Bool

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instBEqInterpretation.beq x✝ y✝ = (x✝.ctorIdx == y✝.ctorIdx)

def Phenomena.Reference.Studies.SikosEtAl2021.instReprInterpretation.repr : Interpretation → Nat → Std.Format

Equations

• One or more equations did not get rendered due to their size.

instance Phenomena.Reference.Studies.SikosEtAl2021.instReprInterpretation : Repr Interpretation

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instReprInterpretation = { reprPrec := Phenomena.Reference.Studies.SikosEtAl2021.instReprInterpretation.repr }

Structural relationships between models

1. The baseline model (prior × literal semantics) IS RSA's L0.
2. In trivial contexts (unique referent), L1 = L0.
3. In pragmatically solvable contexts, L1 ≠ L0 -- RSA's recursive reasoning makes different predictions.

These are mathematical facts about the models, not empirical claims.

What this does NOT show: That RSA is empirically vindicated. Sikos et al.'s Experiment 3 tested contexts specifically designed to be pragmatically solvable (where L0 ≠ L1), and RSA still did not significantly outperform the baseline.

inductive Phenomena.Reference.Studies.SikosEtAl2021.Color : Type

Colors used in the experiments.

• blue : Color
• green : Color
• red : Color

instance Phenomena.Reference.Studies.SikosEtAl2021.instDecidableEqColor : DecidableEq Color

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instDecidableEqColor x✝ y✝ = if h : x✝.ctorIdx = y✝.ctorIdx then isTrue ⋯ else isFalse ⋯

instance Phenomena.Reference.Studies.SikosEtAl2021.instBEqColor : BEq Color

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instBEqColor = { beq := Phenomena.Reference.Studies.SikosEtAl2021.instBEqColor.beq }

def Phenomena.Reference.Studies.SikosEtAl2021.instBEqColor.beq : Color → Color → Bool

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instBEqColor.beq x✝ y✝ = (x✝.ctorIdx == y✝.ctorIdx)

def Phenomena.Reference.Studies.SikosEtAl2021.instReprColor.repr : Color → Nat → Std.Format

Equations

• One or more equations did not get rendered due to their size.

instance Phenomena.Reference.Studies.SikosEtAl2021.instReprColor : Repr Color

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instReprColor = { reprPrec := Phenomena.Reference.Studies.SikosEtAl2021.instReprColor.repr }

inductive Phenomena.Reference.Studies.SikosEtAl2021.Shape : Type

Shapes used in the experiments.

• square : Shape
• circle : Shape
• triangle : Shape

instance Phenomena.Reference.Studies.SikosEtAl2021.instDecidableEqShape : DecidableEq Shape

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instDecidableEqShape x✝ y✝ = if h : x✝.ctorIdx = y✝.ctorIdx then isTrue ⋯ else isFalse ⋯

instance Phenomena.Reference.Studies.SikosEtAl2021.instBEqShape : BEq Shape

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instBEqShape = { beq := Phenomena.Reference.Studies.SikosEtAl2021.instBEqShape.beq }

def Phenomena.Reference.Studies.SikosEtAl2021.instBEqShape.beq : Shape → Shape → Bool

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instBEqShape.beq x✝ y✝ = (x✝.ctorIdx == y✝.ctorIdx)

instance Phenomena.Reference.Studies.SikosEtAl2021.instReprShape : Repr Shape

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instReprShape = { reprPrec := Phenomena.Reference.Studies.SikosEtAl2021.instReprShape.repr }

def Phenomena.Reference.Studies.SikosEtAl2021.instReprShape.repr : Shape → Nat → Std.Format

Equations

• One or more equations did not get rendered due to their size.

structure Phenomena.Reference.Studies.SikosEtAl2021.Object : Type

An object in the reference game.

• color : Color
• shape : Shape

instance Phenomena.Reference.Studies.SikosEtAl2021.instDecidableEqObject : DecidableEq Object

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instDecidableEqObject = Phenomena.Reference.Studies.SikosEtAl2021.instDecidableEqObject.decEq

def Phenomena.Reference.Studies.SikosEtAl2021.instDecidableEqObject.decEq (x✝ x✝¹ : Object) : Decidable (x✝ = x✝¹)

Equations

• One or more equations did not get rendered due to their size.

instance Phenomena.Reference.Studies.SikosEtAl2021.instBEqObject : BEq Object

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instBEqObject = { beq := Phenomena.Reference.Studies.SikosEtAl2021.instBEqObject.beq }

def Phenomena.Reference.Studies.SikosEtAl2021.instBEqObject.beq : Object → Object → Bool

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instBEqObject.beq { color := a, shape := a_1 } { color := b, shape := b_1 } = (a == b && a_1 == b_1)
• Phenomena.Reference.Studies.SikosEtAl2021.instBEqObject.beq x✝¹ x✝ = false

instance Phenomena.Reference.Studies.SikosEtAl2021.instReprObject : Repr Object

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instReprObject = { reprPrec := Phenomena.Reference.Studies.SikosEtAl2021.instReprObject.repr }

def Phenomena.Reference.Studies.SikosEtAl2021.instReprObject.repr : Object → Nat → Std.Format

Equations

• One or more equations did not get rendered due to their size.

inductive Phenomena.Reference.Studies.SikosEtAl2021.Feature : Type

A feature predicate: either a color or a shape word.

• color (c : Color) : Feature
• shape (s : Shape) : Feature

def Phenomena.Reference.Studies.SikosEtAl2021.instDecidableEqFeature.decEq (x✝ x✝¹ : Feature) : Decidable (x✝ = x✝¹)

Equations

• One or more equations did not get rendered due to their size.

instance Phenomena.Reference.Studies.SikosEtAl2021.instDecidableEqFeature : DecidableEq Feature

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instDecidableEqFeature = Phenomena.Reference.Studies.SikosEtAl2021.instDecidableEqFeature.decEq

instance Phenomena.Reference.Studies.SikosEtAl2021.instBEqFeature : BEq Feature

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instBEqFeature = { beq := Phenomena.Reference.Studies.SikosEtAl2021.instBEqFeature.beq }

def Phenomena.Reference.Studies.SikosEtAl2021.instBEqFeature.beq : Feature → Feature → Bool

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instBEqFeature.beq (Phenomena.Reference.Studies.SikosEtAl2021.Feature.color a) (Phenomena.Reference.Studies.SikosEtAl2021.Feature.color b) = (a == b)
• Phenomena.Reference.Studies.SikosEtAl2021.instBEqFeature.beq (Phenomena.Reference.Studies.SikosEtAl2021.Feature.shape a) (Phenomena.Reference.Studies.SikosEtAl2021.Feature.shape b) = (a == b)
• Phenomena.Reference.Studies.SikosEtAl2021.instBEqFeature.beq x✝¹ x✝ = false

def Phenomena.Reference.Studies.SikosEtAl2021.instReprFeature.repr : Feature → Nat → Std.Format

Equations

• One or more equations did not get rendered due to their size.

instance Phenomena.Reference.Studies.SikosEtAl2021.instReprFeature : Repr Feature

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.instReprFeature = { reprPrec := Phenomena.Reference.Studies.SikosEtAl2021.instReprFeature.repr }

def Phenomena.Reference.Studies.SikosEtAl2021.featureMeaning (f : Feature) (o : Object) : Bool

Literal semantics: does the feature apply to the object?

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.featureMeaning (Phenomena.Reference.Studies.SikosEtAl2021.Feature.color a) o = (o.color == a)
• Phenomena.Reference.Studies.SikosEtAl2021.featureMeaning (Phenomena.Reference.Studies.SikosEtAl2021.Feature.shape a) o = (o.shape == a)

def Phenomena.Reference.Studies.SikosEtAl2021.nMatches (ctx : List Object) (u : Feature) : Nat

How many objects in a context match a given utterance.

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.nMatches ctx u = (List.filter (Phenomena.Reference.Studies.SikosEtAl2021.featureMeaning u) ctx).length

def Phenomena.Reference.Studies.SikosEtAl2021.isTrivial (ctx : List Object) (u : Feature) : Bool

A context-utterance pair is trivial when exactly one object matches.

Equations

• Phenomena.Reference.Studies.SikosEtAl2021.isTrivial ctx u = (Phenomena.Reference.Studies.SikosEtAl2021.nMatches ctx u == 1)

def Phenomena.Reference.Studies.SikosEtAl2021.trivialCtx : List Object

Trivial context: each utterance uniquely identifies its referent. {blue_square, green_circle, red_triangle}

Equations

• One or more equations did not get rendered due to their size.

def Phenomena.Reference.Studies.SikosEtAl2021.trivialUtts : List Feature

Utterances for the trivial context.

Equations

• One or more equations did not get rendered due to their size.

def Phenomena.Reference.Studies.SikosEtAl2021.solvableCtx : List Object

FG2012's classic solvable context: {blue_square, blue_circle, green_square}. "square" applies to two objects; pragmatic reasoning breaks the tie.

Equations

• One or more equations did not get rendered due to their size.

def Phenomena.Reference.Studies.SikosEtAl2021.solvableUtts : List Feature

Utterances for the solvable context.

Equations

• One or more equations did not get rendered due to their size.

theorem Phenomena.Reference.Studies.SikosEtAl2021.trivial_blue_unique : isTrivial trivialCtx (Feature.color Color.blue) = true

"blue" uniquely identifies blue_square in the trivial context.

theorem Phenomena.Reference.Studies.SikosEtAl2021.solvable_square_ambiguous : isTrivial solvableCtx (Feature.shape Shape.square) = false

"square" is ambiguous in the solvable context (matches 2 objects).

theorem Phenomena.Reference.Studies.SikosEtAl2021.trivial_ctx_all_trivial : trivialUtts.all (isTrivial trivialCtx) = true

The trivial context has all utterances trivially predicted.

theorem Phenomena.Reference.Studies.SikosEtAl2021.solvable_ctx_has_nontrivial : (List.filter (fun (u : Feature) => !isTrivial solvableCtx u) solvableUtts).length > 0

The solvable context has non-trivial utterances.