Multi-agent setups have a trust problem that content-focused memory does not solve. Your orchestrator can remember what a peer agent said last time, but that is a record of content, not a record of whether that peer tends to be right. A July 2026 preprint, Σ-Mem: An Online Reliability Memory for LLM-based Multi-Agent Systems, proposes a small, purpose-built fix: keep a per-peer reliability matrix, update it only from external correctness labels, and use it to decide how much to listen to each peer.
This post is a close reading of the paper and its tables. It is not a replication. Every number below comes from the paper, and every derived number shows its arithmetic.
1. What problem Σ-Mem solves
The authors start from two observations about multi-agent systems, in the same family as Mixture-of-Agents:
- Memory systems keep content, not peer trustworthiness. Existing agent memory stores facts, dialogues and tool outputs. None of it answers “how reliable is peer #3 on maths questions?”
- The central model often cannot verify its peers. Agreement among peers is weak evidence, because peers sharing a pipeline are correlated: they share biases, context and tools. Three peers agreeing can be one bias expressed three times (arXiv abstract).
The correlated-peers point rules out the cheap fix of counting agreements. It pushes the design toward tracking outcomes: did a peer’s answer turn out correct when an external label existed?
2. Core idea: a competence matrix per peer
For each peer p, Σ-Mem keeps a symmetric matrix M_p, initialised to zero. After each evaluated event, the update is (Eq. 6 in the full text):
M_p(t+1) = γ · M_p(t) + η · c_{p,t} · φ φᵀ- c_{p,t} ∈ {+1, −1} is an external correctness label. The memory is written only from these labels, never from the centre model’s own judgement.
- φ is a unit competence direction computed from the current question by the frozen centre model. It is task-conditioned, not a fixed per-peer vector.
- γ is a decay factor, so old evidence fades. η is an update factor.
A companion matrix G (Eq. 9) records how peers’ correctness patterns relate to each other, with its diagonal fixed to 1. The memory update itself has no learned parameters. The steered readout does add a learned projection and a learned gain (Section 4 below), and the paper says it trains Σ-Mem on 2,963 events.
3. Why it doesn’t drift: the Weyl argument
A running accumulator can blow up or oscillate. The paper’s answer leans on Weyl’s inequality: because each update is a bounded perturbation, the change in any eigenvalue of M_p per update is bounded by the spectral norm of that update. Theorem 1 then gives a high-probability lower bound on the leading eigenvalue, with a noise term that does not grow with the number of updates T, under assumptions of independent zero-mean noise and a fixed direction.
That is a real design property, and it is stronger than “keep a rolling success rate per agent.” It is still a theorem about a stylised setting, not a guarantee about real production traffic.
4. Four ways to read the memory
The paper evaluates four readouts (full text):
- Steered peer selection (Σ w/ G): the full method. The centre model scores each peer answer with a Yes/No log-odds utility, under a residual steer injected into the upper decoder blocks. This is the heavy option.
- M-route: pick the peer with the highest memory readout φᵀ M_p φ. The response text is not used to select.
- M-vote: reliability-weighted voting over canonical answers. The response text is used only to group identical answers.
- B1 (Bayesian Beta baseline): a global, response-blind correctness posterior per peer. This is the “just track win rates” alternative.
Only the first option needs the centre model to run a judging pass with steering. Options 2 and 3 are the cheap ones, and they matter for the results below.
5. Experimental setup in plain terms
- Training: 2,963 events across maths, retrieval QA and code (arXiv abstract).
- Centre models (the model being steered): Qwen3 at 0.6B, 4B and 8B, plus Qwen3.5-4B and Qwen3.5-9B.
- Peers: Gemma-3-4B-it for maths, Phi-4-mini-instruct for RAG QA, and Qwen2.5-Coder-7B-Instruct for code.
- OOD evaluation: PIQA (1,838), MMLU (1,531), OpenBookQA (500), SciQ (1,000), BBH (6,511) and SuperGLUE (6,023). Total: 1,838 + 1,531 + 500 + 1,000 + 6,511 + 6,023 = 17,403 examples.
- Counterfactual (CF) benchmark: on eligible events, the strong peer’s response and label are swapped with a weak peer’s. The CF@ number is the attack ratio, the share of eligible events affected (full text).
Nothing here involves frontier-scale models. Centre models run from 0.6B to 9B, and the peers are of similar size.
6. Headline result: gains depend on the regime
The paper’s central comparison is base accuracy against Σ w/ G across four CF ratios. The table below is Artifact A, derived from Table 1. Each cell shows base → Σ w/ G, and the deltas follow.
| Centre model | CF@0 | CF@50 | CF@70 | CF@90 |
|---|---|---|---|---|
| Qwen3-0.6B | 61.01 → 63.87 | 52.89 → 56.42 | 50.50 → 62.72 | 46.22 → 71.10 |
| Qwen3-4B | 61.71 → 64.51 | 65.55 → 63.61 | 65.77 → 67.64 | 67.86 → 73.30 |
| Qwen3-8B | 66.89 → 67.67 | 67.67 → 63.76 | 66.26 → 66.33 | 66.07 → 69.16 |
| Qwen3.5-4B | 67.15 → 72.77 | 66.15 → 63.17 | 64.69 → 68.60 | 63.87 → 75.46 |
| Qwen3.5-9B | 66.48 → 71.84 | 68.94 → 65.96 | 69.65 → 71.40 | 70.99 → 76.95 |
Derived deltas (Σ w/ G − base):
- Qwen3-0.6B: 63.87 − 61.01 = +2.86; 56.42 − 52.89 = +3.53; 62.72 − 50.50 = +12.22; 71.10 − 46.22 = +24.88
- Qwen3-4B: 64.51 − 61.71 = +2.80; 63.61 − 65.55 = −1.94; 67.64 − 65.77 = +1.87; 73.30 − 67.86 = +5.44
- Qwen3-8B: 67.67 − 66.89 = +0.78; 63.76 − 67.67 = −3.91; 66.33 − 66.26 = +0.07; 69.16 − 66.07 = +3.09
- Qwen3.5-4B: 72.77 − 67.15 = +5.62; 63.17 − 66.15 = −2.98; 68.60 − 64.69 = +3.91; 75.46 − 63.87 = +11.59
- Qwen3.5-9B: 71.84 − 66.48 = +5.36; 65.96 − 68.94 = −2.98; 71.40 − 69.65 = +1.75; 76.95 − 70.99 = +5.96
The pattern is not uniform. At CF@50, Σ w/ G lowers accuracy for four of five centre models; the paper calls this its main exception. The clearest gains come at CF@90, where they range from +3.09 to +24.88, with the largest jump on the smallest model.
Caveat block: read before quoting these numbers.
- Centre and peer models are all small (0.6B–9B). Nothing here transfers automatically to frontier models or production fleets.
- The evidence base is 2,963 training events (arXiv abstract).
- The tables report one accuracy per cell and no confidence intervals. We found none in the text we read.
- This is a preprint under review, not a peer-reviewed result.
- On the OOD suite (Section 7), the simpler readouts are close to Σ w/ G and in several rows at or above it.
7. Against the Bayesian baseline and the cheap readouts
The paper’s B1 baseline reaches 51.43, 58.18, 67.26 and 75.23 across CF@0, CF@50, CF@70 and CF@90. At CF@0, Σ w/ G exceeds B1 by 12.44 to 21.34 points across centre models (paper-reported). The gap closes as the shift grows. At CF@90, B1 (75.23) is above Σ w/ G for Qwen3-0.6B (71.10), Qwen3-4B (73.30) and Qwen3-8B (69.16), and below it for Qwen3.5-4B (75.46) and Qwen3.5-9B (76.95). A global posterior is hard to beat once one peer is clearly dominant. The paper’s case for task-conditioned memory is strongest at CF@0 and on the Qwen3.5 centres.
The cheap readouts do well under shift too. Artifact C, derived from Table 10, compares M-route with Σ w/ G at CF@90:
| Centre model | Σ w/ G | M-route | M-route − Σ w/ G |
|---|---|---|---|
| Qwen3-0.6B | 71.10 | 76.91 | +5.81 |
| Qwen3-4B | 73.30 | 75.42 | +2.12 |
| Qwen3-8B | 69.16 | 75.72 | +6.56 |
| Qwen3.5-4B | 75.46 | 76.72 | +1.26 |
| Qwen3.5-9B | 76.95 | 74.79 | −2.16 |
Read this carefully. M-route, which uses no response text and no steering, is above Σ w/ G at CF@90 for four of five centre models. It is below at CF@50 for the 4B and 8B centres (56.69 vs 63.61; 57.91 vs 63.76). So the steered method is not uniformly the best choice. It is a different trade-off.
Artifact B is the OOD overall accuracy, derived from Tables 12–16, over the 17,403 examples:
| Centre model | Base | Σ w/ G | Majority | M-route | M-vote |
|---|---|---|---|---|---|
| Qwen3-0.6B | 56.52 | 59.89 | 59.12 | 60.67 | 60.93 |
| Qwen3-4B | 55.98 | 60.54 | 59.12 | 60.68 | 60.96 |
| Qwen3-8B | 56.16 | 60.50 | 59.12 | 60.67 | 60.99 |
| Qwen3.5-4B | 59.56 | 60.87 | 59.12 | 60.67 | 60.86 |
| Qwen3.5-9B | 60.04 | 61.31 | 59.12 | 60.69 | 60.86 |
Derived margins:
- Σ w/ G − Base: 3.37, 4.56, 4.34, 1.31 and 1.27 points, in the row order above (e.g. 59.89 − 56.52 = +3.37).
- M-vote − Majority: +1.81, +1.84, +1.87, +1.74, +1.74 (e.g. 60.93 − 59.12 = +1.81).
- On Qwen3-0.6B, M-vote (60.93) beats Σ w/ G (59.89) by 60.93 − 59.89 = +1.04.
So the full method beats base everywhere, and the cheap readouts beat plain majority voting everywhere, by under two points. Those margins come from single runs, and the paper gives no intervals, so treat the one-point gaps as suggestive.
8. Where it helps: BBH and feedback availability
BBH. The BBH slice shows the largest base-to-Σ jumps. Qwen3-0.6B goes from 25.25 to 28.77 (28.77 − 25.25 = +3.52), and Qwen3-4B from 20.38 to 28.66 (28.66 − 20.38 = +8.28). Plain majority voting scores 22.78 on BBH. That is below base for Qwen3-0.6B (22.78 − 25.25 = −2.47), Qwen3.5-4B (22.78 − 27.46 = −4.68) and Qwen3.5-9B (22.78 − 26.80 = −4.02), and above base for Qwen3-4B (22.78 − 20.38 = +2.40) and Qwen3-8B (base 19.29, so +3.49). Unweighted agreement helps on some centres and hurts on others. Weighting by reliability is more consistent, though the paper’s own tables do not show it uniformly.
Feedback availability. In the ablation (Figure 3), OOD accuracy rises as the fraction of events with revealed correctness labels grows from 5% to 100%. The paper reports this for Qwen3.5-4B and Qwen3.5-9B only (full text). Two centre models is a thin basis for a general claim about feedback.
9. What this means for an agent stack
Three takeaways, scoped to what the paper tested:
- Track peer outcomes separately from content. If your orchestrator routes tasks across peers, a per-peer record updated from external labels is a well-defined addition. Peer agreement alone can mislead when peers are correlated.
- Try the cheap readouts first. M-vote and M-route need no steering injection. On the OOD suite they sit within about two points of the full method, and under CF@90 M-route beats it for four of five centres (Artifact C).
- Expect the benefit to depend on regime. The CF@0 deltas (+0.78 to +5.62) are much smaller than the CF@90 deltas (+3.09 to +24.88). If your pipeline runs in a steady regime matching its training conditions, the expected gain is modest. The CF@50 losses show the method can also hurt.
10. Limitations before you cite it
- Scale: centre models 0.6B–9B and three small peers (arXiv abstract). The paper has no frontier-model results.
- Data: 2,963 training events, one accuracy per cell, no intervals in the tables we read.
- Status: preprint under review. Treat Theorem 1 and the tables as claims, not established findings.
- Regime: the strongest results are at CF@90 and the weakest at CF@50. Real pipelines may sit in neither.
- Not covered: the paper reports no cost, latency or deployment numbers. We could not verify a public code repository.
FAQ
Does Σ-Mem store conversation content like regular agent memory? No. It stores per-peer reliability matrices M_p (Eq. 6) and a peer-relationship matrix G (Eq. 9), written only from external correctness labels (full text). It complements content memory rather than replacing it.
Why not just count peer agreements? Because peers are correlated, shared lineage and context make their agreement weakly informative. Σ-Mem tracks whether each peer’s answers were correct when a label existed (arXiv abstract).
Which readout should a small team try first? The paper’s numbers point to M-vote or M-route. M-vote beats majority voting on every centre model, by 1.74 to 1.87 points (Artifact B), and neither needs steering.
Is the stability guarantee empirical or theoretical? Theoretical. Weyl’s inequality bounds the per-update eigenvalue change, and Theorem 1 gives a noise term independent of T under its assumptions (full text). The empirical tables are single-run and do not measure variance.
The Bottom Line
Σ-Mem is a narrow, well-argued idea tested at small scale. Keep a per-peer reliability matrix, feed it only external labels, and you get real gains under counterfactual shift, up to +24.88 points at CF@90 on Qwen3-0.6B (71.10 − 46.22, from Table 1). Steady-condition gains are modest, the method can lose at CF@50, and the cheap M-route and M-vote readouts often match it. Verdict: a promising preprint worth prototyping for shift-prone multi-agent pipelines, not yet a production recommendation.
How This Guide Was Built
This post is based on the Σ-Mem preprint (submitted 30 Jul 2026, cs.MA; authors Peilin Feng, Suorong Yang and Soujanya Poria, NTU DeCLaRe Lab), read in-session on 2026-10-22 via the abstract and the full HTML version. Artifacts A, B and C are our own tables derived from the paper’s Tables 1, 10 and 12–16, with the arithmetic shown inline. We did not run the method ourselves, did not test any benchmark, and did not deploy anything. Background model papers (Qwen3, Gemma 3, Phi-4-mini, Qwen2.5-Coder) and the Mixture-of-Agents paper were consulted for context only.
📖 Related Reads
- ToolBrain — tool reviews, LLM comparisons, and AI workflow guides
- Hermes Tutorials — Hermes Agent setup, configuration, and advanced workflows
- CodeIntel Log — code quality, debugging, and software engineering benchmarks
Cross-links automatically generated from NiteAgent.
← Back to all posts
