Recombine, Then Generate

Research conducted by Claude Fable 5 (Anthropic), Xaos Research Lab.

Abstract

Our first audit established that canonicalizing weight-space symmetries is what makes honest generative models of neural-network weights possible on MLP and CNN zoos. This paper asks whether that recipe survives contact with the weight space practitioners actually care about — LoRA adapter populations on a modern language model — and reports, under the same memorization-guarded protocol, that it survives only in part, and that what replaces it is more interesting. On a population of 1,024 Qwen3-0.6B/BoolQ LoRA adapters (11,264-checkpoint corpus, GL(r) gauge quotiented by SVD canonicalization), we measure three pre-registered campaigns. First, the chart findings: the raw–canonical gap opens with population size (raw PCA can no longer represent its own members at any tested rank by n=1,024, while the canonical chart holds its ceiling), and canonicalization and alignment are distinct moves — the quotient serves generation while pairwise alignment serves interpolation, and neither substitutes for the other. Second, the audit surprise: the latent-flow generator that owned our first paper’s headline cannot reach the corner of accuracy×novelty space that fixed recombination operators saturate. Exact ΔW-averaging of 16 parents at grown rank passes a joint novelty-and-performance criterion at 100% where the flow manages 0.4%, and a functional battery shows the recombinations are genuinely new functions (nearest-corpus disagreement at the corpus’s own 90th percentile), forcing a metric upgrade: geometric novelty without a functional filter is gameable, so the audit criterion becomes accuracy ∧ geometric novelty ∧ functional distinctness (N&P&F). Third, the generative-merge result: a conditional flow over recombination coefficients — a 16-dimensional learned merge operator — ties the best fixed operator’s saturation on the joint criterion (1.000 at 1× and 2× the corpus spread), adds what no fixed operator has (a working accuracy dial, steering ρ = 0.78; achieved-accuracy range 0.78–0.84 where our previous flows were flat), and samples individual objects in the deep frontier. And the deep frontier itself — empty in every audit cell of the previous campaign — turns out to be real and anisotropic: recombine-then-push operators populate it to 4× the corpus spread with no accuracy loss (64/64 candidates above the bar at full test, functional distinctness at the soup signature), while extrapolation along the interpolation axis cliffs after 3×, and unforced averaging converges toward the corpus as parents are added. A pre-registered thin-manifold prediction is thereby falsified in the most informative direction: the empty frontier of our earlier audits was a property of the operator set, not of adapter space. We also close the loop on our first paper’s velocity-extinction law: re-tested at adapter scale in its own units with the registered convergence gate open, the law does not transfer — collapse is a property of its original small-CNN/MLP-zoo regime. All thresholds were pinned before any generator ran; every prediction was adjudicated on the record, including one retraction of our own instrument error.

1. Introduction

The first paper in this series ended with an audit protocol and a warning. The protocol: pin an accuracy bar and a novelty bar from the population’s own geometry before any generator trains, then score every candidate — learned or trivial — jointly, so that memorize-and-perturb cannot pass. The warning: every result was bounded to small vision zoos, and the field’s real object of desire is the weight space of adapted language models.

This paper is the adapter-space sequel. It asks three questions in sequence, each pre-registered before its campaign fired:

1. Does the quotient still matter at population scale? (A2) LoRA’s GL(r) gauge is invisible in function space; does it actually obstruct modeling as populations grow, or is it a finite tax a bigger model absorbs?
2. Does canonicalize-then-generate transfer? (A3) Our audited latent flow beat every trivial baseline on the vision rigs. Adapter space has richer cheap baselines — merging is a mature practical art — so the bars are fiercer: interpolations in four charts, rank-growing ΔW soups, noise at two scales, per-dimension Gaussians, PCA-Gaussian hybrids.
3. If fixed merge operators win, can a learned generator over merges keep the win and add generation’s virtues — steering and diversity? (A4) This is the generative-merge question, and it comes with a sharpened metric: the A3 battery caught geometric novelty divorcing from functional novelty, so A4 scores accuracy ∧ geometric novelty ∧ functional distinctness.

The answers, in brief: yes, the quotient matters more at scale, not less; no, the flow-over-weights recipe does not transfer — fixed recombination dominates it decisively on the joint criterion; and yes — a 16-dimensional conditional flow over recombination coefficients ties the incumbent operator and adds the dial. The campaign’s largest single result is a falsification: the deep novelty frontier (≥3× the corpus’s own spread), empty in every previous audit cell, is populated — cheaply, robustly, and anisotropically — once operators are designed to force distance rather than average it away.

Throughout, the evidentiary régime of the first paper applies unchanged: pre-registration with numbered predictions ratified before firing; thresholds pinned from population geometry before any generator ran; adjudication on the record including misses and retractions; and full seed/job-trace/state-hash reproducibility. Three instrument errors were caught during the campaign; each is recorded in the journal with its correction (§6.7).

2. Related work

(Citation-walked 2026-08-18: three parallel verification/scoop sweeps — the merging canon pinned against arXiv/OpenReview/proceedings pages; a LoRA-generation and merge-coefficient sweep including the Zeng citation graph; a beyond-interpolation/basin-geometry sweep. No scoop on any headline claim; the four nearest works are distinguished inline below and where the corresponding result appears.)

Model merging as practice. Weight averaging and its refinements are a mature art: model soups (Wortsman et al., 2022), Fisher-weighted averaging (Matena & Raffel, 2022), task arithmetic (Ilharco et al., 2023) with its tangent-space theory (Ortiz-Jimenez et al., 2023), TIES (Yadav et al., 2023) and DARE (Yu et al., 2024) interference management, and the industrial frankenmerge practice consolidated in MergeKit (Goddard et al., 2024). In LoRA space specifically: arithmetic composition of adapters (Zhang et al., 2023), LoraHub’s optimized cross-task composition (Huang et al., 2024), and KnOTS (Stoica et al., 2025), which SVD-aligns LoRA deltas so merging methods apply — the same GL(r)-gauge instinct our A2 arms measure at population scale. This literature selects merges by downstream performance; whether a merged model is a new function or a relocated copy of its parents is unmeasured. The nearest thing to an audit compares merged models’ behavior and internals against their parents (Sigrist & Waldis, 2025); nothing in the merging literature applies a memorization-guarded joint criterion, and the citation graph of the field’s memorization referee (Zeng et al., 2026) contains no merging audit. That audit is this paper’s first contribution.

Learned and optimized merge operators. A fast-moving line learns or optimizes the merge itself: AdaMerging’s entropy-minimized layer-wise coefficients (Yang et al., 2024), evolutionary recipe search (Akiba et al., 2025), swarm search over LoRA populations (Feng et al., 2025), LoRA.rar’s amortized coefficient predictor for subject-style pairs (Shenaj et al., 2025), and — closest to arm 1 — EvoGM (Jiang et al., 2026), which trains a generative model over per-layer merging coefficients as an accelerator inside an evolutionary search loop. Our construction differs on every axis that matters for the audit: the coefficient flow is trained on a pooled corpus of measured recombinations from a 1,024-adapter population (not on search history), it is accuracy-conditioned (steering is a registered claim, in the lineage of metric-conditioned weight generation begun by G.pt (Peebles et al., 2022)), its own coefficient-space symmetry is canonicalized before training, and its samples are scored under N&P&F rather than selected for benchmark performance. ZeTT (Minixhofer et al., 2024) is the conditional weight-generation precedent from another direction — a hypernetwork emitting tokenizer-conditioned embedding matrices — and shares only the conditional-generator shape.

Generative models of weights, and the LoRA-generation line. The general machinery descends from G.pt, weight-space diffusion (Wang et al., 2024), hyper-representation sampling (Schürholt et al., 2022), SANE (Schürholt et al., 2024), RPG (K. Wang et al., 2025), and conditional flow matching over parameters (Saragih et al., 2025); the canonicalization-first line (SANE, Erdogan 2025, DeepWeightFlow (Gupta et al., 2026)) is audited in our first paper. For LoRA weights specifically the line is young and crowded: Text-to-LoRA (Charakorn et al., 2025), COND P-DIFF (Jin et al., 2024), ORAL (2025), prompt-to-weights hypernetworks (Liang et al., 2025), structure-aware LoRA diffusion (LoRAGen, 2026), and weights2weights’ population subspace sampling (Dravid et al., 2024). Of these, only COND P-DIFF gestures at novelty (post-hoc distance plots, no criterion) and only weights2weights makes a novelty claim (new identities, evaluated at the identity level); none applies memorization-guarded acceptance bars. DeepWeightFlow is the honorable exception in the general line — it reports functional-disagreement diversity (mIoU over error sets) for its own generator in explicit response to Zeng et al. — and the A3 battery turns that same instinct into a pre-registered acceptance criterion applied to every candidate family symmetrically, learned or trivial.

Trivial baselines as referees. Zeng et al. (2026) established that weight generators are routinely flattered by weak baselines — replicas and simple blends match them, and generated weights concentrate toward the training-set average. Our protocol operationalizes the position paper’s call (Z. Wang et al., 2026) for novelty-and-memorization benchmarks; this paper extends the referee set with the fiercest bars adapter space offers (exact-ΔW recombination at grown rank) and adds the functional filter φ.

Beyond interpolation, and the geometry of averages. Extrapolation outside the convex hull has precedents: ExPO (Zheng et al., 2025) extrapolates a checkpoint pair along its training direction and documents the same rise-then-collapse depth profile our extrapolation family shows (§5.5); soup coefficients outside [0,1] appear in adversarial-robustness soups (Croce et al., 2023); task-vector negation is coefficient −1. None measures distance-to-population or contrasts push directions — the anisotropy result (§5.5) has no precedent we could find. Perturbing averages also has ancestors: SWAG samples covariance-shaped noise around an SGD average (Maddox et al., 2019), shrink-and-perturb restores trainability (Ash & Adams, 2020), and Zeng et al. use noise-on-checkpoints as a debunking baseline; the calibrated isotropic push on a merge, scored for accuracy survival at multiples of the population’s own spread, appears unclaimed. The soup-shallowing fact (§5.5) is the anti-novelty face of known geometry: fine-tuned weights live on a thin shell whose center weight averaging approaches — exploited for accuracy by Model Stock (Jang et al., 2024) and formalized as variance decay by DiWA (Rame et al., 2022); asymmetric valleys (He et al., 2019) and the tiny-subspace structure of trained directions (Gur-Ari et al., 2018) supply the mechanistic background for why an orthogonal push survives where a trained direction cliffs.

3. Setting

The zoo. 1,024 LoRA adapters (Hu et al., 2022; r=8, all linear layers, 196 matrices) trained on BoolQ from Qwen3-0.6B (revision-pinned), seed-only lineages, built and poolability-tested across 8 training segments (journal entry 055). The corpus for all novelty geometry is the canonical merged population: 11,264 checkpoint rows. A zero-adapter evaluation pins the task floor at 0.6517 before any generator ran; the accuracy bar α = 0.8097 follows from the pre-registered floor-span rule. All evaluations run on a governed GPU sidecar with slice-level checkpointing; accuracy scoring is at 1,000 examples for grid cells and the full 3,270-example validation set for flagship confirmations.

The gauge and the charts. A LoRA update ΔW = BA is invariant under (B, A) → (BG⁻¹, GA) for any G ∈ GL(r): functionally identical adapters occupy a 64-dimensional orbit per matrix. We work in three coordinate systems: raw (as trained), balance-only (column-norm equalization), and SVD-canonical (the full quotient; per-matrix SVD with sign convention). Distances and PCA/flow latents are computed within a chart; the quotient chart is the protocol default for generation after A2 (§5.1).

Novelty. A candidate’s geometric novelty is its distance to the nearest corpus row in the appropriate chart (higher-rank candidates compare through their padded embedding, which is exact for zero-padded factor concatenations). The novelty ladder is pinned to the corpus’s own leave-one-out spread: τ = {1, 2, 3, 4, 6} × 7.6486 (the LOO median). Functional distinctness (added at A4 after the A3 battery, §5.3): a candidate must disagree with its nearest corpus row’s predictions on ≥ φ = 0.091 of examples — the corpus’s own 10th percentile of pairwise disagreement, measured over 709 corpus rows at 1,000 examples. The joint criterion N&P&F requires all three.

The recombination space. The exact product-space concatenation identity: for parents with factors (Bᵢ, Aᵢ) and coefficients cᵢ, the block-concatenation \([\sqrt{|c_1|}\,\mathrm{sgn}(c_1)B_1 \mid \cdots] \cdot [\sqrt{|c_1|}A_1 ; \cdots] = \textstyle\sum_i c_i \Delta W_i\) exactly at grown rank r = 8m. Every recombination operator in this paper — soups, interpolations, extrapolations, and the learned recombiner’s samples — materializes through this one code path, smoke-tested to reproduce the incumbent constructions bitwise.

4. Evaluation protocol

Inherited from the first paper, with the A4 upgrade:

• Bars before generators. α and the τ ladder were pinned before any A3 generator trained; φ was pinned from the A3 battery measurement before any A4 generator trained. No threshold moved after a result was visible.
• N&P&F. A cell’s score at rung k is the fraction of its candidates with accuracy ≥ α AND nearest-corpus distance ≥ kτ AND functional NN-disagreement ≥ φ. The pre-φ form (N&P) is reported alongside where the distinction is load-bearing (§5.4).
• Fierce bars. Every cell competes against: random inits, noise on corpus finals (σ ∈ {0.1, 0.5}), per-dimension Gaussian and PCA-Gaussian fits, pairwise interpolation in four charts, exact-ΔW rank-grown soups of 16, a prediction-averaging ensemble bar, and (after A3) the functionally-filtered noise bar that keeps the hollow-novelty lesson visible in every table.
• Pre-registration. Each campaign’s design was ratified as drafted before firing (A2 entry 034; A3 entry 058; A4 entry 063 with two staging addenda recorded before the work they governed). Predictions are numbered; every adjudication cites its cells; misses and retractions are published.
• Integrity mechanics. Corrected artifacts propagate through single access paths; evaluation labels are alignment-asserted per job (predictions must recompute to banked accuracies bitwise); novelty preseeds are CPU-computed and re-verified against banked values before any calibration is trusted.

5. Results

5.1 The quotient matters more at scale, and “canonicalize” ≠ “align” (A2)

Two pre-registered arms on the full zoo (entries 056–057):

The raw−canonical gap opens with population size. PCA roundtrip ceilings per chart at n ∈ {32, 128, 512, 1024}: the canonical chart’s k* (rank within 1 pt of its own plateau) grows 16→64→128→128, while raw’s grows 32→64→256→256 with the last cell right-censored — raw’s curve is still climbing at the maximum tested rank. At the registered n=1,024 cell the ratio meets the ≥2× threshold as a lower bound. The stronger form: raw’s plateau degrades with population (.8270 → .7928, 3.4 pt under the identity ceiling — raw PCA can no longer represent its own members at any tested rank), while the canonical chart holds .8269–.8311 everywhere. Capability-at-novelty exists only in the canonical chart: canonical delivers ceiling-grade accuracy at genuine novelty (13–15 off-corpus), raw only where novelty → 0. An earlier small-population reading (“the GL(r) gauge is a finite rank tax”, n=32) was an artifact exactly as the registered prediction anticipated.

The seed-LMC barrier is a frame artifact — and per-row canonicalization does not fix it. λ=0.5 midpoints of 16 seed pairs at three training depths (\(\lVert \Delta W \rVert_F\) growing 2.3×): the raw barrier (+0.94 pt at depth 500) shrinks with depth (+0.51 at 2,000; ratio 0.55, nowhere near the registered 3× that would have made the published seed-LMC collapse (Frankle et al., 2020; Entezari et al., 2022) an update-magnitude regime), while pairwise-Procrustes and exact-ΔW midpoints gain at every depth (−0.56 to −0.83 pt). The exploratory cell is the sharpest lesson: canonicalized-but- unaligned midpoints barrier like raw (+1.06/+0.92/+0.54). The quotient serves generation; pairwise alignment serves interpolation; the protocol requires both, chosen per operation.

5.2 The audit surprise: fixed recombination saturates the corner the flow cannot reach (A3)

The A3 grid ran k=128 latent flow matching per chart {raw, balance, canonical} × {unconditional, accuracy-conditioned}, a three-seed conditional flagship, a k=64 companion, and the full bar family, all scored N&P (entries 058–062). The verdict table’s spine:

candidate family	acc mean/max	N&P@1×	@2×	@3×+
flow, canonical uncond (n=256)	.692/.819	.004	0	0
flow, canonical cond ×3 seeds	.69/.81	0	0	0
soup-of-16, exact ΔW, r128	.843/.853	1.000	1.000	0
pairwise interp, exact ΔW, r16	.833/.854	.891	.656	0
noise 0.5σ on finals	.826/.843	.973†	0	0
flat-space interps (3 charts)	.82–.83	.32–.62	≤.16	0
everything else	—	0	0	0

† functionally hollow — §5.3.

P-A3-1 (the landscape prediction) was falsified in both halves: six bar members beat the canonical flow at 1×, and the predicted deep-τ soup dominance also failed — no method populated the frontier at ≥3×. The flow’s steering remained real (ρ = .607/.683/.652 across seeds, chart-ordered: canonical .65 > balance .49 > raw .12), and raw-chart flows failed honestly (most-novel objects, never above .79 accuracy). But the reach claim died: a k=128 flow on 11k rows emits inside its codec ball, while the accuracy bar sits within recombination range of the corpus. The winning objects are rank-grown ΔW recombinations — structurally outside the flow’s emission space. Generative merge became the measured frontier, and the empty ≥3× band became a pre-registered question for A4.

5.3 Geometric and functional novelty dissociate; the metric upgrade (A3 battery)

Scoring every α-passer’s predictions against its nearest corpus row (1,772 passers, 709 unique NN rows, 1,000 examples; corpus functional baseline: median pairwise disagreement 11.2%, q10 9.1%, q90 24.7%):

• noise05 is a functional clone factory: geometric novelty 8.1 (past 1×), parent disagreement 2.1% — its .973 N&P is the norm metric gamed by isotropic noise. noise01 likewise (0.7%).
• The soup’s novelty is functionally real: nearest-corpus disagreement 22.6% — the corpus’s own q90 — and its 256 candidates are 256 distinct functions (within-set disagreement 2.7%) forming a neighborhood 4× tighter than the corpus’s own diversity. A novel functional neighborhood, narrow but real.
• k=16 prediction ensembles lift every family 1.3–2 pt (best object of the campaign: a noise01 ensemble at .856, unregistered, reported as found).

The full dissociation cloud — every α-passer of both campaigns on the geometric-novelty × functional-disagreement plane — is Figure 10.

Protocol consequence, pinned before A4 fired: geometric τ requires a functional check. A4’s criterion is N&P&F with φ = corpus q10 = 0.091, and A3’s cells were re-scored retrospectively as A4’s baseline table (a4-baseline-npf.json, zero new compute): φ leaves the soup untouched (1.000/1.000 at 1×/2×), empties the noise bar exactly (0 of 499 noise passers reach φ), and halves exact-ΔW interpolation (.891 → .543 at 1×, with every φ-passing row at ≥2× — functional distinctness and deep geometry select the same mid-t rows). The flow’s 11 α-passers all pass φ: its failure was reach, not realness.

5.4 The learned recombiner ties the operator and adds the dial (A4 arm 1)

Construction (pinned in a prereg addendum before training): candidates are pooled per-parent coefficient vectors \(c \in \mathbb{R}^{16}\) applied uniformly across all 196 matrices — the per-matrix alternative (3,136 dims) was rejected because both incumbent operators are matrix-constant, so its extra dimensions would train on 196 identical copies of the pooled signal and could emit only unlearned noise. Canonical form sorts coefficients descending (parents are exchangeable and drawn fresh at materialization — the permutation symmetry is quotiented out of coefficient space, the paper’s title move applied to its own operator). The generator is the same conditional flow-matching machine (Lipman et al., 2023) the previous audits used, at dim=16, trained for 100k steps (26 s of GPU) on 1,408 coefficient rows: the measured incumbents (soup: uniform 1/16; exact-ΔW interpolation: (1−t, t)) plus 896 fresh operator draws (pairs with t ∈ [−0.5, 1.5]; Dirichlet mixtures at support 3–16 with scale jitter) evaluated for achieved accuracy.

The training set itself produced a finding: the Dirichlet interior is a performant plateau — all 512 dense draws pass α (range [.815, .853]), at every support size, spikiness, and off-simplex jitter tested. The soup neighborhood is not a knife-edge; unforced recombination of corpus parents is robustly performant everywhere we sampled. Extrapolation degrades gracefully and symmetrically (interior mean .833 → .807 at |t−0.5| ≈ 1.0).

Sampling at the standard 8-rung conditioning ladder (cond .65–.86, 64 per cell, server-fixed seed) shows the cond channel structuring coefficient space before any evaluation: negative mass falls monotonically .90 → .005 and peak coefficient 1.72 → .20 as cond rises — low-cond samples are extrapolation-shaped, high-cond samples are flat soup-like mixes. One sample of 512 diverged (Σc = 63); it rode the audit as emitted.

The verdict (grid + φ top-up; flagship at full test; the dial and the coupled geometry slide are Figure 8):

• N&P at 1× and 2×: the top cell (cond .86) scores 1.000/1.000 — tying the incumbent soup’s saturation. Under N&P&F it scores .953/.953 against the soup’s 1.000 — three φ-marginal rows short. The registered prediction’s letter says “N&P” while the campaign’s upgraded audit metric is N&P&F; we adjudicate at both levels rather than collapse the difference: as written, P-A4-1 is confirmed on all three clauses; under the upgraded metric its first clause misses by 3/64.
• Steering: ρ = 0.781 (Spearman, achieved-vs-conditioned accuracy pooled over the ladder) — above the registered 0.5 and above every A3 flow seed.
• The dial is real: achieved-accuracy cell means span .779 → .842 where the A3 flows sat flat at ≈.69. Full-test flagship: 64/64 above α at 3,270 examples (mean .8425).
• The deep tail exists: three sampled objects pass N&P&F at ≥3× (acc .811–.815, novelty 24.5–27.7, φ .204–.337 — the last beyond the corpus’s q90). They arrive at rate 1/64 per low-cond cell via conditioning leakage — un-targeted, but the first deep-frontier objects any learned generator has produced in this program.
• The hollow middle: the mid-cond cells’ 90 α-passers all fail φ (median disagreement .055) — geometric novelty divorcing from functional novelty inside a single generator’s conditioning ladder, precisely the trap the battery built φ to catch.

5.5 The deep frontier is real, and it is anisotropic (A4 arm 2)

Arm 2 asked the thin-manifold question directly with operators designed to force distance, at shells calibrated closed-loop on CPU before any GPU spun (the perturbation law is exact in this regime: \(\mathrm{nov}^2 = \mathrm{nov}_0^2 + k^2\sigma^2\) with k = 14.018 identical at both probe magnitudes; extrapolation novelty is symmetric in |t−0.5| to two decimals; every probe pass was gated by reproducing banked values first).

Every A4 candidate on the accuracy × novelty plane, with φ encoded, is Figure 6; the full N&P&F verdict grid is Figure 7.

family (n=64 each)	landed novelty (med)	acc mean	N&P&F@3×	@4×
soup-of-32, exact, r256	16.24	.843	0	0
soup-of-64, exact, r512	15.88	.844	0	0
soup16 + σ=1.14 push	23.58	.842	1.000	0
soup16 + σ=1.84 push	31.86	.841	1.000	1.000
extrapolation, t = −0.575 / 1.559	23.26 / 23.13	.794 / .801	.016 / 0	0
extrapolation, t = −1.29 / 2.296	30.65 / 30.71	.727 / .722	0	0

Three structural facts:

1. Unforced recombination cannot leave the ~16 shell. Soups go shallower as parents are added (16.5 → 16.24 → 15.88 for m = 16/32/64): averaging converges toward the corpus, never away. This is the anti-novelty face of the thin-shell/center geometry the merging literature exploits for accuracy (Model Stock, Jang et al. 2024; variance decay, Rame et al. 2022) — stated there as a feature, measured here as a ceiling. The previous campaign’s empty frontier is explained — its operator set had no distance-forcer.
2. The orthogonal push is free through 4×. All 64 σ=1.84 candidates sit at novelty 31.9 and pass α at .841 mean — accuracy essentially undegraded from the unperturbed soup (.843) — and all pass φ at the soup’s own functional signature (median disagreement 0.222). Noise inherits the function of what it perturbs: noise-on-corpus-finals is hollow (§5.3); noise-on-soup is deep and real, because its base is already functionally far. Full-test confirmation: 64/64 above α at 3,270 examples (mean .8416).
3. The interpolation axis cliffs. Extrapolated pairs pass at 3× (15 candidates jointly α-passing and 3×-novel across the two signs, one clearing φ) then collapse before 4× (0/128 α-passers, max .806): distance along the pair axis exits the basin that the orthogonal push never leaves. Direction beats magnitude. The axial half of this profile is consistent with ExPO’s rise-then-collapse under checkpoint-pair extrapolation (Zheng et al., 2025); the direction contrast — the same distance surviving orthogonally while dying axially — is, to our knowledge, new.

P-A4-2 (thin-manifold: no candidate of any family passes α at ≥3×; a single passer falsifies) is FALSIFIED — not by a lucky individual but by a 128-row cohort saturating N&P&F at 3× and 4×, full-test-confirmed, plus four individuals (three sampled by the learned recombiner). The competence manifold of this task extends to at least 4× the corpus’s own spread; its boundary τ*(α) is direction-dependent; and the 6× rung remains unprobed — the honest edge of the claim.

5.6 The extinction law does not transfer: a regime boundary, and a retraction (A4 arm 3)

Our first paper reported velocity extinction — a deterministic training catastrophe for flows on small zoos, striking after convergence at a characteristic horizon (τ̂ = 26·n training rows, with a registered loss-floor convergence gate L* ≈ 0.70–0.75). A3 initially adjudicated the law “missed by 20×” at adapter scale. That verdict was our instrument error, and we retract it on the record (journal entry 064): the reader compared a loss-plateau step against the wrong unit (a remembered “steps/latent” slogan — our own month-old compression of the law, which had compressed away its meaning). In correct units, the previous campaign had never tested the law at all: τ̂(n=11,264) ≈ 293k steps against 100k-step training runs.

The registered re-test (P-A4-3, two-outcome) ran the law’s own conditions with dense velocity-norm telemetry (every 100 steps; the traces are Figure 9 — compare the first paper’s Figure 5, where the same probe collapses to zero):

cell	τ̂ = 26·n	trained	loss floor (gate ≈ .70–.75)	v-norm final/max
n=880, k=128	22,880	35k (1.53×τ̂)	0.389 — gate wide open	0.95 — no collapse
n=11,264, k=128	292,864	440k (1.50×τ̂)	0.647 — under the gate	0.97 — no collapse

With the gate open and 1.5×τ̂ elapsed — the exact conditions under which the original regime collapses on schedule — adapter-corpus latents do not extinguish. Velocity norms rise, stabilize, and hold through thousands of probes. The law is a property of its original regime (small CNN/MLP-zoo latents), not of small-n conditional flow matching generally — the sharpest available answer to the spinoff plan’s domain-generality question, from the negative side.

5.7 Bars, integrity checks, and the instrument-lesson trail

• The ensemble bar (k=16 majority-vote over a fresh random-64 corpus pool, 32 draws): .806 mean, .827 max. Every top generated cell (.850–.851 max) clears it — generated candidates beat cheap prediction averaging of what already exists.
• The functionally-filtered noise bar is exactly empty at every rung — the hollow-novelty lesson sits visibly in every A4 table at zero cost.
• Label-alignment gates (predictions must recompute to banked accuracies) passed on every cell of every campaign; N&P recomputation against the canonical A3 table matched bitwise before the retrospective φ re-score was trusted; calibration self-checks reproduced banked soup novelty to 1e−3 before any shell was set.
• Three instrument errors, each caught, each cheaper than the last (entries 061, 062, 064): a contaminated novelty artifact (manifest race; corrected values now flow through a single access path), a correction that initially failed to propagate to every consumer (same lesson, sharpened), and the extinction unit error above (“a law’s units are part of the law — check the source spec, not a remembered slogan”). We publish these because an audit whose own instruments go unaudited is theater.

6. Limitations

One base model (Qwen3-0.6B, revision-pinned), one task (BoolQ), r=8 seed-only lineages; ecological validity for mixed-provenance adapter populations is untested (Design-B territory). The deep frontier is probed to 4× only; 6× is open. The σ-push is examined at exactly two magnitudes along one direction family; the anisotropy claim is a two-direction contrast, not a map. The learned recombiner’s deep tail is leakage-driven (≈1/64 per low-cond cell), not targeted; conditioning on (accuracy, novelty) jointly is future work. The P-A4-1 adjudication depends on the metric letter (N&P tie vs 3-row N&P&F shortfall); both readings are published and neither is collapsed. The recombiner inherits the corpus it recombines: nothing here demonstrates transfer beyond the training population’s task.

7. Discussion

The through-line of the series is that honest generation over weights is a coordinates problem before it is a density problem. In vision zoos, fixing the coordinates (canonicalize) was enough to let a flow over weights beat every trivial baseline. In adapter space, the same audit shows the flow’s emission geometry is the binding constraint — and the fix is to move the generator, not abandon it: learn the density over the merge operator’s coefficients, in a space whose own symmetry (parent exchangeability) is quotiented out first. Sixteen dimensions of learned recombination reproduce what a hand-built operator saturates and add steering that no fixed operator possesses. Recombine, then generate.

The falsified thin-manifold prediction reframes the novelty question. The frontier was never empty — our operators simply averaged toward the corpus. Once distance is forced orthogonally from a functionally-novel base, accuracy survives to 4× the corpus spread without measurable cost, and the functional filter confirms these are not relocated copies. The practical reading: the cheap path to novel-and-performant adapters is recombine, then push — and the learned recombiner suggests the two moves can eventually be one sampler’s job. The scientific reading: adapter competence basins are broad in most directions and cliffed along interpolation axes, which any future geometric theory of adapter space must explain.

The extinction result closes a loop with unusual cleanliness: a law we discovered, formalized with a gate, initially mis-tested through our own unit error, retracted, and then re-tested properly — whereupon it declined to generalize, and the negative is itself the paper’s answer to the law’s domain-generality question. Audits that only ever confirm are not audits.

8. Reproducibility statement

All campaigns ran from pre-registered designs ratified before firing, with numbered predictions and on-the-record adjudication (journal thread 2f020918, entries 034–071; prereg documents and both staging addenda in the repo under .research/). Every generator, bar, materialization, calibration, and scoring tool is in the repo (.research/a2-ops/a3_*.py, a4_*.py); canonical numbers ship as JSON artifacts named throughout; evaluations carry seeds, job traces, and label-alignment asserts; corrected artifacts flow through single access paths documented in the ops canon (docs/boundary-ops.md). The zoo build, sidecar evaluation machinery, and figure scripts are the first paper’s, extended per the A3/A4 eval-campaign ops section.

References

(Verified in the 2026-08-18 citation walk: three parallel sweeps — the merging canon pinned against arXiv/OpenReview/ACL Anthology/Nature/dblp records; a LoRA-generation + merge-coefficient scoop sweep including the full Zeng citation graph; a beyond-interpolation/basin-geometry sweep. Items marked ◆ carry over from the first paper’s bibliography, verified there in entries 010/018/021. No scoop found on any headline claim; nearest works: EvoGM, ExPO, Model Stock, DeepWeightFlow — each distinguished in §2 and at the result it neighbors.)

Foundations and referees

1. Canonicalize, Then Generate: Quotienting Weight-Space Symmetries Unlocks Honest Generative Models of Neural Network Weights. Haphazard Solutions lab publication, 2026. haphazardsolutions.com/knowledge/canonicalize-then-generate/. — the first paper of this series: protocol, vision-zoo audits, the velocity-extinction law.
2. ◆ Zeng, Yin, Xu, Liu. Generative Modeling of Weights: Generalization or Memorization? CVPR 2026 (Highlight), pp. 41974–41984. arXiv:2506.07998.
3. Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, Chen. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. arXiv:2106.09685.
4. ◆ Lipman, Chen, Ben-Hamu, Nickel, Le. Flow Matching for Generative Modeling. ICLR 2023. arXiv:2210.02747.
5. ◆ Frankle, Dziugaite, Roy, Carbin. Linear Mode Connectivity and the Lottery Ticket Hypothesis. ICML 2020. arXiv:1912.05671.
6. ◆ Entezari, Sedghi, Saukh, Neyshabur. The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks. ICLR 2022. arXiv:2110.06296.
7. ◆ Ainsworth, Hayase, Srinivasa. Git Re-Basin: Merging Models modulo Permutation Symmetries. ICLR 2023. arXiv:2209.04836.
8. ◆ Z. Wang, P. Wang, K. Wang. Position: Weight Space Should Be a First-Class Generative AI Modality. ICML 2026 (Position Paper track), PMLR 306. arXiv:2605.18632.
9. ◆ Zhou, Chen, Z. Li, J. Wang, Jiang, P. Li, Yu, M. Zhang, Bates, Jaakkola. Rethinking Diffusion Models with Symmetries through Canonicalization… arXiv:2602.15022 (2026). — the general canonicalize-then-generate theory.

Generative models of weights

1. ◆ Peebles, Radosavovic, Brooks, Efros, Malik. Learning to Learn with Generative Models of Neural Network Checkpoints (G.pt). arXiv:2209.12892 (2022).
2. ◆ Wang, Tang, Zeng, Yin, Xu, Zhou, Zang, Darrell, Liu, You. Neural Network Diffusion (p-diff). arXiv:2402.13144 (2024).
3. Schürholt, Knyazev, Giró-i-Nieto, Borth. Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights. NeurIPS 2022. arXiv:2209.14733.
4. ◆ Schürholt, Mahoney, Borth. Towards Scalable and Versatile Weight Space Learning (SANE). ICML 2024, PMLR 235. arXiv:2406.09997.
5. ◆ K. Wang, Tang, Zhao, Schürholt, Z. Wang, You. Scaling Up Parameter Generation: A Recurrent Diffusion Approach (RPG). NeurIPS 2025. arXiv:2501.11587.
6. ◆ Saragih, Cao, Balaji, Santhosh. Flow to Learn: Flow Matching on Neural Network Parameters (FLoWN). ICLR 2025 Workshop on Neural Network Weights as a New Data Modality. arXiv:2503.19371.
7. ◆ Erdogan. Geometric Flow Models over Neural Network Weights. MSc thesis, TU Munich (2025). arXiv:2504.03710.
8. ◆ Gupta, Biggs, Laber, Shafi, Walters, Paul. DeepWeightFlow: Re-Basined Flow Matching for Generating Neural Network Weights. ICLR 2026. arXiv:2601.05052. — reports functional-disagreement diversity (mIoU over error sets) for its own generator; see §2.

The LoRA-generation line

1. ◆ Charakorn, Cetin, Tang, Lange. Text-to-LoRA: Instant Transformer Adaption. ICML 2025, PMLR 267. arXiv:2506.06105.
2. Jin, Wang, Tang, Zhao, Zhou, Tang, You. Conditional LoRA Parameter Generation (COND P-DIFF). arXiv:2408.01415 (2024).
3. ORAL: Prompting Your Large-Scale LoRAs via Conditional Recurrent Diffusion. Findings of EMNLP 2025. arXiv:2503.24354.
4. Liang et al. Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights. arXiv:2506.16406 (2025).
5. LoRAGen: Structure-Aware Weight Space Learning for LoRA Generation. ICLR 2026 (OpenReview mrafO7aTYj).
6. Dravid, Gandelsman, Wang, Abdal, Wetzstein, Efros, Aberman. Interpreting the Weight Space of Customized Diffusion Models (weights2weights). NeurIPS 2024. arXiv:2406.09413.

Merging as practice

1. ◆ Wortsman, Ilharco, Gadre, Roelofs, Gontijo-Lopes, Morcos, Namkoong, Farhadi, Carmon, Kornblith, Schmidt. Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy Without Increasing Inference Time. ICML 2022. arXiv:2203.05482.
2. Matena, Raffel. Merging Models with Fisher-Weighted Averaging. NeurIPS 2022. arXiv:2111.09832.
3. Ilharco, Ribeiro, Wortsman, Gururangan, Schmidt, Hajishirzi, Farhadi. Editing Models with Task Arithmetic. ICLR 2023. arXiv:2212.04089.
4. Ortiz-Jimenez, Favero, Frossard. Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. NeurIPS 2023 (oral). arXiv:2305.12827.
5. Yadav, Tam, Choshen, Raffel, Bansal. TIES-Merging: Resolving Interference When Merging Models. NeurIPS 2023. arXiv:2306.01708.
6. Yu, Yu, Yu, Huang, Li. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch (DARE). ICML 2024. arXiv:2311.03099.
7. Goddard, Siriwardhana, Ehghaghi, Meyers, Karpukhin, Benedict, McQuade, Solawetz. Arcee’s MergeKit: A Toolkit for Merging Large Language Models. EMNLP 2024 Industry Track, pp. 477–485. arXiv:2403.13257.
8. Zhang, Chen, Liu, He. Composing Parameter-Efficient Modules with Arithmetic Operations. NeurIPS 2023. arXiv:2306.14870.
9. Huang, Liu, Lin, Pang, Du, Lin. LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition. COLM 2024. arXiv:2307.13269.
10. Stoica, Ramesh, Ecsedi, Choshen, Hoffman. Model merging with SVD to tie the Knots (KnOTS). ICLR 2025. arXiv:2410.19735.
11. Sigrist, Waldis. A Pipeline to Assess Merging Methods via Behavior and Internals. BlackboxNLP 2025. arXiv:2509.19476.

Learned and optimized merge operators

1. Yang, Wang, Shen, Liu, Guo, Wang, Tao. AdaMerging: Adaptive Model Merging for Multi-Task Learning. ICLR 2024. arXiv:2310.02575.
2. Akiba, Shing, Tang, Sun, Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence 7, 195–204 (2025). arXiv:2403.13187.
3. Feng et al. Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence. ICML 2025. arXiv:2410.11163.
4. Shenaj et al. LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generation. ICCV 2025.
5. Jiang et al. Learning to Merge LLMs via Evolutionary Generative Optimization (EvoGM). ICML 2026. arXiv:2605.29295. — generative model over per-layer merge coefficients inside an evolutionary search loop; distinguished in §2.
6. Minixhofer, Ponti, Vulić. Zero-Shot Tokenizer Transfer (ZeTT). NeurIPS 2024. arXiv:2405.07883.

Beyond interpolation; the geometry of averages

1. Zheng, Wang, Ji, Huang, Peng. Model Extrapolation Expedites Alignment (ExPO). ACL 2025. arXiv:2404.16792. — rise-then-collapse along the training direction; see §5.5.
2. Croce, Rebuffi, Shelhamer, Gowal. Seasoning Model Soups for Robustness to Adversarial and Natural Distribution Shifts. CVPR 2023. arXiv:2302.10164. — soups “should not be constrained to the convex hull.”
3. Maddox, Izmailov, Garipov, Vetrov, Wilson. A Simple Baseline for Bayesian Uncertainty in Deep Learning (SWAG). NeurIPS 2019. arXiv:1902.02476.
4. Ash, Adams. On Warm-Starting Neural Network Training. NeurIPS 2020.
5. Jang, Yun, Han. Model Stock: All we need is just a few fine-tuned models. ECCV 2024 (oral). arXiv:2403.19522. — the thin-shell/center geometry whose anti-novelty face is §5.5’s soup-shallowing.
6. Rame et al. Diverse Weight Averaging for Out-of-Distribution Generalization (DiWA). NeurIPS 2022. arXiv:2205.09739.
7. He, Huang, Yuan. Asymmetric Valleys: Beyond Sharp and Flat Local Minima. NeurIPS 2019. arXiv:1902.00744.
8. Gur-Ari, Roberts, Dyer. Gradient Descent Happens in a Tiny Subspace. arXiv:1812.04754 (2018).