geolip-multirotary-diffusion-collective
Research prototypes of a diffusion denoiser built as a collective: one shared body, one closed-form spectral anchor, and a set of members (experts) that each have a declared purpose and an address. This repository is a research log: result files, training logs, sample sheets and weights of small, fast experiments that test the mechanisms one at a time. It is not a released model.
What the experiments found (short)
- Depth belongs in a shared trunk. The same seven blocks read 0.3605 in a row, 0.3632 as a deep trunk with the band members as a thin last layer, and 0.3712 as a shallow trunk under two-block parallel members, which loses even to four plain blocks. Over four times the training the deep-trunk collective stays a steady 0.8% behind the plain model.
- A band is owned by filtering the output, not by a rotary table. Rotary "rung" tables alone do not make members divide the work. With each member's output filtered to its band, switching all members off costs exactly the sum of switching each off (1.0000000), for 0.7% of loss on a deep trunk. A member whose attention carries the whole rotary ladder is slightly better than one restricted to its own rung.
- The closed-form anchor helps only when the model reads it. Alone it delivers 81.5% of a trained model's gain over predicting nothing. Added to the output unseen it hurt the collective; read at the embedding as well, it helped every model and made runs agree across seeds.
- An expansion is best added in the trunk. On a frozen model, three blocks added in the trunk (one per noise window, silent at birth) gain 1.7 times what nine band members added side by side gain, and can be removed exactly. Growth of this kind is not a shortcut: at equal steps it matches training everything on.
- Each band's expert matters most where the anchor's gain for that band passes through zero, t = 1 / (1 + S). Where one more expert pays is a different question and depends on how the objective weights the noise levels.
- Noise stages pay upstream of the members and crossfaded; hard windows tie a shared block.
How sure: six seeds of the base give a standard deviation of 0.0014. Findings 1 (the tower shape), 3, and 4 rest on
differences of 0.006 to 0.019. The deep-trunk collective's price against plain (0.003), the whole ladder against own rungs
(0.001), and the crossfade's edge over a shared block (0.001 to 0.002) are small: the same sign on both seeds, suggestive
rather than settled. The exactness results are equalities to rounding, not statistics. See reports/08_how_sure.md.
The design in one formula
a = anchor(z_t, t) closed form, no trained number
h = body( embed([z_t, a]), t, class ) the body reads the noisy tokens AND the anchor's answer
v_hat = a + body_out(h) + sum over members m of gate_m(t, class) * keep_band_m[ member_m(h, t, class) ]
Flow matching throughout: z_t = (1 - t) * noise + t * picture, the target is picture - noise, and t = 0 is pure noise.
| part | purpose | address | how its contribution is read |
|---|---|---|---|
| body | turns the noisy field, the class and the noise level into one state that every member reads | none, always on | the loss with every member off |
| spectral anchor | the physics under the collective: the best linear answer at every spatial frequency and noise level, in closed form from the measured power spectrum; no trained number | none | the loss of the anchor alone; what training adds over it |
| frequency member | the content of one band of wavelengths: its attention carries one rung of a rotary ladder, and what it writes is filtered to its band | its band | the loss gained when it is switched off, by band |
| noise member | one stage of the noise level: on inside its window, exactly zero outside | its window of t |
the loss gained when it is switched off, by noise level |
| content member | one kind of content | a class group | the loss gained when it is switched off, by class group |
| gate | weights computed from the address alone; nothing is picked by comparing candidates | - | the exactness of the zero outside an address |
| expansion | a member added to a trained collective and trained alone | one cell (band x stage, optionally x class) | its cell improves; every other cell stays bit-identical |
The anchor's gain per frequency, for a picture power S at that frequency and unit noise power:
g(S, t) = (t * S - (1 - t)) / (t^2 * S + (1 - t)^2) sign change where S = (1 - t) / t
Bands are disjoint filters on the token grid and noise windows are disjoint, so the squared error is a sum over cells (band x noise stage). A member can then only change the error of its own cell, which is what lets members be measured, removed, added, trained alone and merged.
A "rung" here is one band of wavelengths of a single rotary ladder (2 to 64 tokens on each axis of the token grid). The usual rotary rule gives no pair finer than 6.28 tokens a turn, so a ladder that reaches 2 tokens is built explicitly.
Beds and recipe
| bed | what it is | what it can and cannot say |
|---|---|---|
| fields | 16 x 16 tokens of 4 numbers; fresh Gaussian fields with a power spectrum measured on image latents (power falls with frequency at slope 1.225); the exact best loss is known | exactness, behavior, division of labor; the best answer is a linear filter, so it cannot show what learning adds over the anchor |
| pictures | CIFAR-10 at 32 px, patches of 2 x 2 pixels -> 16 x 16 tokens of 12 numbers; the class as a one-token caption; held-out = the test set | real edges, objects and classes; a trained model beats the linear filter by a wide margin; small pictures in pixel space, a steep spectrum |
| pictures with a latent's spectrum | the same pictures with every token channel refiltered on the token grid so that its power spectrum is the one measured on image latents (slope 1.225) | real picture structure whose frequency bands meet the noise in the middle of the noise range, as a latent's do |
Width 128, 2 heads, pure Adam (no weight decay), learning rate 1e-3 after 100 warm-up steps, batch 128, float32, 6,000 steps unless a page says otherwise (2.4 to 3.4 minutes a run on one RTX 4090). Two seeds for any claim. The bar for a difference is the larger of the same-seed repeat gap and the seed spread. Losses are read on held-out data at ten fixed noise levels (t = 0.05 ... 0.95) with fixed noise. Marks and forecasts are written down before each run; the reports say which forecasts were wrong.
The ledger
| # | question | where | outcome |
|---|---|---|---|
| 1 | Do towers that each carry one rung of a rotary ladder divide the work by frequency band, at no cost in loss? | experiments/e1_fields_all/README.md |
Not worse: no arm differs on the loss (bar 0.00058; the repeat is exact). Division: no. A rung table alone does not make a tower own its band; only the coarsest rung behaves apart. The Gaussian bed is at its ceiling (every arm within 0.3% of the exact best loss). |
| 2 | How much does the closed-form anchor deliver, and does it help a trained model? | reports/02_anchor_and_band_keeping.md |
The anchor alone delivers 81.5% of a trained 4-block model's gain over predicting nothing. Added blindly at the output it does nothing for a plain model and HURTS the collective (0.405 against 0.387). Read by the body as well, it helps the collective by 3% (0.380 against 0.392, both seeds). |
| 3 | What does filtering each member's output to its band cost, and what does it buy? | reports/02_anchor_and_band_keeping.md |
It buys exact independence (all members off = the sum of the single removals, 1.0000000 in every run; 1.2 to 1.95 without the filter). It costs 0.006 to 0.007 of loss (1.6%) without the anchor. Rung tables alone do not assign bands: an unfiltered mid-rung member writes 80% of its output in the fine band. |
| 4 | Does a rotary ladder that reaches 2 tokens beat the usual one? | reports/02_anchor_and_band_keeping.md |
In a plain 4-block model: lower on both seeds, by 0.0013 to 0.0020 (small against the six-seed spread: suggestive). Inside the collective: not run. |
| 5 | Where do expansions pay (band x noise window), on pixels and on a latent-like spectrum? Does a member's measured importance predict where an expansion pays? Is the bookkeeping exact? | reports/03_expansion_maps.md |
Nine expansions trained together on a frozen base: 0.3797 -> 0.3720 (pixels), 0.6058 -> 0.5958 (latent-like). On both beds a little over half of the gain lands near the picture (t 0.65-1); the middle window gets 0.27 (pixels) and 0.35 (latent-like), although the middle is where training as a whole adds most on the latent-like spectrum. Importance predicts gain on pixels (rank correlation 0.77 / 0.68), weakly on the latent-like bed (0.47). Exact: an expansion changes nothing outside its window (0.0), and trained with eight others it equals the one trained alone to 4e-9. Each band's member matters most near t = 1 / (1 + S), the noise level where the anchor's gain for that band passes through zero. |
| 6 | Where should the blocks go: in a row, in a deep shared trunk with the members last, or inside the members? | reports/04_where_the_blocks_go.md |
The same seven blocks: in a row 0.3601 / 0.3609; trunk-heavy (4 body blocks, then the 3 band members) 0.3630 / 0.3633; tower-heavy (1 body block, then 3 members of 2 blocks) 0.3709 / 0.3715, which loses even to four plain blocks (0.3669 / 0.3674). With the depth in the shared trunk the members cost 0.7% and keep exact independence; nine parallel expansions on a shallow body do not make up for depth. |
| 7 | Noise stages: shared or specialized, upstream of the members or at the output, hard windows or crossfaded? | reports/05_noise_stages.md |
On the latent-like bed: a noise-staged trunk block with hard windows ties a shared block (it loses at the first level inside each window and wins near the picture); with neighbors crossfaded over 0.06 of the noise level it beats the shared block on both seeds (0.5899 / 0.5902 against 0.5910 / 0.5919). Upstream of the members a stage expert is worth about twice what it is worth beside them at the output. Splitting the band members by stage does not pay at this length. |
| 8 | What form should an expansion take: members side by side, or a block in the trunk? | reports/06_expansions_in_the_trunk.md |
On a frozen base, three blocks added in the trunk (one per noise window, silent at birth, one runs a picture) gain 0.0136 / 0.0141 where nine band members side by side gain 0.0077 / 0.0083. Skipping them gives the base back exactly. On the trunk-heavy base the same three blocks take 0.3630 / 0.3633 to 0.3555 / 0.3551. |
| 9 | Can members be trained alone and merged? Do content members pay? Does the deep-trunk collective keep up over longer training? Do members need their own rungs? | reports/07_merge_content_long_runs.md |
Members trained in separate runs on a frozen body merge by name with nothing lost (the merged gain equals the sum of the solo gains to 3e-11, and the merged model equals the jointly trained one to 3e-10); building the body first and the members after costs 3% against end to end. Class-addressed expansions do not beat unaddressed ones of the same size; they come closest at the noisy end, where the caption matters. Over 6,000 / 12,000 / 24,000 steps the deep-trunk collective stays 0.003 behind the plain model. Members reading the whole rotary ladder are slightly better than members restricted to their own rung. |
| 10 | How sure are these? Does the arrangement result hold on the second bed? What does a middle-weighted noise schedule change? | reports/08_how_sure.md |
Six seeds of the base: standard deviation 0.0014 (the page sorts every finding into large, moderate, small). On the latent-like bed the deep-trunk collective is 0.7% behind plain (0.5779 / 0.5770 against 0.5737 / 0.5727), down from 3.9% with a shallow trunk. Under a middle-weighted schedule the base is better through the middle and worse at the ends, and about half of an expansion map's schedule-weighted gain lies in the middle window (0.53 / 0.49), against 0.27 to 0.36 with even weights. |
Registered and not run: the side-by-side expansion map on the deep-trunk base; expansions that read an expert's own
state; several expansions in one cell; the content map on the latent-like bed; the usual rotary ladder inside the
collective; unfiltered members with member dropout; the shallow pair at 24,000 steps. reports/00_all_runs.md lists every
finished run with its loss per seed (46 jobs, 99 trained runs).
Limits of everything above: small models (1.5 to 5 million numbers) and short training; class labels, not captions; no image latents yet (the latent-like bed is CIFAR with a latent's spectrum); losses, not picture scores; two seeds a comparison, and the six-seed spread was measured on the pixel bed only.
Layout
README.md this page
reports/NN_name.md one page per question: the verdict in plain words, then the full tables
experiments/<tag>/*.json one result file per run (losses by noise level and band, each part's off-cost, settings)
experiments/<tag>/*.pt the trained weights of every run
experiments/<tag>/*_samples.png a sheet of samples (picture runs)
experiments/<tag>/*_log.txt the training log
Run names carry their parameters: bed, arrangement, seed, steps, width.
The code that produced these files is not part of this repository.