Where this architecture actually stands

A genuinely linear-time attention path, a Hebbian state matrix that is written on every step and read by nothing, and a documentation layer that describes the second as the architecture's core contribution.

An independent, from-scratch implementation of a brain-inspired linear-attention language model: a ~4.79M-parameter model with a genuinely linear-time attention path, trained briefly on TinyStories, alongside a Hebbian state matrix whose update is implemented but whose read path was never wired up — so the memory that the documentation describes as the architecture's core contribution does not currently affect any output. The linear-attention property is real; every memory and stability claim around it is unsupported, and two of them are directly contradicted by the repository's own benchmark output.

The scale qualifier matters: the headline TinyStories run's log reaches step 79,200 of 356,657 — about 22% of one epoch — and ends there with no completion marker and no checkpoint saved. measured · run

training_bbpe_tinystories.txt — last logged line. A separate 29.38M configuration did complete: 7,812/7,812 batches, final loss 5.9990, 11.9 minutes.

The Hebbian state is written on every step and read by nothing. The method that would fold it into the output, _combine_states, is defined in three files and called from none of them. So the attention is genuinely linear-time, and the memory is genuinely instrumented — and no output this model produces depends on it. That is a specific, fixable missing read path, not a dead idea: the state update exists, is correct in form, and is computed on every forward pass.

The linear attention is real and it is in the code: no N by N table is ever built. The state update is real too, and it is computed on every forward pass — but it never reaches the output, because the only method that would fold it in is defined in three files and called from none. What is left is a small transformer-style model with unnormalised linear attention and an instrumented state, trained to about a fifth of one epoch on TinyStories. No run in this repository compares it to a transformer at matched parameters and matched tokens, and the memory and stability numbers in the project's own README are contradicted by the project's own benchmark JSON.

Every number on this page carries exactly one label. Structural claims — statements that are true by construction — carry none, and are followed by the caveat that limits them. The claims this project's documentation makes that its own files contradict are gathered in one place, the claims ledger, and appear nowhere else on the page.

What was built, run, and measured

Everything in this section has a raw artifact in the repository. None of it is a quality comparison, and none of it is a completed training run.

The recurrence that is specified, and the one that runs

The design specifies a matrix that is written by co-activation, decayed by time, and read back with a query. The update exists, and runs on every forward pass. The read is not called.

Two equations describe the intended mechanism. The state is written, per head and per scale, as E ← γ·E + η·W, and it is read as y = q·E. In the shipped configuration the write is W = diag(q ⊙ v); the full outer product W = q vᵀ is implemented behind a flag that is off. Neither one is fed back into any line that produces an output.

Three structural properties, each with its limit

• Structural: the state is a fixed O(d²) per head instead of a cache that grows with context. project-reported · derived

  O(d²) per head can exceed a KV cache at short context; the advantage only appears at long context — and here the state it saves is not used, because nothing reads it.

• Structural: the memory is an explicit object rather than a hidden activation. measured · code

  Readable is not interpretable: no interpretability experiment was run, and in this code the object is not read at all.

• Structural: the write rule is local and Hebbian. Backpropagation still trains every weight; the write itself is local. measured · code

  Nobody has shown this scales. That is the open question, not a result.

The repository's own summary of the feed-forward network states that its ReLU leaves about 5% of neurons active. No sparsity histogram exists in the repository, and an analytic estimate from the initialisation puts the zero fraction nearer half. project-reported · doc-value

The same product, computed in two different orders

Q(KᵀV) equals (QKᵀ)V. The difference is what has to exist at the same time: one order builds an N by N table, the other never does.

Per head, ignoring batch: the quadratic order costs 2N²D multiply–adds and needs an N²-cell intermediate; the linear order costs 2ND² multiply–adds and needs a D²-cell intermediate. The two counts are equal when N equals D. At the benchmark head dimension of 32, that crossover is at 32 tokens; at 2,048 tokens the count ratio is 64.

Operation counts are not time. A count ratio is a statement about arithmetic, not about wall-clock speed, and nothing in it says which order finishes first on a real device. project-reported · derived

The right-hand panel below is what this code runs, and it is not standard attention: there is no softmax, no denominator and no 1/√D scaling anywhere in the attention path. measured · code

No wall-clock measurement as a function of sequence length exists in this repository, so the claim that the linear order is faster is not tested here in either direction. not reproduced · not run

Three decay rates, and what they would carry

γ^Δ is exact: a write made Δ steps ago survives in scale i at exactly γᵢ^Δ times its original size. That is arithmetic of the update rule, not a measurement of what the model remembers — in the shipped code the state is not read by anything, so it is not a description of the model's behaviour either.

With the code's own rates, the arithmetic horizon is: at 500 tokens, γ^Δ is 7.3e-12 for the fast scale, 6.6e-03 for the medium and 8.2e-02 for the slow. At 2,000 tokens only the slow scale is still above 4.4e-05, and none of the three is above the 1e-3 floor this page assumes.

The measured counterpoint. The one executed measurement in this repository's benchmark JSON reports 6.91e-05 for the baseline arm and 7.20e-05 for the arm labelled multi-scale — but that number comes from an untrained 4.79M model, its "multi-scale" arm is only a different single decay constant, and the quantity is a state-norm proxy rather than a fraction kept. It is not the percentage gap the documentation states, and it is not a measurement of memory at all. The design extends the arithmetic horizon; nothing on record shows the extension being used, and nothing on record shows it could be. measured · run

benchmarking/results/benchmark_results.json:34 · the same file's create_multiscale_config() is state_decay=0.995 with the comment "future: multiple decay rates", and the suite benchmarks freshly constructed, untrained models.

The unit that motivated the design

A leaky integrate-and-fire neuron with a threshold that rises as it fires. It integrates, spikes, resets, tires — and under sustained input it settles into a rhythm instead of saturating.

Why the lineage matters to the argument on this page: the Hebbian programme's unit is local, unlabelled and cheap. A threshold that adapts, and a write that depends only on the pre- and post-synaptic pair at one moment, is a different kind of object from a global gradient step. Whether that buys anything at scale is the open question this page keeps open — and nothing in this repository tests it, because the unit itself was never implemented here.

Claims this project makes that its own artifacts contradict

Each row is a sentence from the project's own documentation, followed by what the project's own files say. The struck text is the claim. The text after it is the artifact.

Ten documentation claims

Multi-scale retains 15% at 500 tokens vs baseline 0.6%, a 24× improvement

Reproduced to four different values in four places: 15%, 8%, 10.2%, and a measured 7.2e-05. The only executed measurement is a different quantity — a state-norm proxy, not a percentage — produced by a suite whose "multiscale" arm is just state_decay=0.995 ("future: multiple decay rates") on untrained weights. No measurement can yield four values for one quantity.

not reproduced · contradicted

Baseline 0.6% at 500 tokens is a measurement

baseline_metrics.json metadata calls its values expected/placeholder; 0.0065 is the arithmetic of 0.99^500.

project-reported · placeholder

95% stable runs vs 40% baseline

Measured stability is 0.7644 / 0.7546, and the placeholder file the claim was drawn from says stability 0.7 → 0.85. The row contradicts both the measurement and its own source.

not reproduced · contradicted

2.75× faster training with BBPE (27.5K vs 10K tok/s)

The completed 29.38M run is logged at 273,182 tok/s on the RTX 4070; the 10K baseline is a placeholder. The 27.5K figure originated as a hardcoded literal in a docx-generating script.

not reproduced · contradicted

10.23M parameters

No configuration in the repository yields it. The training config computes to 4,790,784 and the run printed that number; the benchmark config computes to 559,360. The summary's own component list does not sum to its stated total.

not reproduced · contradicted

Perplexity competitive with GPT-2-scale models; ppl_wikitext2_100m: 23.5

23.5 is the placeholder multiscale value in a template file, promoted into a SOTA table beside GPT-2's 28.5. The only executed perplexity is 276.08 / 290.31 for the 4.79M config on a 7,256-byte Shakespeare snippet after about 33 optimizer steps — near-uniform output.

not reproduced · contradicted

The first architecture to combine true Hebbian outer-product state updates, multi-scale decay, positive-orthant constraints and multiplicative gating

No literature review supporting priority exists in the repository, and the outer-product flag is false in every config that produced a result.

not reproduced · unverified

230× compression by distilling a 1.1B teacher

1,100,048,384 ÷ 4,790,784 ≈ 229.6, so the arithmetic is right — but it is a parameter ratio from a 200-sample smoke test, presented as a distillation result. The run passed 200 samples through the model 50 times.

project-reported · doc-value

Trained from scratch; 50 epochs

The recorded TinyStories run's log ends at step 79,200 of 356,657 — about 22% of one epoch — with loss 4.3805 and no completion marker (and the log is provably incomplete: zero checkpoint lines where the code would print seven). The separately completed run reached loss 5.9990 after 7,812/7,812 batches. The "50 epochs" log is 200 synthetic samples, 25 batches, 50 passes in 2.9 minutes, ending at loss 2.2710 by memorising 200 samples.

not reproduced · contradicted

Memory retention improves with multi-scale decay

Not computable from the shipped forward pass: the state is written and never read, and the one method that would fold it into the output is defined three times and called from none. Every claim of this kind is downstream of a read path that does not exist.

not reproduced · uncomputable

The same drift inside the code

Six findings from reading the implementation itself. The fifth is the section's only positive row and is set apart: the linear-attention property is genuinely present in the code, unlike everything above it.

The outer-product flag is true in the shipped configuration

It defaults to False in both config definitions and to False again in the benchmark suite's model factory and ablation configs, so the diagonal write is what every JSON in benchmarking/results/ describes. With the flag off the write is torch.diag_embed(Q_flat * V_flat), touching D of D² entries; with it on, torch.einsum("hd,he->hde", …) is a genuine per-head outer product, and it is never enabled.

measured · code

The state is written per token, in arrival order

Both implementations collapse the sequence before the write: Q.mean(dim=(0,1)) and V.mean(dim=(0,1)). The state is a decayed running average of per-batch mean co-activations, not a per-token sum, and it carries no token order.

measured · code

All three implementations write a full Hebbian matrix

implementation/multiscale_bdh.py has no outer product at all: hebbian = Q_flat * V_flat then reshape(C), a [C] vector added to a [C,C] matrix, so every row of that matrix stays identical. In bdh_gpu_10m.py the state is a length-C vector whose "outer product" is torch.einsum('hd,hd->hd', …), an element-wise product, despite the comment above it saying outer product.

measured · code

Q and V are rectified, so the state write lives in the positive orthant

Q and V are not ReLU'd. ReLU appears only in the feed-forward hidden layer; the gate is a sigmoid. Nothing in the state write constrains the sign of its inputs, and the recurrence plate draws signed values for that reason.

measured · code

The retrieval path never materialises an N×N table

attn = matmul(K_T, V) gives [B,H,D,D], and matmul(Q, attn) gives the output. This one is real, and it is the property the whole architecture rests on. Its limit is stated with it: the same lines apply no softmax and no 1/√D scaling, so the output is not normalised the way standard attention is.

measured · code

Rotary position encoding is active in the shipped model

RoPE is disabled. In bdh_v2_clean.py it is gated by use_rope=False; in multiscale_bdh.py the forward pass has it commented out with # TEMPORARILY DISABLED - causing shape mismatch errors. Whatever order the attention carries is whatever the matmuls give it.

measured · code

Every number this page quotes, and the model it belongs to

The repository conflates three configurations. No number on this page belongs to more than one of them.

What the code does not do

Eight things this repository does not establish. The first is not a missing experiment; it is a missing line of code.

1. The memory does not function at all. The state is written on every forward pass and read by nothing; _combine_states has three definitions and zero call sites. Every memory and context claim in this project is downstream of this, and none of them is computable from the shipped forward pass. measured · code
2. No head-to-head against a transformer at matched parameters and tokens. The comparison table is curated literals and hand-assigned 1–5 scores, on two of which the project awards itself 5/5. It is not an experiment. not reproduced · not run
3. No scale, and no completed run recorded at the headline configuration. ~4.79M parameters on TinyStories, with the captured log ending at about 22% of one epoch and no checkpoint kept. A different 29.38M configuration did complete — 7,812/7,812 batches, final loss 5.9990 — and it is not the configuration the documentation describes. not reproduced · unverified
4. No long-context evaluation. Nothing beyond about 2k tokens; the O(N) advantage is structural, never demonstrated on a task. not reproduced · not run
5. The retrieval tests were run, and the model failed them. Needle-in-haystack scored 0.07 / 0.18 / 0.32 / 0.34 at 32/64/128/256 tokens, and exact associative recall scored 0.0 for every configuration tried — both from the 0.56M v2 configuration on synthetic data, 100 samples. For fixed-state linear attention this is the decisive test, and the honest statement is the result rather than its absence — a result produced by a model whose state is not called anywhere in its forward pass. measured · run
6. No checkpoints survive, so nothing can be re-run from the repository as it stands. The absence is verifiable: there is no checkpoints/ directory and no .pt or .pth file anywhere in the tree. not reproduced · contradicted
7. The paper is third-party work. Kosowski et al., arXiv:2509.26507; this repository is an independent implementation plus extensions, and nothing here reproduces its results. paper · arXiv:2509.26507
8. The compression figure is a parameter ratio from a 200-sample smoke test, not a capability or quality result: 1,100,048,384 ÷ 4,790,784 ≈ 229.6. project-reported · doc-value

None of these is an argument that the approach cannot work. They are the list of things that would have to be true before any of the project's quality claims could be.

Where this sits, by evidence rather than by capability

Two axes: the size of the largest model in a reported result, and the strength of the evidence behind the reported quality. Published systems are plotted from their papers. This project is plotted twice, and the gap between its two markers is what this page is about.

The vertical axis is ordinal, in four levels, and it is this page's judgement — but each level is defined by a checkable criterion, so every placement can be audited:

1. Level 1 — contradicted by its own artifact. A file in the repository says otherwise.
2. Level 2 — reported in prose, no artifact. The number appears in a document and nowhere else.
3. Level 3 — artifact in this repository, reproducible from source. The raw output exists.
4. Level 4 — published result from a third party, with paper and released weights. Not reproduced here.

The horizontal axis is model size in parameters on a log scale, because size is a fact about a result that can be checked against a config or a paper. It is not a performance axis, and the vertical axis is not a measurement of quality.

What would move any of this

Each item converts one claim from unproven to measured, or kills it. Nothing on this list has been run.

1. Wire the read path. Call _combine_states and use the combined state in the attention block, so that a token's output depends on what earlier tokens wrote. Then re-run the retention protocol. This is the only item here that is a code change rather than an experiment, and it decides whether any other memory claim can exist — today the method is not called at all. not reproduced · not run
2. A matched-parameter, matched-token transformer baseline on the same corpus and the same tokenizer. not reproduced · not run
3. One long-context task beyond 2k tokens with a memory measurement attached, so that the O(N) claim meets a task rather than a shape. not reproduced · not run
4. An associative-recall evaluation — MQAR or needle-in-haystack — which is the decisive test for fixed-state linear attention, and which the shipped code's state is not called into at all. not reproduced · not run
5. The outer-product ablation the flag already exists for. Train both variants and publish both JSONs. not reproduced · not run
6. Preserved checkpoints with a hash, so that any of the above can be re-run rather than re-derived. not reproduced · not run
7. A sparsity histogram to replace the ~5% doc value with a measurement. not reproduced · not run
8. A re-run of the retention protocol with the values published as they come out, whatever they are — which the shipped forward pass does not read today. not reproduced · not run