Choosing one option out of many should not cost a paragraph of generation, yet that is how most agents route today. A language model decodes a structured answer one token at a time, and on the project's own benchmark a 50-question batch takes about 11,500 ms.
ELMDE, the Equilibrium Latent-MoE Decision Engine, takes a different route. It reads the context once, lets a small recurrent core settle on a fixed point, and reads the decision directly off that point in a single pass. The same batch takes about 14 ms.
Why token-by-token decisions are the wrong tool
Constrained decoding forces an answer into a valid format, but it still pays for every token and it distorts the probabilities along the way. Splitting a choice into subword tokens makes early tokens that several options share carry more weight than they should, a bias the project calls myopic prefix-sharing distortion.
The fixed-depth classifiers that came before have their own limits. Convai Laya re-encodes the state for each question, so 50 questions mean 50 encodes, and it hits a 192-token header limit when there are many options. On Banking77, which has 77 intents, the README reports Laya falling to 0.425 accuracy for that reason.
The idea: encode once, settle, project
ELMDE sits between a fast classifier and a slow reasoning model, which the project calls System 1.5. It does three things, in order.
- Encode once. The context is compressed into a small latent cache and reused for every option.
- Settle. A recurrent core refines a handful of working-memory slots until they stop changing, at a fixed point called z*.
- Project. The decision heads read z* directly, so the answer comes from one pass instead of a generated sequence. A decision this fast is only useful if it knows when it is unsure, so the last stage is a gate that escalates hard cases to a slower model or a person. How it works Four pieces do the work. • Latent cache. Multi-Head Latent Attention stores each context as a 128-dimension vector plus a 32-dimension rotary key. Because the up-projection matrices are absorbed into the query and output matrices, full keys and values are never built. The repo's check shows a maximum deviation of 5.68e-14 from the standard computation, and a KV cache about 93% smaller in the table, 98.5% in the theorem. • Equilibrium core. A mixture-of-experts layer is applied in a damped loop, z(r+1) = (1 - a) z(r) + a Phi(z(r)), for at most six ticks. It stops early once the change falls below 1e-3. With damping 0.45 and a measured Jacobian bound of 0.1358, the contraction factor is at most 0.6111, so the loop is guaranteed to converge to a single fixed point. • Three heads. Choice scores every option in parallel, with no cap on how many. CORN ordinal keeps ranked scores monotonic by construction. A Beta head returns a probability and a closed-form variance. • Conformal gate. Selective conformal risk control sets a confidence threshold, tau* = 0.9848, calibrated on held-out data. Decisions above it are accepted with a certified error of at most 1%. Everything else is escalated. Results, and what they do not show These are the figures reported in the repository, produced by its own benchmark scripts. Measure ELMDE Compared with 50-question batch latency ~14 ms 771 ms Laya, ~490 ms Jev (cloud API), ~11,500 ms autoregressive CDM Banking77 accuracy (77 intents) 0.912 0.870 Jev, 0.425 Laya Calibration error (ECE) 0.0434 0.144 Jev, 0.081 to 0.213 Laya Conformal risk gate 99.45% certified Error of at most 1% on accepted decisions Three limits matter when reading that table. • The speedup depends on the baseline. The 55x figure is against Laya (771 ms to 14 ms). Against the autoregressive decoder it is closer to 820x. • The benchmarks are the project's own. Nobody outside the project has reproduced them yet, and Jev is a hosted API whose setup I cannot control. Treat the comparison as a claim to test. • The certificate has a condition. Conformal guarantees hold when live data resembles the calibration data. If the distribution shifts, the 1% bound can stop holding, which is why the repo also includes an online adaptive variant. Try it, and tell me where it breaks The repo runs on Python 3.10+ and PyTorch 2.2+, and python elmde_engine.py runs the mathematical checks first. After that, four drop-in patterns in example_injection.py show it in use: • fast-pathing agent tool calls in under 35 ms with zero generated tokens • routing across 77 or more intents • monotonic triage tiers for clinical or legal queues • a decorator that escalates risky calls before they execute I am most interested in three kinds of feedback: someone reproducing the benchmarks on their own hardware, a dataset where the gate fails, and any flaw in the contraction argument. Open an issue on the repository or send a pull request. The code is Apache 2.0.
#ai #programming #opensource #llm #software #coding #development #engineering #inclusive #community
Building ELMDE: Deciding in One Pass
