#transformer-research — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #transformer-research, aggregated by home.social.
-
Kimi Linear is a really interesting way to circumvent the KV cache bottleneck for long-horizon tasks. Basically, it's a hybrid model that blends Linear Attention, State Space Modelling and MLA with NoPE encoding to learn a constant state per head instead of growing a cache.
The gating and rank-1 delta-rule correction ensure the memory doesn't accumulate forever and we can have both weight decay and correction
-
Kimi Linear is a really interesting way to circumvent the KV cache bottleneck for long-horizon tasks. Basically, it's a hybrid model that blends Linear Attention, State Space Modelling and MLA with NoPE encoding to learn a constant state per head instead of growing a cache.
The gating and rank-1 delta-rule correction ensure the memory doesn't accumulate forever and we can have both weight decay and correction