FWKV
Collection
5 items • Updated
FWKV
FWKV (Feed-forward Weighted Key Value, or Floored Weighted Key Value) is a novel efficient language model architecture proposed on May 13, 2026 by Me.
Design Philosophy
FWKV is built on two simple ideas:
- FFNN as the primary compute block — fast, parallelizable, hardware-friendly
- floor(W·K·V) as the sole context mechanism — a multiplicative recurrent state with a learned decay W floored above zero, so the model can never fully forget
Sample
--- Sample generation ---
Once upon a time there was a little girl named Lily. One day, Lily went for a walk in the park and saw a butterfly. It was very pretty and had a pretty wings. Lily was so happy to have a friend and played with it all day.
But then, Lily remembered the butterfly fluttering around in the flowers and never got it again. The butterfly flew away from Lily and her friend had to catch it. Lily was so happy to have a new friend at the butterfly. The end.<|endoftext|>
-------------------------
Inference speed (CPU)
