You Could Have Come Up with Kimi Delta Attention

You Could Have Come Up With Kimi Delta Attention Jamie Dborin Founder & Member of Technical Staff, Doubleword Math notation ⟨k|q⟩ kᵀq A note on notation: this article defaults to bra-ket notation because (in my quantum-inspired opinion) it makes the shapes in this derivation very clear. The Math notation switch above rewrites every equation using conventional bold vectors and explicit transposes instead. In bra-ket mode, ∣ q ⟩ \lvert q\rangle ∣ q ⟩ is a column vector, ⟨ k ∣ \langle k\rvert ⟨ k ∣ is a row vector, ⟨ k ∣ q ⟩ \langle k\rvert q\rangle ⟨ k ∣ q ⟩ is a number, and ∣ v ⟩ ⟨ k ∣ \lvert v\rangle\langle k\rvert ∣ v ⟩ ⟨ k ∣ is a matrix. Vectors face right by default, while keys face left when written into the linear-attention state. We work with one causal attention head and real-valued vectors, assume DeltaNet’s keys are normalized, and let the state map from key space to value space.