Key‑value Cache

A key‑value cache is a lightweight auxiliary structure used by auto‑regressive transformer models during inference. For each processed token, the model’s attention mechanism produces a set of "keys" and "values" that capture how that token should be attended to by later positions. Rather than recomputing these projections every time a new token is generated, the cache stores the keys and values for all previously seen tokens so they can be retrieved instantly when forming the next attention query.

The cache matters because it collapses the quadratic cost of self‑attention into a linear one with respect to sequence length during generation. By reusing already‑computed projections, models can generate long outputs with substantially lower latency and memory bandwidth, making real‑time applications—such as interactive chatbots, on‑device assistants, or streaming transcription—practical even for very large architectures.

Key‑value caching appears whenever a transformer works in a step‑wise fashion: language model decoding, beam search, incremental translation, and any scenario that processes input token by token while retaining the ability to reference earlier context. The pattern is independent of a particular paper or model size; it is a generic engineering primitive baked into most modern large‑scale sequence models.