SK Sunwoo Kim

Legacy Research Blog Category

Probabilistic Heuristics & Bayesian Search

Legacy archive for probabilistic heuristic and Bayesian search notes. New notes should use LLM & Probabilistic Approaches.

This legacy archive remains available for older links. New notes in this area should use LLM & Probabilistic Approaches.

All research notes
Probabilistic Heuristics & Bayesian Search

FlashAttention: Exact Attention as an IO-Aware Streaming Computation

FlashAttention is not an approximation to attention. Its core idea is to avoid materializing the N by N attention matrix in HBM by computing tiled attention in SRAM and maintaining online softmax statistics.

Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 4 - LLM Training