LLM Attention Kernels and GPU Primitives
· â 1 min read · âī¸ k4i
A series index for LLM attention kernels and GPU primitives: fused softmax, online softmax, FlashAttention, PagedAttention kernels, Triton/CUDA, and memory-access optimization.
Attention
From Absolute Positional Encoding to RoPE: Why Position Can Be a Rotation
· â 10 min read · âī¸ k4i
A step-by-step explanation of positional encoding in Transformers, from absolute embeddings to sinusoidal encodings, Euler's formula, and rotary position embeddings.
