how online softmax extends the fused kernel to handle rows that exceed sram capacity, using a numerically stable 2-pass tiling algorithm.

why the key-value cache avoids redundant computation in autoregressive decoding, and the memory/compute tradeoffs it introduces.

how to write a fused softmax kernel in triton that eliminates redundant memory accesses and outperforms pytorch's native implementation.

understand batch gradient descent, stochastic gradient descent, and mini-batch gradient descent.

understand how backward propagation works in gradient descent.