A coding-cost explanation of entropy, cross entropy, and KL divergence: what is unavoidable uncertainty, what is the cost of coding with the wrong distribution, and why cross entropy naturally appears in classification and language-model training.

An intuition-first explanation of KL divergence: why KL(P || Q) differs from KL(Q || P), what each direction penalizes, and how this connects to SFT, cross entropy, and RLHF KL penalties.