Activation Functions: The Small Nonlinearity That Shapes a Network
· â 8 min read · âī¸ k4i
A mechanism-first guide to activation functions: why neural networks need nonlinearities, how sigmoid, tanh, ReLU, GELU, and SiLU differ, and why a 400-function survey is best read as a map rather than a menu.