The Kullback-Leibler (KL) divergence, introduced by Solomon Kullback and Richard Leibler in 1951, quantifies the difference between two probability distributions. Originally termed "discrimination information," it measures the expected number of extra bits needed to code samples from distribution P using a code optimized for distribution Q. In mathematical psychology, KL divergence is central to model selection, Bayesian updating, and the analysis of information processing.
Definition and Properties
For continuous distributions:
D_KL(P‖Q) = ∫ p(x) · log(p(x) / q(x)) dx
Properties:
D_KL(P‖Q) ≥ 0 (Gibbs' inequality)
D_KL(P‖Q) = 0 iff P = Q
D_KL(P‖Q) ≠ D_KL(Q‖P) in general (asymmetric)
The non-negativity of KL divergence follows from Jensen's inequality applied to the convex function −log(x). The asymmetry is a crucial property: D_KL(P‖Q) measures the cost of using Q to approximate P, which differs from the cost of using P to approximate Q. This asymmetry has practical consequences — using the wrong direction of KL divergence in model fitting leads to qualitatively different approximations, a distinction exploited in variational inference.
Relation to Likelihood Ratio and Bayesian Inference
KL divergence is intimately related to the log-likelihood ratio. If P and Q are the probability models for two hypotheses, then D_KL(P‖Q) equals the expected value of the log-likelihood ratio under P — the expected evidence per observation favoring P over Q. This connection makes KL divergence fundamental to hypothesis testing: Stein's lemma shows that the error exponent of the Neyman-Pearson test equals the KL divergence between the hypotheses.
Akaike's Information Criterion (AIC) is grounded in KL divergence. Akaike (1973) showed that the expected KL divergence between the true data-generating distribution and a fitted model is asymptotically estimated by −2·log(L) + 2k, where L is the maximum likelihood and k is the number of parameters. Thus AIC selects the model that minimizes expected information loss — a direct application of KL divergence to the everyday practice of model comparison in psychology.
Applications in Psychology
In cognitive modeling, KL divergence provides a principled measure for comparing competing models of human behavior. When fitting models to response distributions, minimizing KL divergence is equivalent to maximum likelihood estimation. In Bayesian models of cognition, the KL divergence between prior and posterior quantifies the information gained from observed data — a formalization of "learning" as the reduction of divergence between belief and reality.
In the free energy principle, variational free energy is defined as a KL divergence between an approximate posterior and the true posterior plus a constant. The brain, under this framework, minimizes KL divergence by updating its internal model to match the statistics of the environment. KL divergence also appears in information-theoretic analyses of neural coding, where it quantifies how well a neural population discriminates between stimuli.