Matrix models of memory, pioneered by James Anderson (1970), Teuvo Kohonen (1972), and others in the early 1970s, represent associations between items as matrices formed from the outer products of item vectors. These models provided the first rigorous linear-algebraic framework for understanding how distributed representations could support memory storage and retrieval.
The Basic Framework
Items are represented as column vectors in an N-dimensional space. The association between cue vector a and target vector b is encoded as the outer product b·aᵀ, yielding an N×N matrix. Multiple associations are stored by adding their outer-product matrices into a single composite memory matrix:
Retrieval is performed by multiplying the memory matrix by a cue vector, producing a noisy approximation of the desired target:
When the cue vectors are orthogonal, the noise terms vanish and retrieval is perfect. The capacity of the system is therefore related to the number of near-orthogonal vectors that can be packed into N-dimensional space.
Capacity Limits
The critical theoretical result is that a matrix memory storing m associations in N dimensions can achieve accurate retrieval as long as m is proportional to N. When m exceeds this capacity, the cross-talk noise from other stored associations overwhelms the target signal. If vectors are drawn randomly, the expected signal-to-noise ratio is approximately N/m, providing a principled account of memory capacity limitations.
Relationship to Neural Networks
Matrix models are mathematically equivalent to single-layer linear associative networks. The matrix M corresponds to a weight matrix connecting input units (cue features) to output units (target features), and storage via outer products corresponds to the Hebbian learning rule: ΔW = η · b · aᵀ. This connection between abstract mathematical memory models and neural network architectures was foundational for the development of connectionist models of cognition.
Extensions and Limitations
Pike (1984) showed that matrix models can account for recognition memory by using the trace of M · probe · probeᵀ as a familiarity signal. Humphreys, Bain, and Pike (1989) extended matrix models to handle three-way associations using tensor products, creating a more powerful framework for context-dependent retrieval. The main limitation of basic matrix models is their linearity: they cannot store nonlinear mappings without additional mechanisms.
The performance of matrix models depends critically on the orthogonality of stored item vectors. In high-dimensional spaces, random vectors are approximately orthogonal (their dot products are near zero), which is why distributed representations with many features can store many associations. This mathematical insight explains why neural systems use high-dimensional, sparse representations.