maximum likelihood in exponential families
(30 minutes to learn)
Summary
For any exponential family model, the maximum likelihood parameters are such that the model moments match the data moments.
Context
This concept has the prerequisites:
- maximum likelihood
- exponential families
- optimization problems (We characterize the maximum likelihood solutions in terms of solutions to an optimization problem.)
Goals
- Derive the general formula for the maximum likelihood parameter estimate for exponential families: namely, that the model moments match the empirical moments.
Core resources (read/watch one of the following)
-Free-
→ Mathematical Monk: Machine Learning (2011)
Online videos on machine learning.
-Paid-
→ Machine Learning: a Probabilistic Perspective
A very comprehensive graudate-level machine learning textbook.
Location:
Section 9.2.4, "MLE for the exponential family," pages 286-287
→ Pattern Recognition and Machine Learning
A textbook for a graduate machine learning course, with a focus on Bayesian methods.
Location:
Section 2.4.1, "Maximum likelihood and sufficient statistics," pages 116-117
See also
- Estimating the parameters of a Markov random field (MRF) is a surprising example of maximum likelihood in exponential families.
- Bayesian parameter estimation often has a [convenient form](bayes_param_exp_fam) in exponential families.