This roadmap gives the background for my UAI 2012 paper, "Exploiting compositionality to explore a large space of model structures." This covers the basic machine learning concepts that the paper depends on, and should be sufficient for understanding it at a conceptual level.
Section 3: A grammar for matrix decompositions
- We organize the models into a context-free grammar.
- Bayesian versions of the following models are used as production rules in the grammar:
- low-rank approximation, a category which includes principal component analysis (PCA). The production rule corresponds more precisely to probabilistic matrix factorization.
- clustering, for instance mixture of Gaussians
- binary latent feature models (TODO)
- sparse coding
- linear dynamical systems
Section 4: Posterior inference of component matrices
- The following sampling algorithms are used to perform inference in the models corresponding to the production rules:
- For the binary and clustering production rules, we can infer the latent dimensionality by fitting Bayesian nonparametric models:
- For the low rank production rule, we infer the latent dimensionality using reversible jump MCMC.
Section 5: Scoring candidate structures
- In this section, we manipulate multivariate Gaussian distributions. See computations on multivariate Gaussians.
- Certain models require using approximate inference algorithms to evaluate the predictive likelihood:
- variational inference, in particular the mean field approximation
- annealed importance sampling