Expectation Maximization Algorithm

Maximum-likelihood estimation problem

We have a density function p(x|Θ)$p\left(x|\Theta \right)$ that is governed by the set of parameters Θ$\Theta$ (e.g., p$p$ might be a set Gaussians and Θ$\Theta$ could be the means and covariances). We also have a data set of size N$N$, That is , we assume that these data vectors are independent and identically distributed (i.i.d) with distribution p$p$. Therefore, the resulting density for the samples is p(X|Θ)=∑Ni=1p(xi|Θ)=L(Θ|X)$p\left(X|\Theta \right)={\sum }_{i=1}^{N}p\left(x_i|\Theta \right)=L\left(\Theta |X\right)$

This function L(Θ|X)$L\left(\Theta |X\right)$ is called the liklihood of the parameters given the data, or just the likelihood function. The likelihood is thought of as a function of the parameters Θ$\Theta$ where the data X$X$ is fixed. In the maximum likelihood problem, our goal is to find the Θ$\Theta$ that maximizes L$L$. That is, we wish to find Θ‘$\Theta ‘$ where Θ‘=argmaxΘL(Θ|X)$\Theta ‘=argma{x}_{\Theta }L\left(\Theta |X\right)$

Often we maximize log(L(Θ|X))$\log \left(L\right(\Theta |X\left)\right)$ instead because it is analytically easier.

Basic EM

There are two main applications of the EM algorithm. The first occurs when the data indeed has missing values, due to problems with or limitations of the observation process. The second occurs when optimizing the likelihood function is analytically intractable but when the likelihood function can be simplified by assuming the existence of and values for additional but missing (or hidden) parameters. The latter application is more common in the computational pattern recognition community.

As before, we assume that data X$X$ is observed and is generated by some distribution. We call X$X$ the incomplete data. We assume that a complete data set exists Z=(X,Y)$Z=(X,Y)$ and also assume a joint density funciton

p(z|Θ)=p(x,y|Θ)=p(y|x,Θ)p(x|Θ)$p\left(z|\Theta \right)=p(x,y|\Theta )=p(y|x,\Theta )p\left(x|\Theta \right)$

With this new density function, we can define a new likelihood function, L(Θ|Z)=L(Θ|X,Y)=p(X,Y|Θ)$L\left(\Theta |Z\right)=L(\Theta |X,Y)=p(X,Y|\Theta )$, called the complete-data likelihood. Note that this function is in fact a random variable since the missing information Y$Y$ is unknown, random and presumably governed by an underlying distribution. That is, we can think of L(Θ|X,Y)=hX,Θ(Y)$L(\Theta |X,Y)=h_{X,\Theta }\left(Y\right)$ for some function h$h$ where X$X$ and Θ$\Theta$ are constant and Y$Y$ is a random variable. The original likelihood L(Θ|X)$L\left(\Theta |X\right)$ is referred to as the incomplete-data likelihood funciton.

The EM algorithm first finds the expected value of the complete-data log-likelihood logp(X,Y|Θ)$\log p(X,Y|\Theta )$ with respect to the unknown data Y$Y$ given the observed data X$X$ and the current parameter estimates. We define:

Q(Θ,Θi−1)=E(logp(X,Y|Θ)|X,Θi−1)$Q(\Theta ,{\Theta }^{i-1})=E\left(\log p\right(X,Y|\Theta )|X,{\Theta }^{i-1})$

Where Θi−1${\Theta }^{i-1}$ are the current parameters estimates that we used to evaluate the expectation and Θ$\Theta$ are the new parameters that we optimize to increase Q.

The key thing to understand is that in this expression X$X$ and Θi−1${\Theta }^{i-1}$ are constants, Θ$\Theta$ is a normal variable that we wish to adjust, and Y$Y$ is a random variable governed by the distribution f(y|X,Θi−1)$f(y|X,{\Theta }^{i-1})$. The right side of Q$Q$ can be re-written as:

E[logp(X,Y|Θ)|X,Θi−1]=∫y∈Γlogp(X,y|Θ)f(y|X,Θi−1)dy$E\left[\log p\right(X,Y|\Theta )|X,{\Theta }^{i-1}]={\int }_{y\in \Gamma }\log p(X,y|\Theta )f(y|X,{\Theta }^{i-1})dy$

Note that f(y|X,Θi−1)$f(y|X,{\Theta }^{i-1})$ is the marginal distribution of the unobserved data and is dependent on both the observed data X$X$ and on the current parameters, and Γ$\Gamma$ is the space of values y$y$ can take on.

The evaluation of this expectation is called the E-step of the algorithm. Notice the meaning of the two arguments int function Q(Θ,Θ′) The first argument Θ corresponds to the parameters that ultimately will be optimized in an attempt to maximize the likelihood. The second argument Θ corresponds to the parameters that we use to evaluate the expectation.

The second step(M-step) of the EM algorithm is to maximize the expectation we computed in the first step. We find:

Θi=argmaxΘQ(Θ,Θi−1)

These 2 steps are repeated as necessary. Each iteration is guaranteed to increase the log-likelihood and the algorithm is guaranteed to converge to a local maximum of the likelihood function.