Talk:Expectation-maximization algorithm
From Wikipedia, the free encyclopedia
Hello. I have a comment about the recent edit which credits the popularity of EM methods to the convergence proof instead of the ease of formulation. Maybe ease of formulation is not quite on the mark, but it seems that the convergence proof isn't either -- after all, the other methods (gradient ascent, etc) will also converge within some neighborhood of a stationary point, and that's all that's guaranteed for EM methods, right? -- FWIW, my point about ease of formulation was this: suppose you have a method that computes maximum likelihood estimates in the complete data case. When you consider the case of incomplete data, the previous method can often be adapted in a very straightforward way; the canonical example would be Gaussian mixture density estimation, in which computation of mean and variance are just replaced by a weighted mean and a weighted variance. I guess ultimately the reason for popularity is an empirical question; we would have to look at a lot of papers and see what reasons, if any, are given. Happy editing, Wile E. Heresiarch 15:37, 29 Apr 2004 (UTC)
- Given that it is hard to pin down people's motivations for liking or disliking an algorithm, perhaps we should drop the sentence from the article. But, based on what I heard at machine learning conferences, EM was preferred due to convergence. I believe that the paper [2] triggered the shift from gradient descent (used in neural networks) to EM. That paper doesn't have a nice simple M-step like mixture of Gaussians (see equations 27-29, and following), so the popularity wasn't based on simplicity, I think. -- hike395 07:34, 1 May 2004 (UTC)
-
- Well, I agree that it's hard to divine motivation. I dropped the sentence that referred to a reason for popularity. Wile E. Heresiarch 22:45, 1 May 2004 (UTC)
- You note that EM is only guaranteed to maximize likelihood locally. This point should be made clearly from the start of the article. And yes, other methods such as gradient ascent or conjugate gradient will work just as well. (In fact, EM is closely related to gradient ascent and can be regarded as a "self-scaling" version of gradient ascent (i.e., it picks its own stepsize). In many instances of EM, like the forward-backward and inside-outside algorithms, the E step is actually computing the gradient of the log probability.) 70.16.28.121 01:48, 13 August 2006 (UTC)
[edit] Consistency of symbol font
At present, the symbols that are inline are sometimes in a bigger font than the text, sometimes the same font size (look at subscripted thetas for example). The difference seems to depend on whether there's a "/," in the text string between the math tags. They should at least be consistent - I thought maybe I'd just change them to one or the other, but perhaps it's worth asking first whether there's a preference amongst the original authors, or whether there's a general convention.
[edit] Animated picture
On 3 January 2007 someone has removed this animated image :
![Example of applying the EM algorithm to estimate the parameters of a 12-component Bernoulli Mixture Model for the MNIST database of handwritten digits [1].](../../../upload/thumb/2/29/Bernoulli-Mixture-Model.gif/450px-Bernoulli-Mixture-Model.gif)
I thought the image was quite nice, but maybe it shouldn't be at the top of the image as in [3] but more to the bottom. So I thought if someone (with a little bit more knowledge about this topic than me) can write a little piece about the application in question we could move it to the "Examples" section. Skaakt 09:56, 9 February 2007 (UTC)
