Method of Moment & Maximum Likelihood Estimator: Method, Properties and Examples.

 Statistical Inference I:

Method Of Moment:  

One of the oldest method of finding estimator is Method of Moment, it was discovered by Karl Pearson in 1884. 

Let X1, X2, ........Xn be a random sample from a population with probability density function (pdf) f(x, θ) or probability mass function (pmf) p(x) with parameters θ1, θ2,……..θk.

If μ_r^' (r-th raw moment about the origin) then μ_r^' = ∫_-∞^∞ x^r f(x,θ) dx for r=1,2,3,….k .........Equation i

In general, μ₁^', μ₂^',…..μ_k^' will be functions of parameters θ₁, θ₂,……..θ_k.

Let X₁, X₂,……X_n be the random sample of size n from the population. The method of moments consists of solving "k" equations (in Equation i) for θ₁, θ₂,……..θ_k to obtain estimators for the parameters by equating μ₁^', μ₂^',…..μ_k^' with the corresponding sample moments m₁^', m₂^',…..m_k^'.

Where m_r^' = sample moment = (∑x_i^r) / n

We get μ₁^' = m₁^', μ₂^' = m₂^', μ₃^' = m₃^', and so on μ_k^' = m_k^'.

These k equations give (θ̂₁, θ̂₂,……..θ̂_k) k estimators for θ₁, θ₂,……..θ_k by the method of moments.

Procedure to finding Moment estimator:

1. First, equate the first sample moment about the origin m₁^' to the first theoretical moment, i.e., μ₁^' = m₁^'.
2. Then, equate the second sample moment about the origin m₂^' to the second theoretical moment, i.e., μ₂^' = m₂^'.
3. Continue equating the sample moments about the origin to the theoretical moments until you have as many parameters as equations (i.e., if there are 2 parameters, then equate 2 equations; if there are 4 parameters, then equate 4 equations).
4. The equations are solved for the parameters. The resultant values are called method of moment estimators.

Example for Single Parameter:

Let X₁, X₂,……X_n be a random sample from a Bernoulli distribution with parameter p, then find an estimator for parameter p by the method of moments.

Answer:

Let X₁, X₂,……X_n be a random sample from a Bernoulli distribution with parameter p.

Then P(X=x) = p^x q^1-x for x=0,1; 0 ≤ p ≤ 1, q=1-p

= 0; otherwise.

For the Bernoulli distribution, μ₁^' = ∑ X_r·P(X=x)

μ₁^' = p ........Equation i

Now, m₁^' = sample moment = (∑x_i¹) / n = X̄ .......Equation ii

Equating sample moment to theoretical moment, i.e., equating Equation i and ii:

μ₁^' = m₁^'

p = X̄

i.e., p̂ = X̄

Therefore, p̂ = X̄ is the moment estimator for parameter p.

Moment Estimator for μ and σ²

Let x₁, x₂,……x_n be a random sample from a Normal Distribution with parameters N(μ, σ²).

Answer: We have a random sample of size n from a normal distribution, and its probability density function (pdf) is:

f(x, θ) = ¹⁄_(σ√(2π)) * e^{-1 / (2σ²) * (x - μ)²}, -∞ ≤ x, μ < ∞; σ² > 0

We have E(x) = μ and V(x) = σ².

Then we find μ₁^' = m₁^' and μ₂^' = m₂^'.

Two equations are:

μ₁^' = E(x) = μ (Equation 1)

m₁^' = (∑ x_i) / n = x̄ (Equation 2)

Equating Equation 1 and Equation 2:

μ₁^' = m₁^'

μ̂ = x̄

Therefore, x̄ is the method of moment estimator for parameter μ.

Now, μ₂^' = E(x²) = V(X) + (E(X))²

μ₂^' = σ² + μ² (Equation 3)

m₂^' = (∑ x_i²) / n (Equation 4)

Equating Equation 3 and Equation 4:

μ₂^' = m₂^'

σ² + μ² = (∑ x_i²) / n

We find the estimator for μ, which is μ̂ = x̄, and then:

σ² + x̄² = (∑ x_i²) / n

σ̂² = (∑ x_i²) / n - x̄² (Sample Variance)

Sample Variance is the method of moment estimator for σ².

So, x̄ is the method of moment estimator for parameter μ, and Sample Variance is the method of moment estimator for σ².

Maximum Likelihood Estimation (MLE)

The method of Maximum Likelihood Estimator (MLE) was introduced by Prof. R. A. Fisher in 1912.

Likelihood Function:

If we have a random sample of size n from a population with density function f(x, θ) where θ ∈ Θ, then the likelihood function of the sample values x₁, x₂, ..., xₙ is given by:

L(θ) = ∏ᵢ₌₁ⁿ f(xᵢ, θ)

Definition:

If x₁, x₂, …, xₙ is a random sample from any probability density function (PDF) or probability mass function (PMF), then The value of θ for which the likelihood function is maximized is called the Maximum Likelihood Estimator (MLE) of θ.

Maximum Likelihood Estimation Procedure:

The MLE is obtained by solving dL/dθ = 0 and subject to the condition that d²L/dθ² < 0. The same result can be obtained by taking the logarithm of the likelihood function and solving d(log(L(θ))/dθ = 0, subject to the condition d²(log(L(θ))/dθ² < 0

Properties:

• Maximum Likelihood estimator may not be unique.
• If f(X, θ) is a Probability Density Function (PDF) belonging to the exponential family, then the MLE of θ is a function of 1/n ∑T(x).
• If T is the Minimum Variance Unbiased Estimator (MVBUE) of θ, then it is also the MLE of θ.
• If T is the MLE of θ, then any function φ(T) is the MLE of φ(θ).
• MLE is a function of Sufficient Statistics.
• Asymptotic Normality of MLE: A consistent solution of the likelihood equation is asymptotically normally distributed about the true value of θ.

Remark:

MLEs are always consistent estimators but may not be unbiased.