Statistical Inference I ( Theory of estimation : Efficiency)

🔖Statistical Inference I ( Theory of estimation : Efficiency) 

In this article we see the  terms: 

I. Efficiency.

II. Mean Square Error.

III. Consistency.

📚Efficiency: 

We know that  two unbiased estimator of parameter gives rise to infinitely many unbiased estimators of parameter. there if one of parameter have two estimators then the problem is to choose one of the best estimator among the class of unbiased estimators. in that case we need to some other criteria to to find out best estimator. therefore, that situation  we check the variability of that estimator, the measure of variability of estimator T around it mean is Var(T). hence If T is an Unbiased estimator of parameter then it's variance gives good precision. the variance is smaller then it give's greater precision.

📑 i. Efficient estimator: An estimator T is said to be an Efficient Estimator of 𝚹, if T is unbiased estimator of  𝛉. and it's variance is less than any other estimator. i.e. Var(T) < Var(T*).

where T* is any other estimator of  𝛉.

🔖ii. Relative Efficiency: If T1 and T2 are two unbiased estimators of parameter  𝛉 and E(T₁²) < ∞ and  E(T₂²) < ∞ then relative efficiency of T1  with respect to  T2 is denoted by Efficiency( T1 ,T2 ).

and T1  )/ V(T2 )

Remark: i. if relative efficiency =1 it means that  T1 and T2 are equal efficient estimators.

ii. If Relative Efficiency ( T1 and T2 ) < 1, it means that V( T1 ) <  V( T2 ). 

i.e. T1 is more efficient than T2 .

iii. If Relative Efficiency ( T1  and T2 ) > 1, it means that V( T1 ) >  V( T2 ). 

i.e. T2 is more efficient than  T1 

Example: 1. if X1 and X2 are two independent observation from normal distribution with mean 𝚹 and variance  𝛔² , then show thatT1 = ( X1 + X2)/2 , T2 = ( X1 + 2X2 )/3 are unbiased estimators of 𝚹 and also find the relative Efficiency (  T1 ,T2 ).

Solution: if  X1  and X2 are two independent observation from normal distribution with mean 𝚹 and variance  𝛔² 

E( X1 ) = E( X2 ) = 𝚹.

we have T1 = (X1 + X2)/2 , T2 = (X1 + 2X2 )/3

E(T1 ) = E[(X1 + X2 )/2] , E(T2 )= E [(X1 + 2X2 )/3].  we have E(X1 ) = E(X2) = 𝚹.

E(T1 ) = E[(𝚹+𝚹)/2] , E(T2 )= E [(𝚹 + 2𝚹 )/3

E(T1 ) = 2𝚹/2 , E(T2 )=  3𝚹 /3

E(T1 ) = 𝚹 , E(T2 ) =  𝚹 

Thus  T1 &T2 are unbiased estimators of parameter 𝚹.

now we obtain Variation of  T1 &T2 

V(T1 ) =V[ (X1 + X2 )/2] , V(T2)= (X1 + 2X2 )/3

V(T1 ) =[V(X1 )+V( X2 )] /4 , V(T2 )=[ V(X1 )+ 4V(X2 )] /3

V(T1 ) =[V(X1 )+V( X2 )] /4 , V(T2)=[ V(X1 )+ 4V(X2 )] /9

V(T1 ) =[𝛔² +𝛔² ] /4 , V(T2)=[ 𝛔² + 4𝛔² ] /9

V(T1 ) =2𝛔² /4 , V(T2)=5𝛔² /9

V(T1 ) =𝛔² /2 , V(T2)=5𝛔² /9

relative Efficiency  T1 relative to T2 is 

E (  T1 ,T2) =  V(  T1 )/V(T2)

                    = (𝛔² /2 ) /(5𝛔² /9)

                    = 9/10

                    = 0.9 < 1

therefore  E (  T1 ,T2) < 1 , means V( T1 ) =𝛔² /2  < V(T2)=5𝛔² /9 

Hence T1 is more efficient than T2 

 

Example 2. if X1, X2, ........Xn. random sample form normal distribution with mean 𝚹 and variance  𝛔² ,    T1 = sample mean i.e. x̄, and T2= sample median. find the  Efficiency of  T1 relative to T2 .   given that variance of median is  = V( T2)= (𝝅 𝛔² )/2n.    (for large n)

solution:  X1, X2, ........Xn. random sample form normal distribution with mean 𝚹 and variance  𝛔² then we have V(x̄) = V( T1 ) = 𝛔² /n. and  V( T2 )= (𝝅 𝛔² )/2n.

therefore the  Efficiency of T1 relative to T2  = E (  T1 ,T2 ) =  V( T1 )/V(T2 )

E ( T1 ,T2 ) =  V( T1 )/V(T2) 

                       =[ 𝛔² /n] / [ (𝝅 𝛔² )/2n]

                        = [𝛔² /n] x[2n/((𝝅 𝛔² )]

                        = (2/𝝅 )

                        =  0.6369 < 1 

                  therefore  E (   T1 ,T2 ) < 1 , means V( T1  ) =𝛔² /n <  V( T2  )= 𝝅𝛔² /2n

Hence  T1  is more efficient than  T2  

i.e. sample mean is more efficient than sample median.

III. Mean Square Error (M. S. E.)

let   T1  &  T2   be the  two estimators of parameter 𝚹, where  T1   is a unbiased estimator of 𝚹 &   T2  is a biased estimator of 𝚹. suppose the possible value of    T1  is spread   around  𝚹 and the value of  T2  is near to the parameter.  therefore in this case  T2   may be preferred than that of  T1  .

that case we need to study the variability of estimator around the parameter. for this we have to find out the variance of the estimator, which measure the variability of estimator around it's mean. if  T1  is an unbiased estimator of it's variance gives a good measure of precision. but if   T2   is biased estimator of parameter then variability of  T around 𝚹. as measure of it's precision and it is called as Mean Square Error (M. S. E.).

 Mean Square Error : an estimator T is a said to be a good estimator of 𝚹 if it's mean square error is minimum.

 i.e. E(T-𝚹)² ≤ E(T*-𝚹)²

where 𝚹. = T and T* is any other estimator .

this criteria known as mean square error.

Result: show that M. S. E. of T = MSE(T) = Var(T) + b² (T,𝚹) where b(.) is biased estimator.

Proof: 

 by definition of M. S. E. (T) = E(T-𝚹.)²

                                                = E[T-E(T)+E(T)-𝚹]²

   = E{T-E(T)}²+E{E(T)-𝚹}² + E{[T-E(T)]x[E(T)-𝚹]}

                                          = E{T-E(T)}²+E{E(T)-𝚹}² 

                                                 = Var(T) + b² (T,𝚹)

{Var(T)  = E{T-E(T)}²,   b² (T,𝚹) =E{E(T)-𝚹}² }

therefore  M. S. E. of T = MSE(T) = Var(T) + b² (T,𝚹).

Definition: Mean Square Error (M. S. E.) :

The M.S.E.  of estimator T of parameter 𝚹  is defined as 

    M. S. E. of T = MSE(T) = Var(T) + b² (T,𝚹)     if  T is biased estimator of  𝚹.

    M. S. E. of T = MSE(T) = Var(T)            

          if an T unbiased estimator of 𝚹.

this is the definition of M.S.E.

Example: 1. if   T1 & T2 be the two unbiased estimators of parameter 𝚹 with variance 𝞂1² and 𝞂2² respectively and having correlation ρ then find the best liner unbiased combination of T1 & T2  and also find the expression of variance of such combination.

Solution: we have two estimators  T1 & T2 are unbiased estimators of parameter 𝚹 with variance 𝞂1² and  𝞂2².

 E(T1 ) = 𝚹 , E(T2 ) =  𝚹 

V(T1 ) =  𝞂1² , V(T2)=  𝞂2².

and correlation is ρ

consider a linear combination of estimators   T1 & T2 is T = ɑ T1 + (1-ɑ) T2

now taking expectation E(T)

E(T)  = E[ɑ T1 + (1-ɑ) T2]

           =ɑ E[T1 ]+ (1-ɑ)E[ T2]

            = ɑ 𝚹 + (1-ɑ) 𝚹 

                =  ɑ 𝚹 + 1𝚹-ɑ𝚹 

            = 𝚹

E(T) = 𝚹

therefore T is an unbiased estimator of 𝚹

now we find variance of T

Var(T) =  V[ɑ T1 + (1-ɑ) T2]

   =ɑ²V( T1 ) + (1-ɑ)² V(T2) +2ɑ(1-ɑ)COV( T1, T2 )

  V(T)= ɑ² 𝞂1²+ (1-ɑ)²  𝞂2² +2ρ 𝞂1𝞂2 ɑ(1-ɑ)

           we find the value of ɑ and 1-ɑ the variance of (T)is minimise.

for that we find [dV(T) / da] = 0 and  [d²V(T) /da²] > 0.

now,  [dV(T) / da] = 0  i.e. differentiate w.r.t. a

 dV(T) / da  = {d/da} [  ɑ² 𝞂1²+ (1-ɑ)²  𝞂2² +2ɑ(1-ɑ)ρ 𝞂1 𝞂2]

=> 2ɑ 𝞂1² -2 (1-ɑ)  𝞂2² +2(1-2ɑ)ρ 𝞂1 𝞂2  = 0

              ɑ 𝞂1² - (1-ɑ)  𝞂2² + (1-2ɑ)ρ 𝞂1 𝞂2 = 0 

      ɑ 𝞂1² -  𝞂2² +ɑ 𝞂2² + ρ 𝞂1 𝞂2 -2ɑ ρ 𝞂1 𝞂2  = 0

          ɑ( 𝞂1²+𝞂2²-2  ρ 𝞂1 𝞂2 ) -  𝞂2² + ρ 𝞂1 𝞂2 = 0

                         ɑ( 𝞂1²+𝞂2²-2  ρ 𝞂1 𝞂2 ) =  𝞂2² - ρ 𝞂1 𝞂2 

    ɑ =  ( 𝞂2² - ρ 𝞂1 𝞂2 ) / ( 𝞂1²+𝞂2²-2ρ 𝞂1 𝞂2 )       

therefore    

 1-  ɑ= ( 𝞂1² - ρ 𝞂1 𝞂2 )/ ( 𝞂1²+𝞂2²-2 ρ 𝞂1 𝞂2 ) 

and 

d²V(T) /da² = 

{d/da} [ 2ɑ 𝞂1² -2 (1-ɑ)  𝞂2² +2(1-2ɑ)ρ 𝞂1 𝞂2 ]

                    = 2𝞂1²+2𝞂2²-4  ρ 𝞂1 𝞂2

                      =    2( 𝞂1²+𝞂2²-2 ρ 𝞂1 𝞂2 )

                    =2[V( T1 ) +V(T2) -2COV( T1, T2 )]

                    = 2V( T1 - T2 ) > 0

hence variance of (T) is minimum at the value of ɑ =  ( 𝞂2² - ρ 𝞂1 𝞂2 ) / ( 𝞂1²+𝞂2²-2ρ 𝞂1 𝞂2 )       

  1-  ɑ=( 𝞂1² - ρ 𝞂1 𝞂2 )/( 𝞂1²+𝞂2²-2 ρ 𝞂1 𝞂2 )  

for variance of T we put the values of  ɑ =  ( 𝞂2² - ρ 𝞂1 𝞂2 ) / ( 𝞂1²+𝞂2²-2ρ 𝞂1 𝞂2 )       

     1-  ɑ =  ( 𝞂1² - ρ 𝞂1 𝞂2 ) / ( 𝞂1²+𝞂2²-2 ρ 𝞂1 𝞂2 ).

now find the variance of T 

Var(T) =  V[ɑ T1 + (1-ɑ) T2]

   =ɑ²V( T1 ) + (1-ɑ)² V(T2) +2ɑ(1-ɑ)COV( T1, T2 )

 V(T)   = ɑ² 𝞂1²+ (1-ɑ)²  𝞂2² +2ɑ(1-ɑ) ρ 𝞂1 𝞂2.

Note that: Smaller M.S.E. i.e. variance is small means greater precision. hence while comparing two estimators T1 & T2 of  𝞗  we choose an estimator with smaller M.S.E. this used to modifies the formula of efficiency

Efficiency of T1 & T2 = M.S.E(T1 ) / M.S.E(T2 )

If efficiency (T1 , T2 ) < 1

then T1  is more efficient estimator than  T2 

i.e.  M.S.E(T1 )  <  M.S.E(T2 )

Locally Minimum Variance Unbiased Estimator (MVUE).

Definition: Let 𝞗0 ∈ 𝚯 and U(𝞗0) be the class of all unbiased estimates of θ such that     E(T²) <∞. AndT0∈ U(𝞗0) then T0 is called locally minimum variance unbiased estimator(LMVUE).

OR simply minimum variance unbiased estimator( SMVUE). IF E(T0-θ0)² ≤E(T -θ0)² holds for all T.

Uniformly Minimum Variance Unbiased Estimator (UMVUE)

Definition: Let U be the set of all unbiased estimates of T     of 𝞗 ∈ 𝚯, such that  E(T²) <∞. and estimate T0 ∈ U is called uniformly minimum variance unbiased estimator of 𝞗 if 

E(T0-θ)² ≤E(T -θ)² for all 𝞗 ∈ 𝚯 and every T Ɛ U.

Now we see the Definition of Minimum Variance Unbiased Estimator (MVUE)

Definition: if a statistics T=T(X1, X2, .....Xn) based on a Random sample of size n such that i) T is unbiased estimator for Ɣ(θ) for all 𝞗 ∈ 𝚯  ii) it has smallest variance among the class of all unbiased estimators of  Ɣ(θ), then T  is called  Minimum Variance Unbiased Estimator (MVUE)

OR

T is MVUE of  Ɣ(θ) if 

i)E(T) = Ɣ(θ)  for all 𝞗 ∈ 𝚯

ii) V(T) ≤ V(T*)

where T* is any other unbiased estimator of   Ɣ(θ).

Result: If UMVUE exist then it is unique.

Proof : Suppose that  T1 & T2 are two uniformly minimum variance unbiased estimator of parameter θ then 

we have     E(T1) =θ,   E(T2) =θ   for all θ ∈ 𝚯

and V(T1) =  V(T2)    for all θ ∈ 𝚯         ............1

now we define new estimator T =(T1 + T2 ) /2  which is unbiased estimator 

since E(T ) = E[(T1 + T2 ) /2]

                    =  (θ+θ)/2

                     =2θ /2

                    =θ

And variance of T 

Var(T) = Var[(T1 + T2 ) /2]

            = {V(T1 ) +V( T2 ) + 2 Cov( T1 , T2)}/4

            = {2V(T1 ) + 2 ⍴ √(V(T1 ) xV( T2 ))} / 4

            = { V(T1 ) +  ⍴ V(T1 ) } / 2

            = { V(T1 )  (1+ ⍴)} / 2          ...............2

where   ⍴  is the Karl Pearson coefficient of correlation between T1 & T2 .

Since T1 is minimum variance unbiased estimator of  θ.

then 

 V(T1 ) ≤ V( T ) 

i.e.   V(T1 ) ≤ { V(T1 )  (1+ ⍴)} / 2   

      V(T1 ) ≤ { V(T1 )  (1+ ⍴)} / 2   

 1≤  (1+ ⍴) / 2   

 2 ≤  (1+ ⍴)   

 1≤  ⍴ 

we have |⍴ | ≤  1 therefore we must have ⍴  = 1.

i.e.   T1 & T2 have linear relationship of the form 

 T1  = ɑ + 𝜷T2.........3

Where  ɑ and  𝜷 are constant independent on parameterθ

E( T1 ) = E(ɑ + 𝜷T2)

E( T1 ) = E(ɑ )+ E(𝜷T2)

θ   = ɑ + 𝜷θ. ........4

now variance 

 V(T1  )=V( ɑ + 𝜷T2)

   V(T1  )=V( 𝜷T2)  

  V(T1  )=𝜷²V(T2)

therefore   𝜷² = 1

𝜷=+-1

𝜷 =1 ................5

because ⍴  = 1. T1 & T2 are positive 

from equation 4 and 5 we get

θ   = ɑ + θ

 ɑ = 0

putting 𝜷 =1,ɑ = 0 in equation 3

 T1  = 0 + 1xT2

     T1  = T2 

Thus if UMVUE exist, it is unique.

III. Consistency: