Stein and the Normal Distribution

- 1 min

Hello and happy 2021!

Inspired from this tweet, I wanted to understand the basics of Stein’s characterization of the Normal distribution.

With Stein’s idea, we can identify the distribution of a random variable by checking that it satisfies some condition in expectation. For example, for the standard normal, we have that $$X\sim N(0,1)$$ if and only if for all $$f$$ with $$E[f']<\infty$$ we have:

$\mathbb{E}[xf(x)-f'(x)]=0.$

The operator $$Af:=xf-f'$$ is then called the Stein operator and we can rewrite the result with this operator as: $$\mathbb{E}_{P}[Af]=0$$ for all $$f\in C^1_b$$ iff $$P$$ is $$N(0,1)$$.

This operator is not unique as we can always add P-measure zero parts; see $$Bf:=xf-f'+x$$ which satisfies

$\mathbb{E}[Bf]=\mathbb{E}[Af]+\mathbb{E}[x]=\mathbb{E}[Af].$

How can we show $$A$$ characterises the standard Normal? A key identity is that the probability density function (PDF) of the $$N(0,1)$$ satisfies $$P'+xP=0$$. So, if $$P$$ is indeed the PDF of the standard normal, then applying integration by parts to $$E_{P}[f']$$ and using the differential equation gives us Stein’s formula.

Now, if $$P$$ is the PDF of any other distribution and it satisfies Stein’s formula for all $$f\in C^1_b$$, then integration by parts on $$E_{P}[f']$$ leads us back to the $$P'+xP=0$$. That ODE is separable with solution $$P\propto \exp(-x^2/2)$$, which, up to the normalisation, is the PDF of the standard normal!

This strategy of deriving an ODE for the density function, getting its weak form by multiplying with a smooth function $$f$$ and integrating can be repeated to get Stein operators for other distributions, e.g., the exponential, etc.