Stein and the Normal Distribution

Hello and happy 2021!

Inspired from this tweet, I wanted to understand the basics of Stein’s characterization of the Normal distribution.

With Stein’s idea, we can identify the distribution of a random variable by checking that it satisfies some condition in expectation. For example, for the standard normal, we have that $X\sim N(0,1)$ if and only if for all $f$ with $E[f^{\prime }]<\mathrm{\infty }$ we have:

The operator $Af:=xf-f^{\prime }$ is then called the Stein operator and we can rewrite the result with this operator as: ${\mathbb{E}}_P[Af]=0$ for all $f\in C_b^1$ iff $P$ is $N(0,1)$.

This operator is not unique as we can always add P-measure zero parts; see $Bf:=xf-f^{\prime }+x$ which satisfies

How can we show $A$ characterises the standard Normal? A key identity is that the probability density function (PDF) of the $N(0,1)$ satisfies $P^{\prime }+xP=0$. So, if $P$ is indeed the PDF of the standard normal, then applying integration by parts to $E_P[f^{\prime }]$ and using the differential equation gives us Stein’s formula.

Now, if $P$ is the PDF of any other distribution and it satisfies Stein’s formula for all $f\in C_b^1$, then integration by parts on $E_P[f^{\prime }]$ leads us back to the $P^{\prime }+xP=0$. That ODE is separable with solution $P\propto \exp (-x^2/2)$, which, up to the normalisation, is the PDF of the standard normal!

This strategy of deriving an ODE for the density function, getting its weak form by multiplying with a smooth function $f$ and integrating can be repeated to get Stein operators for other distributions, e.g., the exponential, etc.