Natural Gradient Descent

Fisher Information Matrix (FIM)

Fisher Infomation measures the quantity of information from the random variable $X$, while $X$ follows some model parametrized by the parameter $\theta$; thus $X \sim p(x | \theta)$.

In the point of view of frequentist, our goal is to maximize probability of $x$ by finding the optimal $\theta$ which maximized $p(x

\theta)$. To do so, we are training a model on $\theta$. To evaluate this model, the score function $s(\theta)$ is set as

\[s(\theta) = \nabla_\theta \log p(x | \theta)\]

which equals the gradient of log-likelihood. One notable characteristic of this function is that it’s expectation with respect to our model is zero:

\[\begin{align} \mathbf{E}_{p(x | \theta)} \left[ s(\theta) \right] &= \mathbf{E}_{p(x | \theta)} \left[ \nabla_\theta \log p(x | \theta) \right] \\ &= \int \nabla \log p(x | \theta) p(x | \theta) dx \\ &= \int \frac{\nabla p(x | \theta)}{p(x | \theta)} p(x | \theta) dx \\ &= \int \nabla p(x | \theta) dx \\ &= \nabla \int p(x | \theta) dx \\ &= \nabla 1 = 0 \end{align}\]

How much can we rely on the estimation? To answer this question, the scale of uncertainty created by estimation should be defined. Think of a covariance matrix of our score function:

\[\begin{align} & \mathbf{E}_{p(x | \theta)} \left[ \left(s(\theta) - \mathbf{E}_{p(x | \theta)}\left[ s(\theta) \right] \right) \left(s(\theta) - \mathbf{E}_{p(x | \theta)}\left[ s(\theta) \right] \right)^T \right] \\ &= \mathbf{E}_{p(x | \theta)} \left[ \left(s(\theta) - 0 \right) \left(s(\theta) -0 \right)^T \right] \\ &= \mathbf{E}_{p(x | \theta)} \left[ s(\theta) s(\theta)^T \right] \end{align}\]

By using the log-likehood’s gradient as a score function (as we’ve defined above), we get a Fisher Information Matrix (FIM): $\mathbf{F} = \mathbf{E}_{p(x | \theta)} \left[ \nabla_\theta \log p(x | \theta) \nabla_\theta \log p(x | \theta)^T \right]$

Calculating this expectation is (in most of the cases) very difficult. We often approximate $F$ by the means obtained from the sample:

\[\frac{1}{N} \sum_{1}^{N} \nabla_\theta \log p(x | \theta) \nabla_\theta \log p(x | \theta)^T\]

Hessian of the probability space

Alternatively, we can understand the uncertainty and shape of our probability space by examining its curvature, which is captured by the second-order derivatives of the log-likelihood function, known as the Hessian matrix.

Using $s(\theta) = \log p(x

\theta)$,

\[\begin{align} H_{\log p(x | \theta)} &= J \left[ \frac{\nabla p(x | \theta)}{p(x | \theta)} \right] \\ &= \frac{H_{p(x | \theta)}p(x | \theta) - \nabla p(x | \theta) \nabla p(x | \theta)^T }{p(x | \theta) p(x | \theta)} \\ &= \frac{H_{p(x | \theta)} p(x | \theta) }{p(x | \theta) p(x | \theta)} - \frac{ \nabla p(x | \theta) \nabla p(x | \theta)^T }{p(x | \theta) p(x | \theta)} \\ &= \frac{H_{p(x | \theta)} }{ p(x | \theta)} - \left( \frac{ \nabla p(x | \theta) }{p(x | \theta)} \right) \left( \frac{ \nabla p(x | \theta) }{p(x | \theta)} \right)^T \end{align}\]

The expectation of $H_{\log p(x

\theta)}$ is:

\[\begin{align} & \mathbf{E}_{p(x \mid \theta)} \left[ H_{\log p(x | \theta)} \right] \\ &= \mathbf{E}_{p(x \mid \theta)} \left[ \frac{H_{p(x | \theta)} }{ p(x | \theta)} - \left( \frac{ \nabla p(x | \theta) }{p(x | \theta)} \right) \left( \frac{ \nabla p(x | \theta) }{p(x | \theta)} \right)^T \right] \\ &= \mathbf{E}_{p(x \mid \theta)} \left[ \frac{H_{p(x | \theta)} }{ p(x | \theta)} \right] - \mathbf{E}_{p(x \mid \theta)} \left[ \left( \frac{ \nabla p(x | \theta) }{p(x | \theta)} \right) \left( \frac{ \nabla p(x | \theta) }{p(x | \theta)} \right)^T \right] \\ &= \int \frac{H_{p(x | \theta)}}{p(x | \theta)} p(x | \theta) dx - \mathbf{E}_{p(x \mid \theta)} \left[ \nabla \log p(x | \theta) \nabla \log p(x | \theta)^T \right] \\ &= H_{\int p(x | \theta)dx} - \mathbf{F} \\ &= H_{1} - \mathbf{F} \\ &= -\mathbf{F} \end{align}\]

Geometrically, this means that the Fisher Information Matrix (FIM) measures the curvature of the model distribution’s log-likelihood. By defining the FIM as the covariance of the score function and proving its equivalence to the expected negative Hessian, we can interpret it as a curvature matrix, making it a powerful tool for second-order optimization.

One remarkable property of the FIM is its connection to KL-divergence; a metric of measuring the distance between two probability distributions. In addition, this insight is what made the Natural Gradient Descent possible.