Let \(\Theta = \mathbb{R}^p \) and \( \mathcal{S} = \mathbb{R}^n \), and let \(E: \Theta \times \mathcal{S} \to \mathbb{R} \) and \(C: \mathcal{S} \to \mathbb{R} \) be two scalar functions. Consider the problem of minimizing \begin{equation} {\mathcal L}(\theta) := C(s(\theta)) \end{equation} with respect to \(\theta \in \Theta\), subject to the constraint that \( s(\theta) \) is an equilibrium state, i.e. a stationary point of the energy: \begin{equation} \frac{\partial E}{\partial s}\big(\theta,s(\theta)\big) = 0. \end{equation} The equilibrium state need not be a minimum of \( E \): it can be any stationary point (minimum, maximum or saddle point). We assume that \( E \) and \( C \) are twice continuously differentiable, and that the equilibrium is non-degenerate, i.e. the Hessian \( \frac{\partial^2 E}{\partial s^2}\big(\theta,s(\theta)\big) \) is invertible. We wish to optimize \({\mathcal L}(\theta)\) by gradient descent. Equilibrium propagation (EP) achieves this by defining a family of equilibrium states \(s_\theta^\beta\) parameterized by \(\theta \in \Theta \) and a scalar \(\beta \in \mathbb{R} \), as stationary points of the total energy: \begin{equation} \frac{\partial E}{\partial s}(\theta,s_\theta^\beta) + \beta \, \frac{\partial C}{\partial s}(s_\theta^\beta) = 0. \end{equation} By the implicit function theorem (see the lemma), there exists a neighborhood of \( (\theta, 0) \) and a unique continuously differentiable mapping \( (\theta, \beta) \mapsto s_\theta^\beta \) defined on this neighborhood, satisfying \( s_\theta^0 = s(\theta) \) and the above stationarity condition.
The gradient of the loss is related to the partial derivatives of the function \(E\) by \begin{equation} \nabla_\theta \mathcal{L} (\theta) = \left. \frac{d}{d\beta} \right|_{\beta=0} \frac{\partial E}{\partial \theta} \left( \theta,s_\theta^\beta \right). \end{equation}
In this expression, \( \frac{\partial E}{\partial \theta}(\theta,s) \) denotes the partial derivative of the function \( E(\theta,s) \) with respect to its first argument, and \( \left. \frac{d}{d\beta} \right|_{\beta=0} \) denotes the (total) derivative with respect to \( \beta \) (through \( s_\theta^\beta \) ) at the point \( \beta=0 \).
See the proof of the EP formula. As a consequence of this identity, the EP algorithm performs (or approximates) gradient descent on the loss.