An overview of the concept “Jensen Inequality”. In this work, we discuss two important concepts to theoretical machine learning, often used in proofs/derivation of energy functions, convergence, etc.

Jensen Inequality

If is a convex function, then

Similarly, if is a concave function, then:

The difference between the two sides of the inequality, \mathbb{E}[f(x)] - f(\mathbb{E}[x]), is called Jensen gap.