Gradient, Hessian, and Convexity

07/30/2025

Definition: The vector of partial derivatives of a scalar-valued function.

Use: Points in the direction of steepest ascent of the function

Formula:

For $f : ℝ^{n} \to ℝ$ ,

\nabla f (x) = [\frac{\partial f}{\partial x_{1}}, \frac{\partial f}{\partial x_{2}}, \dots, \frac{\partial f}{\partial x_{n}}]^{T}

Definition: The square matrix of second-order partial derivatives.

Use: Describes the curvature of a function; important for Newton's method.

Formula:

H_{f} (x) = [\begin{matrix} \frac{\partial^{2} f}{\partial x_{1}^{2}} & \dots & \frac{\partial^{2} f}{\partial x_{1} \partial x_{n}} \\ ⋮ & ⋱ & ⋮ \\ \frac{\partial^{2} f}{\partial x_{n} \partial x_{1}} & \dots & \frac{\partial^{2} f}{\partial x_{n}^{2}} \end{matrix}]

Definition: A function $f$ is convex if the line segment between any two points on its graph lies above the graph.

Mathematical Condition:

f (y) \geq f (x) + \nabla f (x)^{T} (y - x), \forall x, y

Second-order:

if $f$ is twice-differentiable, $f$ is convex iff its Hessian is positive semidefinite:

H_{f} (x) ⪰ 0, for all x

Strict Convexity: if $H_{f} (x) ≻ 0 for all x$ , then $f$ is strictly convex.

Note

A matrix $A \in ℝ^{n \times n}$ is positive semidefinite (PSD) if:

v^{T} A v \geq 0 for all v \in ℝ^{n}