On the Geometry of Loss Functions

This post exists to show what the machinery can do: prose, code, mathematics and images, mixed freely. Delete it once you have written something real.

The problem

Consider a function $f: \mathbb{R}^n \to \mathbb{R}$ that we want to minimise. The key question is: when does gradient descent converge, and how fast?

$$\nabla f(x^*) = 0 \quad \text{and} \quad \nabla^2 f(x^*) \succ 0$$

If the Hessian $\nabla^2 f(x)$ is $L$-Lipschitz continuous and $f$ is $\mu$-strongly convex, then gradient descent with step size $\eta = \tfrac{1}{L}$ converges linearly:

$$f(x_{k+1}) - f(x^*) \leq \left(1 - \frac{\mu}{L}\right)^k \left(f(x_0) - f(x^*)\right)$$

The condition number $\kappa = L/\mu$ determines everything. A well-conditioned problem converges fast; an ill-conditioned one crawls.

Implementation

import numpy as np


def gradient_descent(f, grad_f, x0, lr=0.01, tol=1e-6, max_iter=1000):
    """Minimize f using gradient descent."""
    x = x0.copy()
    history = [x.copy()]

    for _ in range(max_iter):
        g = grad_f(x)

        # Check convergence
        if np.linalg.norm(g) < tol:
            break

        # Update step
        x = x - lr * g
        history.append(x.copy())

    return x, history

And the usage is straightforward:

# Define a simple quadratic
def f(x):
    return 0.5 * x @ Q @ x - b @ x


def grad_f(x):
    return Q @ x - b


Q = np.array([[4, 1], [1, 2]])
b = np.array([1, 1])
x0 = np.array([0.0, 0.0])

x_star, hist = gradient_descent(f, grad_f, x0, lr=0.2)
print(f"Converged to {x_star} in {len(hist)} steps")

Key takeaways

• Convexity guarantees a global minimum — no saddle points to worry about.
• The convergence rate depends entirely on the condition number $\kappa = L/\mu$.
• Step size matters: too large diverges, too small crawls.
• Preconditioning (essentially rescaling the problem) can dramatically improve convergence.

---

The maths here connects directly to machine learning: every time you train a neural network with SGD, you are doing a stochastic version of exactly this. The loss landscape is not convex, but the local geometry still governs whether you converge or oscillate. Understanding the convex case deeply is what lets you reason about the non-convex one.