Optimization is easier to revise when the concepts connect. Derivatives describe change, convexity helps certify minima, and smoothness helps us choose useful steps. Subgradients and proximal methods extend the picture to objectives with corners.
This guide revisits those foundations in the order they depend on one another, using small examples and derivations. It covers the core ideas behind unconstrained smooth and convex nonsmooth optimization, then connects them to Mixture of Experts routing.
Much of optimization begins with a simple instruction: take a derivative, set it to zero, and solve.
For a function like \(f(x)=x^2\), this works beautifully. The derivative is \(2x\), it vanishes at zero, and zero is the minimum.
But a few small changes expose the limits of that recipe. The function \(x^3\) also has a zero derivative at the origin, yet no minimum there. The function \(|x|\) has a minimum at the origin, but no derivative there at all.
These examples raise two different questions. What does a derivative actually tell us? And what additional structure lets us turn that information into an optimization method?
Following those questions leads from ordinary calculus to gradients, convexity, subgradients, and proximal methods. It also helps explain some of the difficulties that arise when neural networks make discrete routing decisions.
A derivative describes local change
For a function of one variable, the derivative measures how its output changes under a small change in its input:
For \(f(x)=x^2\),
At \(x=-2\), the slope is \(-4\): moving a little to the right decreases the function. At \(x=2\), the slope is \(4\): the same movement increases it.
At zero, the tangent is flat.
A flat tangent is useful evidence, but it does not identify a minimum on its own. For \(f(x)=x^3\), the derivative \(3x^2\) also vanishes at zero, even though the function continues increasing through that point.
The derivative describes first-order behavior near a point. To understand whether that point minimizes the function, we need more information.
From one slope to a gradient
A machine-learning objective usually depends on many parameters. Consider a small example:
We can measure change along each coordinate while holding the other fixed:
Collecting these partial derivatives gives the gradient:
At \((1,2)\),
Here, a small positive change in \(y\) affects the function more strongly than an equally sized positive change in \(x\).
More generally, the gradient describes change along any direction \(d\):
This is the directional derivative. Among directions of unit Euclidean length, it is largest in the direction of the gradient and smallest in the opposite direction, provided the gradient is nonzero.
That observation gives us gradient descent:
where \(\eta_t>0\) is the step size.
Indeed, choosing \(d=-\nabla f(x)\) gives
This is negative whenever the gradient is nonzero. A sufficiently small step in that direction therefore decreases the objective. How small the step needs to be will become important later.
Where the differential fits
The gradient also gives a linear approximation to a change in the function:
The corresponding linear map is the differential, often written as
For our quadratic,
Starting at \((1,2)\), let both coordinates increase by \(0.01\). The predicted change is
The actual change is \(0.1404\). The small discrepancy comes from the quadratic terms omitted by the linear approximation.
For vector-valued functions, these derivatives are organized into a Jacobian. Under the usual convention, the Jacobian of a scalar-valued function is the transpose of its gradient. This distinction matters in backpropagation, where intermediate computations often produce vectors.
Curvature explains how the gradient changes
The gradient tells us the local slope. The second derivative tells us how that slope changes.
For \(f(x)=x^2\),
In several variables, second derivatives form the Hessian matrix:
For \(f(x,y)=x^2+3y^2\),
The curvature along a direction \(d\) is described by
In this example, the Hessian’s eigenvalues are \(2\) and \(6\). Both are positive, so the function bends upward in every nonzero direction. It bends more sharply along the \(y\)-axis.
This is also the geometry that makes optimization easier: the surface has a single bowl-shaped structure.
Convexity turns local information into a global guarantee
A function on a convex domain is convex if
Geometrically, the line segment joining two points on its graph lies on or above the graph.
For a differentiable convex function, an equivalent characterization is
The tangent approximation is a lower bound on the entire function.
Now suppose \(\nabla f(x^\star)=0\). Substituting gives
So \(x^\star\) is a global minimizer.
This is the missing ingredient in the “set the derivative to zero” recipe. For a differentiable convex objective in unconstrained optimization, a zero gradient certifies global optimality.
There are useful distinctions within convexity:
- Convexity guarantees that every local minimum is global.
- Strict convexity guarantees that there is at most one minimizer.
- Strong convexity imposes a positive lower bound on curvature.
For a differentiable \(\mu\)-strongly convex function,
For twice continuously differentiable functions on an open convex domain, the corresponding Hessian conditions are
Our quadratic has smallest Hessian eigenvalue \(2\), so it is \(2\)-strongly convex under the Euclidean norm.
But convex functions need not be smooth bowls. They can have corners.
A minimum without a derivative
Consider
To the left of zero, its slope is \(-1\). To the right, its slope is \(1\). Since these one-sided derivatives disagree, \(f'(0)\) does not exist.
Yet zero is plainly the global minimizer.
We need an optimality condition that survives this corner. The supporting-line interpretation of convexity provides one.
A vector \(g\) is a subgradient of a convex function at \(x_0\) if
The affine function on the right touches \(f\) at \(x_0\) and stays below it everywhere.
At a smooth interior point, the gradient is the only such vector. At a corner, several supporting slopes may work.
Finding every supporting slope of absolute value at zero
Substituting \(f(x)=|x|\) and \(x_0=0\) gives
For \(x>0\),
For \(x<0\),
where the inequality reverses when we divide by a negative number.
Together,
Every slope in this interval supports the graph from below.
The set of all subgradients is called the subdifferential, denoted by \(\partial f(x)\). For absolute value,
A subgradient is one vector; the subdifferential is a set of vectors.
For a convex function differentiable at an interior point,
Boundary points require more care: a convex function can have an empty subdifferential at the boundary of its domain.
Zero still certifies a minimum
The familiar stationary-point condition now becomes
For a convex objective, this follows directly from the definition. If zero is a subgradient, then
Conversely, if \(x^\star\) is a global minimizer, this inequality holds, so zero is a subgradient.
For absolute value,
We have certified the minimum without needing a derivative there.
The supporting-hyperplane definition is central to convex analysis. General nonconvex, nonsmooth functions require other notions of generalized derivatives; the same conclusions should not be carried over automatically.
What happens when the objective takes a maximum?
Another common source of corners is a maximum of smooth functions.
Consider
For positive \(x\), the active branch is \(2x\). For negative \(x\), it is \(-x\). At zero, the branches tie.
Their slopes are \(2\) and \(-1\). At the tie, every convex combination of those slopes is a subgradient:
For a finite maximum of differentiable convex functions,
the general rule is
where
is the active set.
Only branches attaining the maximum contribute. When one branch is active, its gradient determines the subdifferential. When several tie, we take the convex hull of their gradients.
Subgradient steps behave differently
Subgradient descent replaces the gradient with a selected subgradient:
For \(f(x)=|x|\), starting at \(x_0=2\) with step size \(0.5\) gives
At zero, choosing \(g=0\) keeps the iterate at the minimum.
This example lands exactly at the optimum. A slightly different starting point shows why step sizes matter. Starting at \(0.2\) with the same fixed step size gives
The iterates oscillate, and the objective increases on every move from \(0.2\) to \(-0.3\).
A subgradient step need not decrease the function at each iteration. Convergence guarantees therefore depend on assumptions about the objective, the step-size schedule, and which iterate or average is reported.
To understand the stronger descent guarantees available for smooth functions, we need to distinguish two kinds of Lipschitz continuity.
Bounding the function versus bounding its gradient
A function is \(B\)-Lipschitz if
This bounds changes in function values.
Absolute value is \(1\)-Lipschitz because
It is still nondifferentiable at zero.
A differentiable function is \(L\)-smooth if its gradient is \(L\)-Lipschitz:
This bounds changes in slope.
For \(f(x)=x^2\),
so the function is \(2\)-smooth. However, it is not globally Lipschitz on \(\mathbb R\): its slope grows without bound.
These conditions control different quantities.
Why smoothness helps choose a step size
For an \(L\)-smooth function, the descent lemma gives
The quadratic term bounds the error in the first-order approximation.
Taking a gradient step with \(L>0\),
yields
Unless the gradient is zero, the objective decreases.
This descent statement does not require convexity. Convexity becomes important when we want to connect progress to a global minimum.
For twice continuously differentiable, convex functions on an open convex domain, smoothness corresponds to an upper curvature bound. Combining it with strong convexity gives
The lower bound prevents excessively flat directions; the upper bound limits how sharply the function can bend.
Proximal methods make use of nonsmooth structure
Many learning problems combine a smooth loss with a nonsmooth regularizer:
A simple example is
The squared-error term is smooth. The absolute-value penalty has a corner.
Proximal gradient descent treats these terms separately. First, take a gradient step on the smooth part:
Then apply the proximal operator of the regularizer:
where
This small optimization problem balances reducing the regularizer against staying close to the gradient-step result.
For \(h(w)=\lambda|w|\), its optimality condition is
Solving separately for positive, negative, and zero \(w\) gives soft thresholding:
Equivalently,
Small values become exactly zero. This is the mechanism behind the sparsity encouraged by \(L_1\) regularization.
For the scalar example above, taking \(\eta=1\) gives
Applying soft thresholding immediately produces the minimizer:
The subdifferential provides the condition that makes this derivation possible.
The connection to Mixture of Experts—and its limit
A dense Mixture of Experts combines expert outputs using gating weights:
For scalar expert outputs, the chain rule gives
These derivatives describe how the loss changes with each expert output and gating weight. Backpropagation carries them through the networks producing those quantities.
Hard routing introduces a different operation:
It is tempting to treat this as another application of the maximum rule. But the maximum value and the index attaining that maximum are different objects.
The function
returns a score. The convex subgradient rule applies when its component score functions satisfy the relevant convexity and differentiability assumptions.
The expression
selects an index. If that index determines which expert output is used, crossing a routing boundary can switch between different outputs. The resulting model can even be discontinuous.
Understanding subgradients helps us analyze maximum-score functions, but it does not automatically solve differentiation through discrete routing.
Putting the ideas together
The derivative of \(x^2\), the corner of \(|x|\), and the soft-thresholding operator are parts of the same story.
Derivatives and gradients describe local change. The Hessian describes how that local change varies. Convexity lets a local supporting condition certify a global minimum. At a corner, subgradients preserve that supporting geometry even when ordinary derivatives fail.
Algorithm design then depends on the structure available. Smoothness gives control over gradient steps. Nonsmooth convex objectives admit subgradient methods. A smooth loss paired with a structured regularizer often leads naturally to a proximal method.
Before choosing an optimizer, it is worth asking what the objective allows us to know: whether it is differentiable, whether it is convex, how its curvature behaves, and whether its nonsmooth terms can be handled directly.
Those questions explain both why familiar methods work and where their guarantees stop.
A small problem to check your understanding
Consider
- Why is \(f\) convex?
- Is it differentiable at zero?
- What is \(\partial f(0)\)?
- Does the subdifferential certify that zero is a global minimizer?
- Starting from \(x_0=2\), what is one subgradient step with \(\eta=0.5\)?
Show the solution
The maximum of two affine functions is convex. At zero, its left and right slopes are \(-1\) and \(2\), so it is not differentiable there. Its subdifferential is \([-1,2]\), which contains zero and therefore certifies a global minimum. At \(x_0=2\), the active branch is \(2x\), so the subgradient is \(2\) and