argmin, constraints and Lagrange multipliers

min is a value, argmin is where it happens. When the answer must obey a rule, the best point is where the contours touch the constraint and the two gradients line up.

10 min Prerequisites: Lesson 27 (the loss landscape), Module 4 lesson 19 (the gradient)

Symbols in this lessonargmin arg min / arg max min / max min / max λ lambda ∇ nabla / gradient

1. The idea

min f is the lowest value of f; argmin f is where it happens, the input that gives it. Training is an argmin: θ* = argmin L(θ). Prediction is often an argmax: the class with the highest probability.

Sometimes the answer has to obey a constraint: probabilities must add up to 1, a weight vector must have norm at most c, a fine-tuned model must stay close to its reference. Then the best point usually isn't where the gradient is zero, but where you can't improve any further without breaking the rule.

Lagrange multipliers make that precise for an equality constraint g(θ) = c. At the constrained optimum the contour of f just touches the constraint, so their gradients are parallel: ∇f = λ∇g. The number λ says how much the best value of f would change if the constraint were loosened a little. The same idea, extended to inequalities, sits under support vector machines and the KL penalty in RLHF.

2. Try it: slide along the rule

Find the point on the dashed line closest to the green target. Slide along it and watch the two arrows: when −∇f and ∇g line up, you have it. Then move the line with c.

3. The math

A value, and where it happens

\min_{\theta} f(\theta) \text{ is a number}; \qquad \arg\min_{\theta} f(\theta) \text{ is a } \theta

np.min and np.argmin.

A constrained problem

\min_{\theta}\ f(\theta) \quad \text{subject to} \quad g(\theta) = c

Only θ on the constraint count.

The Lagrangian

\mathcal{L}(\theta, \lambda) = f(\theta) - \lambda\,\big(g(\theta) - c\big)

Setting its gradient to zero gives both conditions below.

At the optimum

\nabla f(\theta^{*}) = \lambda\,\nabla g(\theta^{*}), \qquad g(\theta^{*}) = c

The gradients are parallel, and the rule holds.

The widget's problem

\min \|\theta - (2.5,\ 2)\|^2 \ \text{ s.t. } \ \theta_1 + \theta_2 = 1 \;\Rightarrow\; \theta^{*} = (0.75,\ 0.25),\ \lambda = -3.5

The closest point on the line to the target.

4. The code (python)

mm6_lagrange.py

import numpy as np

target = np.array([2.5, 2.0])
# minimize ||theta - target||^2 subject to theta1 + theta2 = 1:
#   2 (theta - target) = lambda * (1, 1)   and   theta1 + theta2 = 1
A = np.array([[2.0, 0.0, -1.0],
              [0.0, 2.0, -1.0],
              [1.0, 1.0,  0.0]])
rhs = np.array([2 * target[0], 2 * target[1], 1.0])
t1, t2, lam = np.linalg.solve(A, rhs)
print(round(t1, 3), round(t2, 3), round(lam, 3))    # 0.75 0.25 -3.5

scores = np.array([0.2, 1.7, 0.4])
print(scores.max(), scores.argmax())                # 1.7 1

5. You’ll see this in

Support vector machines in the catalog a constrained problem solved with multipliers
RLHF in the catalog a penalty that keeps the model near its reference
Reference notes: convexity, optimization and duality

6. Check yourself

Answer all 4 questions correctly to complete the chapter · 0 / 4 done

Q1/4 For scores (0.2, 1.7, 0.4), what does argmax return?

Q2/4 At a constrained optimum, how are ∇f and ∇g related?

Q3/4 Which of these is an equality constraint you meet in ML?

Q4/4 What does the multiplier λ tell you?

Finished this chapter?

Weekly newsletter

What changed in ML architectures this week

New models worth knowing, benchmark moves, and one architecture explained in plain words. Free, once a week, unsubscribe anytime.

Learn machine learning from scratch