Sc. 01Constrained gradient optimization / PyTorch + NumPy

Descend hard landscapes, without falling off.

A box-constrained optimizer that stays numerically stable when the loss is badly scaled, steep, or full of overflow traps. Derivation, code, unit tests and benchmarks, built end to end as a working sample.

Watch it run See the numbers
0x

fewer evaluations than the best-tuned Adam on constrained Rosenbrock

0

learning rates to tune. The line search finds the step

0/18

unit tests passing, including stability edge cases

0

max relative gap between autograd and finite differences

Sc. 02The landscape

Rosenbrock's valley, fenced in.

The valley floor is easy to see and hard to walk. The fence is a box constraint that cuts off the true minimum, so the answer sits on the boundary. Same start, five optimizers, every point below is a real iterate exported from demo.py.

iteration 0

    Drag to orbit

    Sc. 03What usually breaks

    Three ways a correct formula still fails.

    Overflow

    The textbook log(1 + exp(-m)) returns inf at m = -1000, and its gradient is nan. One bad sample poisons the whole batch.

    Step size

    Fixed-step gradient descent with lr = 0.005 on Rosenbrock diverges at iteration 5. With 0.001 it is still far away after 1500 steps. There is no safe constant.

    Silent gradient bugs

    The popular max(z,0) + log1p(exp(-|z|)) has the right value, but autograd gives gradient 0 at z = 0 instead of 0.5. A model started at w = 0 never moves. I found this one while building the sample.

    Sc. 04The method

    Projected quasi-Newton, with a safety net.

    L-BFGS curvature on the free variables, a projection that respects the box, and an Armijo line search that refuses any step that does not pay for itself.

    feasible box x x + t d P(x + t d)
    Projection onto a box is a clip, so every trial point is feasible by construction.
    1. 01

      Stationarity that knows about the box

      Projection onto the box is a componentwise clip, and a point is a KKT point exactly when a projected gradient step does not move it:

      $$P(x)=\min(\max(x,\ell),u),\qquad x^\star = P\big(x^\star-\nabla f(x^\star)\big).$$

      So the stopping test is \(\|x-P(x-\nabla f(x))\|_\infty\le \text{tol}\). A plain \(\|\nabla f\|\) test is wrong whenever a bound is active.

    2. 02

      Two-metric direction

      Variables pinned to a bound with the gradient pushing outward form the \(\varepsilon\)-active set \(A\). Curvature is only used on the free set \(F\):

      $$d_F=-H_k\,g_F,\qquad d_A=-g_A,\qquad H_k\succ 0\ \text{(L-BFGS, curvature pairs with } s^\top y>0\text{)}.$$
    3. 03

      Armijo along the projected arc

      $$f\big(P(x_k+t\,d_k)\big)\;\le\; f(x_k)+c_1\,g_k^\top\big(P(x_k+t\,d_k)-x_k\big),\qquad t\leftarrow \beta t.$$

      A trial point with a non-finite loss counts as a failed test, so overflow shrinks the step instead of poisoning the iterate. If the quasi-Newton step is not a descent step, memory resets and a projected gradient step is taken.

    4. 04

      What is guaranteed

      For an \(L\)-smooth \(f\), backtracking always ends with a step bounded away from zero, without knowing \(L\):

      $$t_k\;\ge\;\min\{t_0,\;2\beta(1-c_1)/L\},\qquad \sum_k\|x_{k+1}-x_k\|^2<\infty.$$

      The loss never increases, and every limit point is a KKT point. Convex problems converge at \(O(1/k)\), and strongly convex ones converge linearly. The full argument is in DERIVATION.md.

    Sc. 05Benchmark

    Ill-conditioned, constrained, measured honestly.

    Logistic regression, 600 samples, 24 weights boxed to \([-2,2]\), feature scales spanning six orders of magnitude in curvature. The x-axis counts loss evaluations, so line-search trial steps are paid for. Adam gets six learning rates, GD gets four, and my method gets none.

    f(x) - f*  (log scale)  vs  loss evaluations

      Methodevals to 1e-3evals to 1e-6evals to 1e-9final gap

      Sc. 06Stability

      Values and gradients, finite everywhere.

      The logistic term \(\log(1+e^{-m})\) evaluated three ways on extreme margins. Red cells are where a training run silently breaks.

      gradient check, autograd vs central differences1e-7
      fix"double where": each branch only sees inputs where exp cannot overflow

      Sc. 07Unit tests

      18 of 18 passing.

      Run on every export of this page. Known solutions, edge cases, regression tests for the bugs above, and the guarantees from Sc. 04 checked directly: iterates never leave the box, and the loss never goes up.

      Sc. 08Drop-in code

      Four lines into your pipeline.

      your_pipeline.py
      from stableopt import ProjectedLineSearch

opt = ProjectedLineSearch(lower=-2.0, upper=2.0, tol=1e-8)
result = opt.minimize(loss_fn, w0)      # loss_fn: tensor -> scalar tensor

result.x, result.fun, result.converged, result.message
result.history                           # per-iteration f, projected grad, step
      terminal
      pip install numpy torch pytest
python -m pytest -q        # unit tests
python demo.py             # benchmarks + site data

      source

      • stableopt/optimizer.pyprojected L-BFGS + Armijo, safeguards
      • stableopt/safe_ops.pyoverflow-safe softplus, log, division
      • stableopt/gradcheck.pyfinite-difference gradient check
      • stableopt/problems.pybenchmark problems and baselines
      • tests/unit tests
      • DERIVATION.mdderivation and convergence proof
      • demo.pyreproduces every number on this page
      Download source (.zip)

      Sc. 09On a real project

      Same discipline, built around your loss.

      1. 01
        Read the objective

        Loss, constraints, boundary conditions: check smoothness, scaling and where it can overflow.

      2. 02
        Derive and document

        Pick the method, write the derivation and convergence notes another researcher can follow.

      3. 03
        Implement for your stack

        A clean NumPy / PyTorch module that drops into the existing pipeline.

      4. 04
        Prove it

        Unit tests for edge cases and a reproducible notebook against your baseline metric.