Problem and setup
Choose a dimensionless operating variable u∈[0,1] that minimizes a convex cost surrogate. A one-dimensional hinge network is a simple input-convex model: an affine term plus nonnegative multiples of ReLU(u−knot). First solve a hand model, then fit a convex quadratic reference and compare prediction error with decision loss.
Extract the lab ZIP, keeping requirements.txt, the notebook, Python scripts, and supplied models together. These exercises were checked with Python 3.12. Use that folder as the working directory and run the commands below, or execute the notebook cells in order. Downloading the notebook alone can omit required scripts or model files.
python -m pip install -r requirements.txt
python guided_lab.py
guided_lab.py executes the calculations below in order and checks key values with assertions. Files in guided-results/ belong to your new run; compare them with the supplied reference results. For Jupyter, select a kernel from the same Python environment. For ModuleNotFoundError, check that requirements were installed in that environment. For missing local files, check the working directory.
1. Calculate values and slopes by interval
Use f(u)=0.4−0.6u+ReLU(u−0.4)+0.5ReLU(u−0.7). Before 0.4, both hinges are zero and the slope is −0.6. After the first knot, add 1 to the slope; after the second, add 0.5. Nonnegative additions make slopes nondecreasing.
from pathlib import Path
import json
import numpy as np
from scipy.optimize import linprog, lsq_linear
import week07_lab as lab
hand = {"knots":[.4,.7],"coefficients":[.4,-.6,1.,.5]}
points = np.array([0.,.4,.7,1.])
design = lab.features(points,np.array(hand["knots"]))
values = lab.forward(points,hand)
print("columns [1,u,hinge1,hinge2]:",design)
print("values:",values,"slopes:",[-.6,.4,.9])
np.testing.assert_allclose(values,[.4,.16,.28,.55])
The cost falls until u=0.4 and then rises, so the minimum is f(0.4)=0.16. The convexity argument depends on the nonnegative hinge weights, not on ReLU alone.
2. Let minimization tighten the lifted activations
Replace each hinge by h≥u−knot and h≥0. With positive objective weights, increasing h raises cost, so minimization chooses the smallest feasible h. The resulting optimal objective matches the forward network without binaries.
solution = lab.solve_lp(hand)
print("LP result:",solution)
u = solution["u"]
hidden = np.maximum(u-np.array(hand["knots"]),0)
print("minimal hidden values:",hidden)
np.testing.assert_allclose([u,solution["objective"]],[.4,.16])
assert solution["binaries"] == 0
At u=0.4, hidden values are [0,0]. Tightness is induced by the minimization direction and weights. A constraint that rewards inflated h can break this equivalence.
3. Construct the misleading lower-bound example
For h=ReLU(u) with u≥0, requiring h≥0.8 means u≥0.8. But the epigraph inequalities h≥u,h≥0,h≥0.8 allow u=0,h=0.8. Minimizing u chooses that false candidate because the objective no longer penalizes h.
relaxed = linprog([1.,0.],A_ub=[[1.,-1.],[0.,-1.]],b_ub=[0.,-.8],
bounds=[(0,1),(0,None)],method="highs")
print("lifted [u,h]:",relaxed.x,"true ReLU at u:",max(0,relaxed.x[0]))
assert relaxed.success and abs(relaxed.x[0]) < 1e-9
assert max(0,relaxed.x[0]) < .8
unbounded = linprog([0.,-1.],A_ub=[[1.,-1.]],b_ub=[0.],bounds=[(0,1),(0,None)],method="highs")
assert unbounded.status == 3
The lower-bound model is falsely feasible; a negative coefficient rewarding unbounded h makes the second LP unbounded. Always check the direction in which the objective and constraints use a lifted network output.
4. Fit, then distinguish prediction and decision error
The reference q(u)=+0.1u+0.2 has derivative 2u−1.1, so its minimum is at u=0.55 with cost 0.2575. For 41 training points, build columns [1,u,hinges] and fit 11 coefficients. The first two coefficient bounds are unrestricted; the remaining nine lower bounds are zero to preserve convexity. Evaluate 200 separate midpoint samples. The fitted LP selects an operating point; q at that point measures the real decision cost.
reference = lambda z: (np.asarray(z)-.6)**2+.1*np.asarray(z)+.2
knots = np.arange(.1,1.,.1)
train = np.linspace(0,1,41)
design = lab.features(train,knots)
lower = np.r_[[-np.inf,-np.inf],np.zeros(len(knots))]
fit = lsq_linear(design,reference(train),bounds=(lower,np.full(len(lower),np.inf)),tol=1e-12)
assert fit.success
print("training matrix:",design.shape,"first feature row:",design[0])
print("coefficients [b,a,hinge weights]:",fit.x)
report = lab.run(Path("guided-results"))
model = json.loads(Path("guided-results/fitted_icnn.json").read_text())
np.testing.assert_allclose(fit.x,model["coefficients"])
u = report["fitted"]["u"]
decision_loss = float(reference(u)-reference(.55))
print("selected u:",u,"surrogate cost:",lab.forward([u],model)[0])
print("reference cost:",reference(u),"decision loss:",decision_loss)
np.testing.assert_allclose(reference(.55),.2575)
np.testing.assert_allclose(decision_loss,report["reference"]["decision_loss"])
print("held-out prediction errors:",report["heldout"])
print("fixed-context solutions:",report["conditional"])
RMSE averages prediction errors over independent points; decision loss compares reference costs at two selected points. They answer different questions. In the conditional examples, context is fixed while u is optimized; convexity in u does not imply joint convexity in context and u. Change knot spacing using training/validation data, then report held-out error and reference decision loss separately.
What to report
Record the inputs, outputs, units and fixed conditions; your hand prediction; intermediate computed values; and final metrics with interpretation. Before changing one setting, predict the direction of its effect, then revise that explanation after execution. Record the environment and seed.