Surrogate Modeling for Process Prediction and Optimal Decision-Making · Week 6

Week 6: Solve Time and Results in Neural-Network Optimization

Bounds, scaling, OMLT, and solver interpretation with open-source HiGHS.

Week 6: Solve Time and Results in Neural-Network Optimization

Reactor conditions that meet product specifications

A reactor design review requires repeated searches over temperature and residence time. After finding conditions that meet the product specifications, we still need to assess how much improvement may remain.

Reactor input domain and concentration specifications

This week: unfinished-solve assessment, bounds, and formulation choices.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 1 / 33

The problem and the comparison conditions

Add temperature as an input to the Week 5 reactor setting. At a fixed flow of 1 L/min, residence time also selects reactor volume and is therefore a design variable.

Hold the fitted ReLU model, input domain, objective, and product specifications fixed while comparing formulations. Arbitrarily shrinking the temperature interval changes the design problem.

Solver for all LPs and MILPs: open-source HiGHS.

Learning objectives for formulations and solver results

  1. Incumbent objective, objective bound, and gap
  2. Interval propagation versus LP-based bound tightening
  3. Frozen network and scaling in Pyomo and OMLT
  4. Termination, time, forward agreement, and constraint residuals

Review the Week 5 Big-M equivalence and input-bound calculations first.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 2 / 33

The frozen network and its MILP representation

For a ReLU input a with valid bounds L ≤ a ≤ U and L < 0 < U, use these inequalities with binary z.

h≥0, h≥ah≤Uz, h≤a−L(1−z)

Weights, biases, and scaling remain fixed. The full problem is an MILP when the other constraints and objective are linear. Exact network representation is separate from process prediction error.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 3 / 33

Incumbents, objective bounds, and gaps

For minimization, the incumbent is the best integer-feasible solution found so far. Its objective value supplies an upper bound that, together with a valid lower bound, brackets the optimum.

Jbound≤Jopt≤Jincgabs=Jinc−Jbound, grel=Jinc−Jbound|Jinc|

An upper bound of 100 and lower bound of 80 give an absolute gap of 20 and a relative gap of 20% under this definition. Without an incumbent this gap is unavailable; near a zero objective, also report the absolute gap.

A feasible design supplies an upper bound because there is no need to select anything more expensive. A relaxation admits even more favorable choices and supplies a lower bound. The interval between them is the remaining possible improvement.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 4 / 33

How branch-and-bound progresses

Relaxing binaries to continuous intervals gives an LP lower bound for minimization. A fractional binary can be branched into subproblems with z = 0 and z = 1.

An integer-feasible solution updates the incumbent. Infeasible nodes and nodes whose lower bounds cannot improve the incumbent need no further search.

Valid bounds from open nodes form the global lower bound. HiGHS also uses presolve, cuts, and heuristics, so the original LP relaxation value can differ from the final solver bound.

If the incumbent costs 100 and a node has lower bound 105, every integer solution in that node costs at least 105, so it can be pruned. A node bounded by 80 may still contain an improvement below 100 and needs further investigation.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 5 / 33

The continuous relaxation of a ReLU

Relaxing z to [0, 1] and eliminating it gives this triangle for L < 0 < U.

h≥0, h≥a, h≤UU−L(a−L)

At a = 0, the largest allowed h is −UL/(U − L). It is 0.5 when L = −1 and U = 1, although the ReLU output is zero.

This triangle is the convex hull of the scalar (a, h) graph. Correlations across inputs and neurons may require additional constraints.

With L=−1,U=1 and a=0, the upper bound is (0+1)/2=0.5. Allowing this value above the true h=0 creates relaxation slack. Tighter valid bounds can reduce it.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 6 / 33

Two neurons coupled by an equality

Dimensionless example: a dependence missed by box bounds

0≤x1,x2≤1, x1+x2=1h1=max(0,x1+x2−1),h2=max(0,x1−x2)minx1,x2 x1 subject to h1+h2≥.4

Using only the input box gives [−1, 1] for both pre-activations. The input equality, however, makes the first pre-activation identically zero.

Box bounds alone allow combinations x₁=x₂=0 or 1. The equality x₁+x₂=1 excludes them and fixes the first pre-activation at zero. Individual variable ranges and relations between variables therefore contain different information.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 7 / 33

The exact solution of the two-neuron example

h1=0, h2=max(0,2x1−1)h2≥.4⇒x1≥.7

The minimum x1 is therefore 0.7. The MILP with box bounds and the MILP using the equality both attain this optimum.

The LP relaxation can place h above the network output, so this conclusion does not immediately apply to its solution.

When x₁≤0.5, the second ReLU is also zero and cannot meet 0.4. In its active branch, solve 2x₁−1≥0.4 to get x₁≥0.7. Substituting 0.7 verifies feasibility and makes the lower limit attainable.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 8 / 33

LP relaxations before and after tightening

Box-bound and equality-aware LP envelopes compared with the network

The original LP allows h1 = 0.4 even at x1 = 0. Enforcing h1 = 0 raises the LP optimum to 0.4, still below the MILP optimum of 0.7.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 9 / 33

Relaxed solutions versus the actual network

Each formulation minimizes x₁ with x₂=1−x₁ and required output sum 0.4.

Formulation x₁,x₂ Allowed h₁,h₂ Actual ReLU output Interpretation
Box-bound LP 0,1 0.4,0 0,0 Inflated h₁ meets the limit
Equality-aware LP 0.4,0.6 0,0.4 0,0 Inflated h₂ meets the limit
MILP 0.7,0.3 0,0.4 0,0.4 Matches the original output

After incorporating the equality, the second triangle has upper bound h₂≤x₁. Thus h₂≥0.4 requires x₁≥0.4. Yet at x₁=0.4, the actual a₂=−0.2 gives zero ReLU output. Improving the lower bound does not make the relaxation identical to the graph.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 10 / 33

Branching and termination in the two-neuron example

After enforcing h₁=0, only h₂’s binary z₂ remains. Its bounds are L=−1,U=1 and a₂=2x₁−1. Adding the root LP rows h₂≤z₂ and h₂≤2x₁−z₂ gives h₂≤x₁. Since h₂≥0.4, x₁≥0.4 is a lower bound.

Problem Calculation Conclusion
Root LP, 0≤z₂≤1 x₁=0.4,h₂=0.4,z₂=0.4 is feasible lower bound 0.4; not integer
Child z₂=0 h₂=0 contradicts h₂≥0.4 infeasible; close
Child z₂=1 h₂=2x₁−1≥0.4 minimum attained at x₁=0.7

The children cover both possible binary values. One is infeasible and the other has minimum 0.7, establishing a global lower bound of 0.7. It matches feasible cost 0.7, so the gap is zero.

With several open nodes, the smallest node lower bound identifies a branch that may still improve the incumbent. A branch whose bound reaches the incumbent cannot improve it and can be closed. A real solver may use different branching and cuts.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 11 / 33

The domain over which bounds are valid

Bounds must hold at every allowed input. Training-sample extrema alone do not guarantee coverage of the full input domain.

Overly narrow bounds can cut off valid network points. Wider valid bounds preserve the integer graph but can weaken its relaxation.

Bounds computed using product specifications are valid only over that constrained feasible set. Recheck them when specifications or the input domain change.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 12 / 33

Interval propagation with weight signs

Suppose the previous layer satisfies ℓ ≤ h ≤ u, with w+ = max(w, 0) and w− = min(w, 0).

Lj=bj+∑i(wij+ℓi+wij−ui)Uj=bj+∑i(wij+ui+wij−ℓi)

After a ReLU, pass [max(0, L), max(0, U)] to the next layer. This is inexpensive but does not check whether different variable extremes can occur simultaneously.

If h₁=h₂ but both are represented only by [0,1], interval arithmetic gives h₁−h₂ in [−1,1], although the true difference is zero. Interval propagation quickly constructs valid outward bounds but loses such dependencies.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 13 / 33

LP-based bound tightening

Let P be an LP relaxation containing the network MILP feasible set. Minimize and maximize each pre-activation over P.

Lj=minv∈P aj(v)Uj=maxv∈P aj(v)

This is optimization-based bound tightening, or OBBT. The lab solves 54 LPs over the full input domain without product specifications, then rebuilds the MILP with the resulting bounds.

The lower-bound problem asks how small this neuron can become over every allowed input. The upper-bound problem maximizes the same expression. Solving over a relaxation containing every original network point produces bounds that do not exclude those points.

Suppose minimizing a stops with feasible value −0.2 and valid lower bound −0.6. Its true minimum lies in [−0.6,−0.2]. Use −0.6 as the neuron’s lower bound: using −0.2 could cut off a valid point with a=−0.4. For a maximum, use a valid upper bound instead.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 14 / 33

Handling neurons with fixed activation states

U≤0⇒h=0L≥0⇒h=a

These cases need no binary variable. Only neurons whose intervals cross zero remain candidates for branching.

Hand-built MILP binary counts: 24 with loose bounds, 22 with interval propagation, 19 with LP-OBBT. Counts before solver presolve.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 15 / 33

Input scaling and physical units

x1=T−31040, x2=τ−28T=310+40x1, τ=2+8x2

Reuse exactly the offsets and scales used during fitting. Supplying physical-input bounds as scaled-input bounds changes the domain evaluated by the network.

Outputs in this lab already use mol/L. If outputs were standardized, include the fixed inverse transform before connecting product specifications.

330 K and 6 min are both interval midpoints, giving scaled input [0.5,0.5]. Swapping input order makes temperature weights read residence time. Comparing forward outputs at identical physical inputs also detects this ordering error.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 16 / 33

Objective constants and relative gaps

J=τ+α(T−310), α=.02 min/KJ=2+8x2+.8x1

Educational objective: residence time + temperature increase weighted by α. Unit: min-equivalent score.

Dropping the constant 2 preserves the optimizer but changes the denominator of a relative gap. The lab reports objectives and bounds with the constant included.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 17 / 33

Computing gaps with an objective constant

Keep the same incumbent and valid bound, removing only constant 2 from the objective.

Calculation J with constant J−2 without constant
Feasible objective 3.2 1.2
Valid lower bound 2.8 0.8
Absolute gap 3.2−2.8=0.4 1.2−0.8=0.4
Relative gap 0.4/3.2=12.5% 0.4/1.2≈33.3%

Subtracting the same number from every candidate preserves ranking and the optimal input. The relative-gap denominator changes, however. Percentages alone cannot compare solve quality when the objective definitions differ.

Without an incumbent there is no value for that denominator. Report the valid bound and absence of a feasible solution; do not interpret initialized variables as the answer.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 18 / 33

Roles of the open-source tools

Tool Role
NumPy Network forward pass and optional refitting
Pyomo Variables, objective, and process constraints
OMLT Frozen-network constraints in Pyomo
HiGHS / highspy LP and MILP solution

All LPs and MILPs use open-source HiGHS under the MIT license.

Check the input ordering, scaling, and bounds of the constraints generated by OMLT. HiGHS official site

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 19 / 33

Connecting OMLT to process constraints

Build definition from the weights and scaling, then connect the process constraints. nn.inputs uses physical units; scaled_inputs holds transformed inputs.

model.nn = OmltBlock()
model.nn.build_formulation(ReluBigMFormulation(definition))
model.product = pyo.Constraint(expr=model.nn.outputs[1] >= .38)
model.byproduct = pyo.Constraint(expr=model.nn.outputs[2] <= .25)
model.obj = pyo.Objective(
    expr=model.nn.inputs[1] + .02*(model.nn.inputs[0]-310))

outputs[1] is B and outputs[2] is C. inputs[0] is temperature in K; inputs[1] is residence time in min. OMLT scaling equalities connect physical inputs to network coordinates; the objective must use the intended physical quantities.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 20 / 33

Reproducing Week 5 with OMLT

First check the existing interpolant with residence time as its sole decision, at 330 K and a feed A concentration of 1.2 mol/L. Apply B ≥ 0.40 and C ≤ 0.25 mol/L.

Formulation Minimum residence time [min] Generated binaries
Hand-built 5.353846 3
OMLT 5.353846 4

This OMLT version creates a binary for each ReLU. The hand-built model represents one always-active neuron linearly, using one fewer binary while giving the same outputs.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 21 / 33

Choosing temperature and residence time

minT,τ J=τ+.02(T−310)310≤T≤350, 2≤τ≤10y=fθ(T,τ), yB≥.38, yC≤.25

Units: concentrations mol/L, temperature K, residence time min. Additional constraint: nonnegative predicted concentrations.

Architecture: two inputs, two hidden layers of 12 ReLUs each, and three linear outputs. All comparisons: the same frozen coefficients.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 22 / 33

Fitting and independently evaluating the frozen model

Generate concentrations at 1,200 training inputs from ideal isothermal A → B → C CSTR balances and fit with 6,000 NumPy Adam updates. Evaluate physical-unit errors at 500 independent random inputs.

Component Test RMSE [mol/L]
A 0.00546
B 0.00440
C 0.00564

The default lab loads the supplied coefficients. --refit optionally refits them; weights never change during optimization.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 23 / 33

Comparing relaxation values across bounds

LP objectives for three bound settings and their common MILP optimum

LP-OBBT raises the original LP lower bound from 2.0000 to 2.5215. All three MILPs attain 3.1656. All three comparisons use the same predictor.

The interval from lower bound 2.5215 to feasible value 3.1656 is about 0.6441, narrower than 1.1656 from the loose bound 2.0000. Predictions are unchanged, but less optimality uncertainty remains at the start of the search.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 24 / 33

Timing including bound computation

Time for bounds, model construction, and the MILP solve

In the supplied run, LP-OBBT reduces MILP solve time, but its total time including bounds exceeds interval propagation. These single-run measurements include interface overhead and do not establish a general speed ranking.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 25 / 33

Total time when bounds are reused

In the recorded run, interval-bound MILP time was 0.1539 s and LP-OBBT MILP time was 0.0955 s. OBBT preprocessing required 0.1132 s.

Solves m Interval: 0.0001+m×0.1555 OBBT: 0.1132+m×0.0971
1 0.1556 s 0.2103 s
2 0.3111 s 0.3074 s
5 0.7776 s 0.5987 s

Each per-solve cost includes 0.0016 s for construction. This arithmetic uses rounded recorded times and assumes identical solve times on every repetition; it is not a repeated timing experiment.

Bounds reusable for fixed weights and input domain spread preprocessing cost across solves. A new model or expanded domain requires renewed bound validation, so this calculation cannot automatically be carried over.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 26 / 33

Selected conditions and solver conclusions

Item Supplied run
Temperature [K] 342.059300
Residence time [min] 2.524440
Objective [min-equivalent] 3.165626
Predicted B [mol/L] 0.380000
Termination / relative gap optimal / 0

HiGHS establishes optimality for this surrogate MILP within the configured numerical tolerances. The B lower bound is active; the C upper bound has slack.

The objective is 2.524440 + 0.02 × (342.059300−310) = 3.165626. The temperature contribution 0.641186 adds to residence time. Recomputing it checks units and offset handling in the reported solver value.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 27 / 33

Checking forward agreement and residuals

Remove product specifications, fix the input, and solve only the network representation. Evaluate the four domain corners and 12 independent interior points with both the hand-built and OMLT models.

Across 32 checks, the maximum absolute output difference is 7.77e-16 mol/L. Recompute the forward pass at the selected optimum too.

Check constraint and integrality residuals. Finite samples help detect implementation errors; equivalence over the entire domain follows from the formulation conditions developed in Week 5.

Define inequality violations to be zero when satisfied: max(0,0.38−B) for B≥0.38 and max(0,C−0.25) for C≤0.25. For binary value z=0.999999, the integrality residual is its distance 0.000001 to the nearest integer 1; compare it with the solver tolerance.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 28 / 33

Termination and incumbent availability

Termination Interpretation
optimal Completed within gap and numerical tolerances
Time limit, incumbent found Report the feasible solution and remaining gap
Time limit, no incumbent No usable solution has been obtained
infeasible The formulated feasible set is empty
unbounded / error Inspect bounds, objective, logs, and model

A time limit alone does not establish infeasibility. The lab returns maxTimeLimit with no incumbent at a zero-second limit, and infeasible when B ≥ 2.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 29 / 33

Checking solution availability and loading results

opt = Highs()
opt.config.load_solution = False
opt.config.time_limit = 30.0
result = opt.solve(model)
inc = result.best_feasible_objective
print(result.termination_condition)
if inc is not None and np.isfinite(inc):
    result.solution_loader.load_vars()
    print(inc, result.best_objective_bound)

Reading variables after termination without a solution can mistake initial or stale values for results. The complete lab function also returns gaps and constraint residuals.

load_solution=False prevents automatic loading immediately after the solve. Call load_vars only after confirming a finite feasible objective. Then compute specification and integrality residuals from those loaded values.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 30 / 33

Solver gaps and process prediction error

At the same selected input B concentration [mol/L]
Network and MILP 0.380000
Reference CSTR 0.371157
Required minimum 0.380000

The reference CSTR misses the B specification. Forward agreement and a zero solver gap still require a separate process-model feasibility check.

Week 8 uses errors at selected points to reconsider decisions and identify additional data.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 31 / 33

Running the lab and reporting comparisons

Extract the lab bundle and run these commands in its directory.

python -m pip install -r requirements.txt
python week06_lab.py --output-dir my-results
  1. Verify the toy LP values 0 and 0.4 and MILP value 0.7 by hand.
  2. Comparison for the same frozen model: bounds, binary counts, LP lower bounds, and total times
  3. Report: termination, gap, forward differences, and reference CSTR B-specification violation

Reading companion · Lab ZIP

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 32 / 33
Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 6 · 33 / 33