Surrogate Modeling for Process Prediction and Optimal Decision-Making · Week 4 reading

Week 4 Reading: Physical Constraints and Projection

Residual losses, KKT projection, scaling, and differentiable physical equalities.

Week 4 Reading: Physical Constraints and Projection

1. Derive the sum equality from steady balances

For first-order consecutive reactions A → B → C, steady state requires r_A = r_B = r_C = 0. Solve A first, then substitute into B and C:

CA=CAf1+k1τCB=k1τCA1+k2τ, CC=k2τCB

The B balance gives k₁τC_A = (1 + k₂τ)C_B, hence C_B + C_C = k₁τC_A. Combine with C_Af = (1 + k₁τ)C_A from the A balance to obtain the concentration-sum equality.

Check: 330 K, τ = 5 min, and C_Af = 1.2 mol/L give [0.6, 0.4, 0.2].

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 1 / 21

2. The stationary point of a soft penalty

For J(z) = ½(z − a)² + ½λ(z − b)² with λ ≥ 0, dJ/dz = (z − a) + λ(z − b). The second derivative is 1 + λ > 0, so the stationary point is the unique minimizer.

z∗=a+λb1+λz∗−b=a−b1+λ

Exercise: set a = 1.4 and b = 1.2; compute the solution and residual for λ = 1 and 9.

Answer check: solutions are 1.30 and 1.22; residuals are 0.10 and 0.02. A finite penalty can leave a residual even at its exact minimizer.

Collecting the derivative terms gives (1+λ)z=a+λb. With a=1.4,b=1.2,λ=9, this is 10z=12.2, hence z=1.22. The data derivative −0.18 cancels the penalty derivative 9×0.02=+0.18, although the equality residual remains 0.02.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 2 / 21

3. Equality QP and its KKT derivation

Let B ∈ ℝᵐˣⁿ have rank m and let d ∈ ℝᵐ. Hold the input and raw prediction fixed and solve:

yp=argminz 12‖z−y^‖2 subject to Bz=d(x) z−y^+BTλ=0, Bz=d

Stationarity gives z = ŷ − Bᵀλ. Substitution into feasibility gives BBᵀλ = Bŷ − d. Invertibility makes λ and z unique. The equality set is convex and the objective Hessian is I.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 3 / 21

4. Equality guarantee and projector identities

G=BBT, Gλ=By^−dyp=y^−BTG−1(By^−d)

Multiplying by B shows that correction cancels the original residual.

Byp=By^−GG−1(By^−d)=d

P = I − Bᵀ(BBᵀ)⁻¹B satisfies Pᵀ = P, P² = P, and BP = 0. It projects onto the null space of B. Feasible raw outputs receive zero correction, and projecting an already projected output changes nothing.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 4 / 21

5. Hand calculation: correction and error

B=[111], d=1.2yp=[.70.45.25]−.203[111]=[.633333.383333.183333]

Against reference [0.6, 0.4, 0.2], the raw sum of squared errors is 0.015 (mol/L)². Squared correction norm is 0.04/3 and projected squared error is 1/600 ≈ 0.001667.

The A error changes from +0.10 to +0.033333. B and C change from +0.05 to −0.016667. Calculate aggregate and individual errors separately.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 5 / 21

From the sum residual to projected output

The raw output is [0.70,0.45,0.25] and the target total is 1.20 mol/L. Equal corrections Δ give 1.40−3Δ=1.20, hence Δ=0.20/3.

Step Value
Bŷ−d 1.40−1.20=0.20
G=BBᵀ 1²+1²+1²=3
Solve Gλ=residual λ=0.0666667
Component corrections [−0.0666667,−0.0666667,−0.0666667]
Projected output [0.6333333,0.3833333,0.1833333]

Equal correction has squared distance 3×(0.20/3)²=0.013333. Subtracting 0.20 only from A also repairs the sum but has squared distance 0.04. This explains equal correction under an equally weighted Euclidean metric.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 6 / 21

Projecting onto two independent equalities

Consider a linear example requiring both total concentration 1.2 and A−B=0.2. The rows of constraint matrix B are [1,1,1] and [1,−1,0], with right side [1.2,0.2]. Project raw output [0.70,0.45,0.25].

Step Calculation
Residuals Bŷ−d [0.20,0.05]
Diagonal entries of G=BBᵀ row squared norms: 3 and 2
Off-diagonal entry 1×1+1×(−1)+1×0=0
Solve Gλ=residual λ=[0.20/3,0.05/2]
Bᵀλ [0.091667,0.041667,0.066667]
Projected output [0.608333,0.408333,0.183333]

The sum is 1.2 and A−B=0.2. Independence allows the two residuals to be removed separately. If the second row were twice the first, G would have no inverse.

Here B names the constraint matrix, distinct from product species B. The additional A−B equality is a condition of this linear calculation example.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 7 / 21

6. Chain rule for parameters and inputs

Let J_f,θ be the raw-output parameter Jacobian. For fixed A, B, b:

∇θL=Jf,θTPT∇ypL∂yp∂x=P∂fθ∂x−BTG−1A

Input gradients include the direct path through d(x) = b − Ax. If B depends on inputs, P does too and additional derivative terms are required; the fixed-projector formula is insufficient.

Check: upstream [1, 0, 0] and B = [1, 1, 1] yield [2/3, −1/3, −1/3].

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 8 / 21

7. Equality and distance in standardized coordinates

y=μ+Dz, BDz=d−BμD=diag(s1,…,sn)

Euclidean projection in z minimizes ‖D⁻¹(y − ŷ)‖² in physical coordinates, weighting each squared component correction by 1/s_j².

Example: raw [0.70, 0.45, 0.25], total 1.20, and s = [0.2, 0.5, 0.1] give restored standardized-projection output [0.673333, 0.283333, 0.243333]. Corrections are proportional to s_j².

Both methods satisfy the sum. For outputs with different units, even a physical Euclidean distance depends on unit choices; design the metric explicitly.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 9 / 21

Projection metric and component corrections

Minimizing distance in standardized coordinates with scales [0.2,0.5,0.1] allocates physical corrections in proportion to squared scales. Split the residual 0.20 in the ratio 0.04:0.25:0.01.

Component Squared-scale fraction Subtracted concentration Result
A 0.04/0.30 0.026667 0.673333
B 0.25/0.30 0.166667 0.283333
C 0.01/0.30 0.006667 0.243333
scales = np.array([0.2, 0.5, 0.1])
raw = np.array([0.70, 0.45, 0.25])
weights = scales**2 / np.sum(scales**2)
projected = raw - (raw.sum() - 1.2) * weights

B has the largest scale, making a physical correction less costly in standardized distance. It therefore changes more. Checking only the final sum 1.20 hides the difference between these metrics.

Let Δⱼ be the correction. Minimize ½∑Δⱼ²/sⱼ² subject to ∑Δⱼ=−0.20. Stationarity gives Δⱼ/sⱼ²+λ=0, hence Δⱼ=−λsⱼ². Substituting into the sum constraint gives λ=0.20/∑sⱼ², which explains the squared-scale proportions.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 10 / 21

8. Redundant equalities and inconsistent right sides

B=[111222]

The second row is twice the first, so rank is 1. With d = [1.2, 2.4], both equations define the same set; retaining one independent row is sufficient.

With d = [1.2, 2.5], twice the first equation contradicts the second. No feasible output exists. A pseudoinverse can return a least-squares result, but cannot make every residual zero.

When discarding small singular values, recheck residuals of removed equations in their original units.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 11 / 21

9. Nondimensionalization of residuals with different units

The sum residual has units mol/L; species-balance residuals have units mol/(L·min). Adding their squares without scales makes the meaning of λ depend on the unit convention.

For example, choose reference concentration c∗ and time t∗; divide the sum residual by c∗ and species residuals by c∗/t∗. Still specify loss averaging and λ after nondimensionalization.

The lab penalizes the sum residual; known kinetics can also define species-balance residuals.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 12 / 21

10. Accuracy conditions and incorrect physics

‖y^−y∗‖2=‖y^−yp‖2+‖yp−y∗‖2

This decomposition requires y∗ in the same affine set. The correction is normal to the set while y_p − y∗ is tangent, so their inner product vanishes.

If the true total is 1.2 but an incorrect equality enforces 1.1, the true output lies outside the set. The minimum Euclidean distance from truth is already 0.1/√3 ≈ 0.057735 mol/L.

Check sensor bias, omitted species, and stoichiometric errors first. Projection enforces an incorrectly specified equality too.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 13 / 21

11. Joint nonnegativity and concentration-sum constraints

Clamping negative values after equality projection changes the sum. Requiring both the sum and nonnegativity gives:

minz 12‖z−y^‖2 s.t. 1Tz=CAf, z≥0

For C_Af ≥ 0 the feasible set is a simplex. The solution is affine within a fixed active-set region; derivatives need care where a component reaches zero.

The required lab stops at linear equalities. Report results and gradient checks separately if implementing this inequality extension.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 14 / 21

Recomputing projection with nonnegativity

Subtracting the sum residual 0.22 equally from [0.02,0.90,0.50] makes A negative. Requiring both the sum and nonnegativity leads to recomputation on the boundary A=0.

Step Calculation
Fix A at zero B+C=1.20
Remaining raw sum 0.90+0.50=1.40
Correction to each free component (1.40−1.20)/2=0.10
Result [0,0.80,0.40] mol/L

Subtract 0.10 from each positive component while A stops at zero. This is the simplex projection of the raw output. Clipping only the negative entry does not readjust B and C, so it does not restore the sum.

As raw values change, the active constraint changes when A becomes positive again. The derivative is therefore not one fixed matrix over the entire domain, unlike fixed linear-equality projection.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 15 / 21

12. Derive the dynamic total-concentration transition

Adding dynamic species balances gives dS/dt = (C_Af − S)/τ. With constant C_Af and τ within an interval, the derivative of S − C_Af is −(S − C_Af)/τ, hence:

Sk+1=CAf+(Sk−CAf)e−Δtτ

Exercise: use S_k = 1.0, C_Af = 1.2 mol/L, τ = 5 min, and Δt = 0.2 min. Compute the next total.

Answer check: 1.2 − 0.2 exp(−0.04) ≈ 1.007842 mol/L. Forcing the steady-state total immediately would change this transition. Project the raw next state onto this right side instead.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 16 / 21

13. General-B implementation and gradient check

raw has shape (batch, n), rhs (batch, m), and B (m, n). B is fixed and must have independent rows.

def linear_project(raw, rhs, B):
    residual = raw @ B.T - rhs
    lam = torch.linalg.solve(B @ B.T, residual.T).T
    return raw - lam @ B

# Use float64 and independent inputs for finite differences.
assert torch.autograd.gradcheck(
    lambda a, d: linear_project(a, d, B), (raw, rhs)
)

Create checked raw and rhs tensors with requires_grad=True. The lab’s –check-only checks feasibility, idempotence, Jacobians, scaling, and counterexamples.

The first residual has shape (batch,m). Transpose it to (m,batch) so solve handles all right-hand sides, then transpose back. Finally lam @ B has shape (batch,n), the output correction. gradcheck compares automatic derivatives with central differences from perturbing an input component by +ε and −ε and dividing the output difference by 2ε.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 17 / 21

14. Define the comparison metrics

Component-j RMSE is the square root of mean squared prediction error over test rows. Pooled RMSE averages over rows and components. Sum MAE and maximum use the absolute sum residual.

Count equality violations when the absolute sum residual exceeds 10⁻⁶ mol/L and negative entries when a predicted concentration is below −10⁻⁸ mol/L. Also report the minimum predicted concentration.

Species-balance RMS averages r_j² over all three species and test rows, then takes the square root; units are mol/(L·min). Do not select settings using test. Assess variation across seeds in a separate repeated experiment.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 18 / 21

15. Exercises with short answer checks

  1. How many output directions remain trainable for B = I₃? Answer: P = 0 fixes the output to d; there are no free directions.
  2. How many remain for B = [1, 1, 1]? Answer: null-space dimension is 3 − 1 = 2.
  3. What happens to a raw output whose sum is already 1.2? Answer: zero residual means no change.
  4. Does zero test sum-violation count make the hard model fully physically feasible? Answer: inspect negative concentrations and species balances separately.
  5. Does post-projection reduce every component error? Answer: no. Nonincrease applies to total squared error in the same metric with a feasible reference.
Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 19 / 21

16. Primary-source reading guide

Chen, H., Constante Flores, G. E., & Li, C. (2024). Physics-informed neural networks with hard linear equality constraints. C&CE 189, 108764. DOI · Manuscript · Author repository. Compare Section 3’s fixed projection and assumptions; read one Section 4 process case.

Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. JCP 378, 686-707. DOI · 2017 Part I. Read Part I’s continuous-time residual and coordinate differentiation.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 20 / 21

17. Sources for mathematics and implementation

Boyd, S., & Vandenberghe, L. (2004). Convex Optimization. Official book. Sections 5.5.3 (KKT conditions) and 10.1 (equality-constrained minimization).

PyTorch: torch.linalg.solve solves a linear system without explicitly forming an inverse. gradcheck compares automatic differentiation with finite differences; default tolerances assume double precision.

Return to the lecture.

Sunwoo Kim · Surrogate Modeling for Process Prediction and Optimal Decision-Making Week 4 reading · 21 / 21