Skip to content

Nonlinear uncertainty: equivalence regions and swarm sampling

The Fisher bound and the Laplace approximation of the Bayesian view are linear statements: they describe the posterior by an ellipse around one estimate. A body of work on geophysical inverse problems by Fernández-Martínez, García-Gonzalo and co-workers (University of Oviedo) argues that the object to report is not the estimate but the region of equivalence, the set of all parameters that fit the data within the noise, and that this region is sampled rather than linearized. This page summarizes the nine papers that make that case and what they change in the design of pinnrl.uq.

The region of equivalence

With a forward map \(F\), observations \(d_{\text{obs}} = F(m_{\text{true}}) + \delta d\) and a tolerance \(\mathrm{tol}\) set by the noise, the solution of an inverse problem is the set

\[ M_{\mathrm{tol}} = \{\, m : \|F(m) - d_{\text{obs}}\| \le \mathrm{tol} \,\}, \]

not its minimum-misfit member (Pallero et al., 2018, after Tarantola). Three facts about it carry over to PINN identification unchanged:

  1. The best-fit point is not the truth. With noise, the minimizer \(m_0\) of the perturbed misfit satisfies \(\nabla c(m_0) = 2 J^\top \delta d \ne 0\) for the noise-free misfit \(c\): the true model lies inside \(M_{\mathrm{tol}}\) with a higher misfit than the best fit. Reporting the minimizer alone reports the noise.
  2. The linearized region is only local. The ellipse from the Jacobian at \(m_0\) (the Fisher ellipse of identifiability) matches \(M_{\mathrm{tol}}\) near \(m_0\) and nowhere else; for nonlinear forward maps the true region is a curved valley (the "banana") elongated along the small singular values of \(J\), possibly disconnected, and the ellipse both includes models outside it and misses models inside it. F02 is such a case in pinnrl: the PINN's optimum for the two KdV coefficients sits 20 % from the truth with a tight Fisher ellipse around it.
  3. Noise and regularization deform the region. Noise shrinks the low-misfit part of the valley and may move the best fit into a different basin (Fernández-Martínez et al., 2014a, b); a Tikhonov term bounds the region anisotropically, most along the directions of highest uncertainty, and can create local minima. For a PINN the physics weights play the role of the regularization: F07 found that the soft physics term flattens the data-misfit profile 2.5–12×, which is the same deformation seen from inside the loss.

Sampling while optimizing

The Oviedo group samples \(M_{\mathrm{tol}}\) with particle swarms used in their exploratory form: a swarm of parameter vectors moves under

\[ v_i^{k+1} = \omega\, v_i^k + \phi_1 (g^k - x_i^k) + \phi_2 (l_i^k - x_i^k), \qquad x_i^{k+1} = x_i^k + v_i^{k+1}, \]

with \(g^k\) the best position of the swarm, \(l_i^k\) the best of particle \(i\), \(\phi_1, \phi_2\) random accelerations and \(\omega\) the inertia; every evaluated position with misfit below the tolerance is kept as a sample. The method needs only forward solves (no gradient, no inverse operator, no regularization or reference model), and its exploration is controlled by one theory:

  • PSO is a damped mass–spring system. Each particle obeys the stochastic second-order difference equation \(x^{k+1} + (\phi - \omega - 1) x^k + \omega x^{k-1} = \phi_1 g^k + \phi_2 l^k\), the discretization of \(x'' + (1 - \omega) x' + \phi (x - o) = 0\) with a stochastic centre of attraction \(o\) (Fernández-Martínez & García-Gonzalo, 2008, 2011). The generalized PSO uses a time step \(\Delta t \ne 1\): larger steps explore, smaller steps freeze the swarm.
  • A family from the discretization. Centering or delaying the velocity and acceleration differences gives CC-PSO, CP-PSO, PP-PSO and RR-PSO (Fernández-Martínez & García-Gonzalo, 2009, 2012; García-Gonzalo et al., 2014). All are linearly isomorphic for the same delay; they differ in convergence rate and exploration. CC-PSO and PSO converge fastest; CP-PSO and PP-PSO are the most exploratory and the recommended samplers; RR-PSO performs best on benchmarks with its good parameters on the line \(\phi = 3(\omega - 3/2)\).
  • Stability regions replace tuning. The mean trajectory is stable for \(|\omega| < 1\), \(0 < \bar\phi < 2(\omega + 1)\); the variance has its own, smaller region, whose upper border (the USL curve) is given analytically for any distribution of \((\omega, \phi_1, \phi_2)\) with finite first moments (García-Gonzalo & Fernández-Martínez, 2014). Good parameters lie just inside that border, where the variance is large and successive positions are uncorrelated: high variance and temporal uncorrelation improve exploration of ill-posed problems (2008). "Cloud" versions give every particle its own parameters drawn near the border, and its own time step, so nothing is tuned by hand. The history and the plea for uniformisation (García-Gonzalo & Fernández-Martínez, 2012; Fernández-Martínez & García-Gonzalo, 2013) argue that most published PSO variants are unnecessary once the stability analysis is understood.
  • Collapse is the failure mode. The swarm's dispersion, the median distance of the particles to their centre of gravity relative to its initial value, is monitored every iteration; when it collapses the sampling stops, exploration is raised (larger \(\Delta t\), repulsive forces), or the collapsed particles are counted as one sample. The sampled set is an approximation of the posterior, not a posterior sample: whether and when PSO performs importance sampling is stated as the open question (2013).
  • Posterior statistics by counting. From the retained samples: min–max ranges per parameter, a mean or median model with its interquartile range, percentiles and empirical CDFs of any derived quantity, and histograms that reveal multimodality (Pallero et al., 2018, on a 300-parameter gravity inversion with 40,000 forward solves and 25,429 retained models).
  • Model reduction before sampling. An isotropic search space cannot be sampled efficiently beyond about ten dimensions, so field-valued unknowns are expanded in a reduced basis (principal components of prior scenarios, or the singular vectors of a model already in the low-misfit region, plus one high-frequency term) and the swarm moves in the coefficients (Pallero et al., 2018; Fernández-Martínez, 2015).

What this changes in pinnrl.uq

The roadmap's UQ module was planned as a likelihood-based data term, deep ensembles, a Laplace approximation and calibration metrics. The equivalence-region view adds four things and sharpens two.

  1. Report regions, not points. The primary output of an identification is the set of parameter values whose misfit is within the noise tolerance, \(\mathrm{tol}^2 = (1 + \varepsilon)\, N \sigma^2\) for Gaussian noise, with the point estimate as one member. Intervals, percentiles and CDFs are derived from the set by counting. The Fisher ellipse and the Laplace Gaussian are reported next to it as the linear approximation, so the two can disagree visibly (the F02 case).
  2. A swarm sampler over the parameters. For \(p \lesssim 10\) parameters a cloud CP-PSO over \(\theta\) is the sampler: each particle's parameters drawn inside the second-order stability border, dispersion monitored, collapse handled. It needs a cheap forward map for each \(\theta\), which is the real design question for a PINN, where a forward solve is a training run. Two routes: a parametric surrogate \(u_w(x, t;\, \theta)\) with \(\theta\) as a network input, trained once over the prior range and evaluated in microseconds (the forward emulator the swarm needs), or warm-started retraining, a few hundred steps from the nearest particle's weights (the F07 profile, generalized to \(p\) dimensions and to a moving swarm). The parametric surrogate is the one to build first; the KdV pair and the Burgers viscosity are its test cases, since their regions are known to be displaced from the truth.
  3. A tolerance from the noise model, and the physics weight as regularization. The tolerance is set by \(\sigma\) (K39), and the dependence of the region on the physics weight is measured and documented, since it plays the role Tikhonov's term plays in the papers: too strong, and the region shrinks anisotropically toward what the surrogate can represent; too weak, and the data are fitted by a field that violates the equation.
  4. Reduced bases for fields. When the unknown is a coefficient field or a closure (roadmap v0.6), it is parametrized by fewer than ten modes (principal components of prior scenarios, or Karhunen–Loève modes) and the swarm samples the coefficients. This is also the bridge to the measurement-design work: the modes the data do not constrain are the ones a new sensor should.
  5. Ensembles are a swarm without dynamics. Deep ensembles over seeds are a crude sample of \(M_{\mathrm{tol}}\); the papers' dispersion diagnostic and the "collapsed particles count once" rule apply to them. K10 compares ensembles, the Laplace ellipse, the swarm sample and a reference posterior (K42) by coverage; the open question of when a swarm sample is a posterior sample is answered empirically there, by weighting retained samples with \(\exp(-\text{misfit}/2\sigma^2)\) and checking coverage against HMC.
  6. Optimizer stability from the same mathematics. The second-order (variance) stability regions of a stochastically forced damped oscillator are the same analysis that sets a safe step for a momentum optimizer under gradient noise. D01/D02 can use it to choose the learning rate and momentum from the Hessian spectrum of K03 rather than by trial.

Backlog entries K45–K48 carry these items; the roadmap's item 5 lists the swarm sampler and the parametric surrogate next to the Laplace approximation.

References

  • Pallero, J. L. G., Fernández-Muñiz, M. Z., Cernea, A., Álvarez-Machancoses, Ó., Pedruelo-González, L. M., Bonvalot, S. & Fernández-Martínez, J. L. (2018). Particle swarm optimization and uncertainty assessment in inverse problems. Entropy 20, 96. doi:10.3390/e20020096
  • Fernández-Martínez, J. L. & García-Gonzalo, E. (2008). The generalized PSO: a new door to PSO evolution. Journal of Artificial Evolution and Applications 2008, 861275. doi:10.1155/2008/861275
  • Fernández-Martínez, J. L. & García-Gonzalo, E. (2009). The PSO family: deduction, stochastic analysis and comparison. Swarm Intelligence 3, 245–273. doi:10.1007/s11721-009-0034-8
  • Fernández-Martínez, J. L. & García-Gonzalo, E. (2011). Stochastic stability analysis of the linear continuous and discrete PSO models. IEEE Transactions on Evolutionary Computation 15, 405–423. doi:10.1109/TEVC.2010.2053935
  • Fernández-Martínez, J. L. & García-Gonzalo, E. (2012). Stochastic stability and numerical analysis of two novel algorithms of the PSO family: PP-GPSO and RR-GPSO. International Journal on Artificial Intelligence Tools 21, 1240011. doi:10.1142/S0218213012400118
  • García-Gonzalo, E. & Fernández-Martínez, J. L. (2012). A brief historical review of particle swarm optimization (PSO). Journal of Bioinformatics and Intelligent Control 1, 3–16. doi:10.1166/jbic.2012.1002
  • Fernández-Martínez, J. L. & García-Gonzalo, E. (2013). Particle swarm optimisation: time for uniformisation. International Journal of Computing Science and Mathematics 4, 16–33. doi:10.1504/IJCSM.2013.054671
  • García-Gonzalo, E., Fernández-Martínez, J. L. & Cernea, A. (2014). Four-points particle swarm optimization algorithms. Journal of Multiple-Valued Logic and Soft Computing 22, 239–266.
  • García-Gonzalo, E. & Fernández-Martínez, J. L. (2014). Convergence and stochastic stability analysis of particle swarm optimization variants with generic parameter distributions. Applied Mathematics and Computation 249, 286–302. doi:10.1016/j.amc.2014.10.066
  • Fernández-Martínez, J. L., Pallero, J. L. G., Fernández-Muñiz, Z. & Pedruelo-González, L. M. (2014a, b). The effect of noise and Tikhonov's regularization in inverse problems. Parts I and II. Journal of Applied Geophysics 108, 176–185 and 186–193. (Cited through Pallero et al., 2018.)
  • Fernández-Martínez, J. L. (2015). Model reduction and uncertainty analysis in inverse problems. The Leading Edge 34, 1006–1016. (Cited through Pallero et al., 2018.)