Research Papers

New Metrics for Validation of Data-Driven Random Process Models in Uncertainty Quantification

[+] Author and Article Information
Hongyi Xu, Zhen Jiang

Department of Mechanical Engineering,
Northwestern University,
Evanston, IL 60208

Daniel W. Apley

Department of Industrial Engineering and
Management Science,
Northwestern University,
Evanston, IL 60208

Wei Chen

Department of Mechanical Engineering,
Northwestern University,
Evanston, IL 60208
e-mail: weichen@northwestern.edu

1Corresponding author.

Manuscript received December 9, 2014; final manuscript received September 10, 2015; published online December 10, 2015. Assoc. Editor: Kevin Dowding.

J. Verif. Valid. Uncert 1(2), 021002 (Dec 10, 2015) (14 pages) Paper No: VVUQ-14-1007; doi: 10.1115/1.4031813 History: Received December 09, 2014; Revised September 10, 2015

Data-driven random process models have become increasingly important for uncertainty quantification (UQ) in science and engineering applications, due to their merit of capturing both the marginal distributions and the correlations of high-dimensional responses. However, the choice of a random process model is neither unique nor straightforward. To quantitatively validate the accuracy of random process UQ models, new metrics are needed to measure their capability in capturing the statistical information of high-dimensional data collected from simulations or experimental tests. In this work, two goodness-of-fit (GOF) metrics, namely, a statistical moment-based metric (SMM) and an M-margin U-pooling metric (MUPM), are proposed for comparing different stochastic models, taking into account their capabilities of capturing the marginal distributions and the correlations in spatial/temporal domains. This work demonstrates the effectiveness of the two proposed metrics by comparing the accuracies of four random process models (Gaussian process (GP), Gaussian copula, Hermite polynomial chaos expansion (PCE), and Karhunen–Loeve (K–L) expansion) in multiple numerical examples and an engineering example of stochastic analysis of microstructural materials properties. In addition to the new metrics, this paper provides insights into the pros and cons of various data-driven random process models in UQ.

Copyright © 2016 by ASME
Your Session has timed out. Please sign back in to continue.


Xiong, F. , Yin, X. , Chen, W. , and Yang, S. , 2010, “ Enhanced Probabilistic Analytical Target Cascading With Application to Multi-Scale Design,” Eng. Optim., 42(6), pp. 581–592. [CrossRef]
Tamura, Y. , Suganuma, S. , Kikuchi, H. , and Hibi, K. , 1999, “ Proper Orthogonal Decomposition of Random Wind Pressure Field,” J. Fluids Struct., 13(7–8), pp. 1069–1095. [CrossRef]
Yin, X. L. , Lee, S. , Chen, W. , Liu, W. K. , and Horstemeyer, M. F. , 2009, “ Efficient Random Field Uncertainty Propagation in Design Using Multiscale Analysis,” ASME J. Mech. Des., 131(2), p. 021006. [CrossRef]
Missoum, S. , 2008, “ Probabilistic Optimal Design in the Presence of Random Fields,” Struct. Multidiscip. Optim., 35(6), pp. 523–530. [CrossRef]
Xi, Z. M. , Youn, B. D. , and Hu, C. , 2010, “ Effective Random Field Characterization Considering Statistical Dependence for Probability Analysis and Design,” ASME Paper No. DETC2010-29183.
Greene, M. S. , Xu, H. , Tang, S. , Chen, W. , and Liu, W. K. , 2013, “ A Generalized Uncertainty Propagation Criterion From Benchmark Studies of Microstructured Material Systems,” Comput. Methods Appl. Mech. Eng., 254, pp. 271–291. [CrossRef]
Xu, H. Y. , Greene, M. S. , Deng, H. , Dikin, D. , Brinson, C. , Liu, W. K. , Burkhart, C. , Papakonstantopoulos, G. , Poldneff, M. , and Chen, W. , 2013, “ Stochastic Reassembly Strategy for Managing Information Complexity in Heterogeneous Materials Analysis and Design,” ASME J. Mech. Des., 135(10), p. 101010. [CrossRef]
Guilleminot, J. , and Soize, C. , 2011, “ Non ‐ Gaussian Positive ‐ Definite Matrix ‐ Valued Random Fields With Constrained Eigenvalues: Application to Random Elasticity Tensors With Uncertain Material Symmetries,” Int. J. Numer. Methods Eng., 88(11), pp. 1128–1151. [CrossRef]
Greene, M. S. , Liu, Y. , Chen, W. , and Liu, W. K. , 2011, “ Computational Uncertainty Analysis in Multiresolution Materials Via Stochastic Constitutive Theory,” Comput. Methods Appl. Mech. Eng., 200(1–4), pp. 309–325. [CrossRef]
Allaix, D. L. , and Carbone, V. I. , 2013, “ Karhunen–Loève Decomposition of Random Fields Based on a Hierarchical Matrix Approach,” Int. J. Numer. Methods Eng., 94(11), pp. 1015–1036. [CrossRef]
Allen, M. , Raulli, M. , Maute, K. , and Frangopol, D. M. , 2004, “ Reliability-Based Analysis and Design Optimization of Electrostatically Actuated MEMS,” Comput. Struct., 82(13–14), pp. 1007–1020. [CrossRef]
Ditlevsen, O. , 1988, “ Stochastic Model of Self-Weight Load,” J. Struct. Eng., 114(1), pp. 222–230. [CrossRef]
Nayak, P. R. , 1971, “ Random Process Model of Rough Surfaces,” ASME J. Tribol., 93(3), pp. 398–407.
Noh, Y. , Choi, K. K. , and Du, L. , 2009, “ Reliability-Based Design Optimization of Problems With Correlated Input Variables Using a Gaussian Copula,” Struct. Multidiscip. Optim., 38(1), pp. 1–16. [CrossRef]
Xi, Z. , and Wang, P. , 2012, “ A Copula Based Sampling Method for Residual Life Prediction of Engineering Systems Under Uncertainty,” ASME Paper No. DETC2012-71105.
Zavala, V. M. , Constantinescu, E. M. , Krause, T. , and Anitescu, M. , 2009, “ On-Line Economic Optimization of Energy Systems Using Weather Forecast Information,” J. Process Control, 19(10), pp. 1725–1736. [CrossRef]
Wiener, N. , 1938, “ The Homogeneous Chaos,” Am. J. Math., 60(4), pp. 897–936. [CrossRef]
Acharjee, S. , and Zabaras, N. , 2007, “ A Non-Intrusive Stochastic Galerkin Approach for Modeling Uncertainty Propagation in Deformation Processes,” Comput. Struct., 85(5–6), pp. 244–254. [CrossRef]
Xiu, D. B. , and Karniadakis, G. E. , 2003, “ Modeling Uncertainty in Flow Simulations Via Generalized Polynomial Chaos,” J. Comput. Phys., 187(1), pp. 137–167. [CrossRef]
Holmes, P. , Lumley, J. L. , and Berkooz, G. , 1998, Turbulence, Coherent Structures, Dynamical Systems and Symmetry, Cambridge University Press, New York.
Feeny, B. F. , and Kappagantu, R. , 1998, “ On the Physical Interpretation of Proper Orthogonal Modes in Vibrations,” J. Sound Vib., 211(4), pp. 607–616. [CrossRef]
Xi, Z. M. , Jung, B. C. , and Youn, B. D. , 2012, “ Random Field Modeling With Insufficient Data Sets for Probability Analysis,” Annual Reliability and Maintainability Symposium (RAMS), Reno, NV, Jan. 23–26.
Craig, K. , and Roux, W. , 2008, “ On the Investigation of Shell Buckling Due to Random Geometrical Imperfections Implemented Using Karhunen–Loève Expansions,” Int. J. Numer. Methods Eng., 73(12), pp. 1715–1726. [CrossRef]
Peacock, J. A. , 1983, “ Two-Dimensional Goodness-of-Fit Testing in Astronomy,” Mon. Not. R. Astron. Soc., 202(3), pp. 615–627. [CrossRef]
Fasano, G. , and Franceschini, A. , 1987, “ A Multidimensional Version of the Kolmogorov–Smirnov Test,” Mon. Not. R. Astron. Soc., 225(1), pp. 155–170. [CrossRef]
Pollard, D. , 1979, “ General Chi-Square Goodness-of-Fit Tests With Data-Dependent Cells,” Z. Wahrscheinlichkeitstheorie Verw. Geb., 50(3), pp. 317–331. [CrossRef]
Dobric, J. , and Schmid, F. , 2005, “ Testing Goodness of Fit for Parametric Families of Copulas—Application to Financial Data,” Commun. Stat.-Simul. Comput., 34(4), pp. 1053–1068. [CrossRef]
Fermanian, J. D. , 2005, “ Goodness-of-Fit Tests for Copulas,” J. Multivar. Anal., 95(1), pp. 119–152. [CrossRef]
Huard, D. , Evin, G. , and Favre, A. C. , 2006, “ Bayesian Copula Selection,” Comput. Stat. Data Anal., 51(2), pp. 809–822. [CrossRef]
Silva, R. D. , and Lopes, H. F. , 2008, “ Copula, Marginal Distributions and Model Selection: A Bayesian Note,” Stat. Comput., 18(3), pp. 313–320. [CrossRef]
Ferson, S. , and Oberkampf, W. L. , 2009, “ Validation of Imprecise Probability Models,” Int. J. Reliab. Saf., 3, pp. 3–22. [CrossRef]
Li, W. , Chen, W. , Jiang, Z. , Lu, Z. , and Liu, Y. , 2014, “ New Validation Metrics for Models With Multiple Correlated Responses,” Reliab. Eng. Syst. Saf., 127, pp. 1–11. [CrossRef]
Bishop, C. M. , 2006, Pattern Recognition and Machine Learning, Vol. 4, Springer, New York.
Arbenz, P. , 2013, “ Bayesian Copulae Distributions, With Application to Operational Risk Management—Some Comments,” Methodol. Comput. Appl. Probab., 15(1), pp. 105–108. [CrossRef]
Xiu, D. , 2010, Numerical Methods for Stochastic Computations: A Spectral Method Approach, Princeton University Press, New York.
Sakamoto, S. , and Ghanem, R. , 2002, “ Polynomial Chaos Decomposition for the Simulation of Non-Gaussian Nonstationary Stochastic Processes,” J. Eng. Mech., 128(2), pp. 190–201. [CrossRef]
Stark, H. , and Woods, J. W. , 1986, Probability, Random Processes, and Estimation Theory for Engineers, Vol. 1, Prentice Hall, Englewood Cliffs, NJ.
Ghanem, R. G. , and Spanos, P. D. , 2003, Stochastic Finite Elements: A Spectral Approach, Courier Corporation, New York.
Huang, S. , Quek, S. , and Phoon, K. , 2001, “ Convergence Study of the Truncated Karhunen–Loeve Expansion for Simulation of Stochastic Processes,” Int. J. Numer. Methods Eng., 52(9), pp. 1029–1043. [CrossRef]
Wang, S. , 1998, “ Aggregation of Correlated Risk Portfolios: Models and Algorithms,” Casualty Actuarial Society.
Liu, Y. , Greene, M. S. , Chen, W. , Dikin, D. A. , and Liu, W. K. , 2013, “ Computational Microstructure Characterization and Reconstruction for Stochastic Multiscale Material Design,” Comput.-Aided Des., 45(1), pp. 65–76. [CrossRef]
Xu, H. , Dikin, D. , Burkhart, C. , and Chen, W. , 2014, “ Descriptor-Based Methodology for Statistical Characterization and 3D Reconstruction for Polymer Nanocomposites,” Comput. Mater. Sci., 85, pp. 206–216. [CrossRef]
Hill, R. , 1963, “ Elastic Properties of Reinforced Solids—Some Theoretical Principles,” J. Mech. Phys. Solids, 11(5), pp. 357–372. [CrossRef]
Xu, H. , Li, Y. , Brinson, C. , and Chen, W. , 2014, “ A Descriptor-Based Design Methodology for Developing Heterogeneous Microstructural Materials System,” ASME J. Mech. Des., 136(5), p. 051007. [CrossRef]


Grahic Jump Location
Fig. 3

Illustration of u-pooling method. Right: u-values at three validation locations; left: area metric of mismatch between the empirical complementary CDF of u-values and the standard uniform distribution.

Grahic Jump Location
Fig. 2

Flowchart of testing the GOF of a random process model

Grahic Jump Location
Fig. 1

Random process model of high-dimensional data and its random realizations. This figure shows two examples of random process modeling. Left: spatial distribution of elastic modulus is modeled as a random process with respect to spatial location x. Right: stochastic constitutive law of microstructural materials, where the stress is modeled as a random process with respect to strain.

Grahic Jump Location
Fig. 4

SMM and MUPM's performances: (a) detection of the marginal skewness of random process data and (b) necessity of including high-order metric values. For illustration purpose, 50 out of 5000 samples of each dataset are plotted. Three orders of statistical information (M = 1, 2, 3) are compared.

Grahic Jump Location
Fig. 8

Visual comparison and covariance RMSE comparison of original NSta-NSym data and realizations by five different random process models. For visual comparison, 20 realizations are plotted in each figure. For covariance matrix comparison, RMSE between the original and the regenerated covariance matrices is plotted in the bar chart.

Grahic Jump Location
Fig. 5

SMM and MUPM of five random process models in benchmark 2 (NSta-Sym): SMM on the left is low-order-dominant; MUPM on the right is high-order-dominant. Low metric value means high accuracy.

Grahic Jump Location
Fig. 6

Visual comparison and covariance matrix RMSE comparison of original NSta-Sym data and realizations by five different random process models. For visual comparison, 20 realizations are plotted in each figure. For covariance matrix comparison, RMSE between the original and the regenerated covariance matrices is plotted in the bar chart.

Grahic Jump Location
Fig. 7

SMM and MUPM of five random process models in benchmark 4 (NSta-NSym): SMM on the left is low-order-dominant; MUPM on the right is high-order-dominant. Low metric value means high accuracy.

Grahic Jump Location
Fig. 9

Benchmark 4 with ten original samples: (a) original data, 20 randomly selected realizations for each random process model and covariance matrix RMSEs and (b) comparisons of SMM and MUPM

Grahic Jump Location
Fig. 11

Comparison of original data and random process model realizations of G′, classes 2 and 8. The solid lines are the mean and the dashed lines mark the 5th and 95th percentile at each x location.

Grahic Jump Location
Fig. 10

A stochastic decomposition and reassembly strategy for material property analysis

Grahic Jump Location
Fig. 12

Comparison of simulated mosaic RVE properties using: (1) original SVE properties data, (2) regenerated SVE properties data using GP, (3) regenerated SVE properties data using copula, and (4) regenerated SVE properties data using K–L. In the table, we compare the percent errors on L, P, and H, as well as the SSE of the three model predictions.

Grahic Jump Location
Fig. 13

SMM and MUPM comparisons of benchmarks 1 and 3 (Sta-NSym)



Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In