0
Research Papers

Robust Resource Allocation for Calibration and Validation Tests

[+] Author and Article Information
Chenzhao Li, Sankaran Mahadevan

Department of Civil and
Environmental Engineering,
Vanderbilt University,
Nashville, TN 37235

Manuscript received July 25, 2016; final manuscript received July 15, 2017; published online August 1, 2017. Editor: Ashley F. Emery.

J. Verif. Valid. Uncert 2(2), 021004 (Aug 01, 2017) (15 pages) Paper No: VVUQ-16-1021; doi: 10.1115/1.4037313 History: Received July 25, 2016; Revised July 15, 2017

Model calibration and validation are two activities in system model development, and both of them make use of test data. Limited testing budget creates the challenge of test resource allocation, i.e., how to optimize the number of calibration and validation tests to be conducted. Test resource allocation is conducted before any actual test is performed, and therefore needs to use synthetic data. This paper develops a test resource allocation methodology to make the system response prediction “robust” to test outcome, i.e., insensitive to the variability in test outcome; therefore, consistent system response predictions can be achieved under different test outcomes. This paper analyzes the uncertainty sources in the generation of synthetic data regarding different test conditions, and concludes that the robustness objective can be achieved if the contribution of model parameter uncertainty in the synthetic data can be maximized. Global sensitivity analysis (Sobol’ index) is used to assess this contribution, and to formulate an optimization problem to achieve the desired consistent system response prediction. A simulated annealing algorithm is applied to solve this optimization problem. The proposed method is suitable either when only model calibration tests are considered or when both calibration and validation tests are considered. Two numerical examples are provided to demonstrate the proposed approach.

Copyright © 2017 by ASME
Your Session has timed out. Please sign back in to continue.

References

Kennedy, M. C. , and O'Hagan, A. , 2001, “ Bayesian Calibration of Computer Models,” J. R. Stat. Soc., 63(3), pp. 425–464. [CrossRef]
Conti, S. , and O'Hagan, A. , 2010, “ Bayesian Emulation of Complex Multi-Output and Dynamic Computer Models,” J. Stat. Plann. Inference, 140(3), pp. 640–651. [CrossRef]
Ainsworth, M. , and Oden, J. T. T. , 1997, “ A Posteriori Error Estimation in Finite Element Analysis,” Comput. Methods Appl. Mech. Eng., 142(1–2), pp. 1–88. [CrossRef]
Sankararaman, S. , McLemore, K. , Mahadevan, S. , Bradford, S. C. , and Peterson, L. D. , 2013, “ Test Resource Allocation in Hierarchical Systems Using Bayesian Networks,” AIAA J., 51(3), pp. 537–550. [CrossRef]
Li, C. , and Mahadevan, S. , 2014, “ Uncertainty Quantification and Output Prediction in Multi-Level Problems,” AIAA Paper No. 2014-0124.
Li, C. , and Mahadevan, S. , 2016, “ Role of Calibration, Validation, and Relevance in Multi-Level Uncertainty Integration,” Reliab. Eng. Syst. Saf., 148, pp. 32–43. [CrossRef]
Urbina, A. , 2009, “ Uncertainty Quantification and Decision Making in Hierarchical Development of Computational Models,” Ph.D. dissertation, Vanderbilt University, Nashville, TN.
Vanlier, J. , Tiemann, C. A. , Hilbers, P. A. J. , and van Riel, N. A. W. , 2012, “ A Bayesian Approach to Targeted Experiment Design,” Bioinformatics, 28(8), pp. 1136–1142. [CrossRef] [PubMed]
Coles, D. , and Prange, M. , 2012, “ Toward Efficient Computation of the Expected Relative Entropy for Nonlinear Experimental Design,” Inverse Probl., 28(5), p. 55019. [CrossRef]
Sebastiani, P. , and Wynn, H. P. , 2000, “ Maximum Entropy Sampling and Optimal Bayesian Experimental Design,” J. R. Stat. Soc., 62(1), pp. 145–157. [CrossRef]
Terejanu, G. , Upadhyay, R. R. , and Miki, K. , 2012, “ Bayesian Experimental Design for the Active Nitridation of Graphite by Atomic Nitrogen,” Exp. Therm. Fluid Sci., 36, pp. 178–193. [CrossRef]
Mullins, J. , Li, C. , Mahadevan, S. , and Urbina, A. , 2014, “ Optimal Selection of Calibration and Validation Test Samples Under Uncertainty,” 32nd IMAC, A Conference and Exposition on Structural Dynamics, Orlando, FL, Feb. 3–6, pp. 391–401.
Saltelli, A. , Ratto, M. , Andres, T. , Campolongo, F. , Cariboni, J. , Gatelli, D. , Saisana, M. , and Tarantola, S. , 2008, Global Sensitivity Analysis: The Primer, Wiley, Chichester, UK.
Hu, Z. , and Mahadevan, S. , 2016, “ Global Sensitivity Analysis-Enhanced Surrogate (GSAS) Modeling for Reliability Analysis,” Struct. Multidiscip. Optim., 53(3), pp. 501–521. [CrossRef]
Li, C. , and Mahadevan, S. , 2016, “ Relative Contributions of Aleatory and Epistemic Uncertainty Sources in Time Series Prediction,” Int. J. Fatigue, 82(Pt. 3), pp. 474–486. [CrossRef]
Marrel, A. , Iooss, B. , Laurent, B. , and Roustant, O. , 2009, “ Calculations of Sobol Indices for the Gaussian Process Metamodel,” Reliab. Eng. Syst. Saf., 94(3), pp. 742–751. [CrossRef]
Owen, A. , 2013, “ Better Estimation of Small Sobol' Sensitivity Indices,” ACM Trans. Model. Comput. Simul., 23(2), p. 11. [CrossRef]
Ginot, V. , Gaba, S. , Beaudouin, R. , Aries, F. , and Monod, H. , 2006, “ Combined Use of Local and ANOVA-Based Global Sensitivity Analyses for the Investigation of a Stochastic Dynamic Model: Application to the Case Study of an Individual-Based Model of a Fish Population,” Ecol. Modell., 193(3–4), pp. 479–491. [CrossRef]
Zhang, X. , and Pandey, M. D. , 2014, “ An Effective Approximation for Variance-Based Global Sensitivity Analysis,” Reliab. Eng. Syst. Saf., 121, pp. 164–174. [CrossRef]
Chen, W. , Jin, R. , and Sudjianto, A. , 2005, “ Analytical Variance-Based Global Sensitivity Analysis in Simulation-Based Design Under Uncertainty,” ASME J. Mech. Des., 127(5), pp. 875–886. [CrossRef]
Sobol’, I. M. , 2001, “ Global Sensitivity Indices for Nonlinear Mathematical Models and Their Monte Carlo Estimates,” Math. Comput. Simul., 55(1–3), pp. 271–280. [CrossRef]
Homma, T. , and Saltelli, A. , 1996, “ Importance Measures in Global Sensitivity Analysis of Nonlinear Models,” Reliab. Eng. Syst. Saf., 52(1), pp. 1–17. [CrossRef]
Sobol’, I. M. , and Myshetskaya, E. E. , 2008, “ Monte Carlo Estimators for Small Sensitivity Indices,” Monte Carlo Methods Appl., 13(5–6), pp. 455–465.
Saltelli, A. , Tarantola, S. , and Chan, K. , 1999, “ A Quantitative Model-Independent Method for Global Sensitivity Analysis of Model Output,” Technometrics, 41(1), pp. 39–56. [CrossRef]
Tarantola, S. , and Koda, M. , 2010, “ Improving Random Balance Designs for the Estimation of First Order Sensitivity Indices,” Procedia: Soc. Behav. Sci., 2(6), pp. 7753–7754. [CrossRef]
Sankararaman, S. , and Mahadevan, S. , 2013, “ Separating the Contributions of Variability and Parameter Uncertainty in Probability Distributions,” Reliab. Eng. Syst. Saf., 112, pp. 187–199. [CrossRef]
Li, C. , and Mahadevan, S. , 2015, “ Global Sensitivity Analysis for System Response Prediction Using Auxiliary Variable Method,” AIAA Paper No. 2015-0661.
Schrijver, A. , 1998, Theory of Linear and Integer Programming, Wiley, Chichester, UK.
Alrefaei, M. H. , and Andradóttir, S. , 1999, “ A Simulated Annealing Algorithm With Constant Temperature for Discrete Stochastic Optimization,” Manage. Sci., 45(5), pp. 748–764. [CrossRef]
Red-Horse, J. R. , and Paez, T. L. , 2008, “ Sandia National Laboratories Validation Workshop: Structural Dynamics Application,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2578–2584. [CrossRef]
Chopra, A. K. , 2011, Dynamics of Structures: Theory and Applications to Earthquake Engineering, Prentice Hall, Upper Saddle River, NJ.
Sankararaman, S. , and Mahadevan, S. , 2015, “ Integration of Model Verification, Validation, and Calibration for Uncertainty Quantification in Engineering Systems,” Reliab. Eng. Syst. Saf., 138, pp. 194–209. [CrossRef]
Sankararaman, S. , and Mahadevan, S. , 2013, “ Assessing the Reliability of Computational Models Under Uncertainty,” AIAA Paper No. 2013-1873.
Mullins, J. , Li, C. , Sankararaman, S. , Mahadevan, S. , and Urbina, A. , 2013, “ Uncertainty Quantification Using Multi-Level Calibration and Validation Data,” AIAA Paper No. 2013-1872.

Figures

Grahic Jump Location
Fig. 1

System response prediction: nonrobust to robust

Grahic Jump Location
Fig. 2

Synthetic data: single type of test and single specimen

Grahic Jump Location
Fig. 3

Synthetic data: q types of tests and single specimen for each type

Grahic Jump Location
Fig. 4

Simulated annealing algorithm

Grahic Jump Location
Fig. 5

System response prediction after model calibration with test data

Grahic Jump Location
Fig. 6

Optimization of the mathematical example based on Eq. (16): (a) history of accepted random walks and (b) history of the Sobol’ indices sum SθE(Y)+SθV(Y)

Grahic Jump Location
Fig. 7

Family of system response prediction PDFs at the solution of Eq. (16) of (N1,N2)=(2,8): (a) θ1=4.9,θ2=9.5, (b) θ1=5.4,θ2=9.8, and (c) θ1=5.0,θ2=10.5

Grahic Jump Location
Fig. 8

Family of system response prediction PDFs at the suboptimal solution of (N1,N2)=(1,12): (a)θ1=4.9,θ2=9.5, (b) θ1=5.4,θ2=9.8, and (c) θ1=5.0,θ2=10.5

Grahic Jump Location
Fig. 9

Optimization of the mathematical example based on Eq. (17): (a) history of accepted random walks and (b) history of cost 4N1+N2

Grahic Jump Location
Fig. 10

Family of system response prediction PDFs at the solution of Eq. (17) of (N1,N2)=(3,7): (a) θ1=5.7,θ2=10.5, (b) θ1=5.2,θ2=9.1, and (c) θ1=4.6,θ2=10.8

Grahic Jump Location
Fig. 11

Optimization of the mathematical example based on Eq. (18): (a) history of accepted random walks and (b) history of the Sobol' indices sum SθE(Y)+SθV(Y)

Grahic Jump Location
Fig. 12

Family of system response prediction PDFs at the solution of Eq. (18) of (N1,N2)=(5,13): (a) Pθ={4.2,0.9,8.3,1.1}, (b) Pθ={5.8,0.4,9.1,0.9}, and (c) Pθ={4.7,0.6,9.6,1.2}

Grahic Jump Location
Fig. 13

Family of system response prediction PDFs at the suboptimal solution of (N1,N2)=(4,17): (a) Pθ={4.2,0.9,8.3,1.1}, (b) Pθ={5.8,0.4,9.1,0.9}, and (c) Pθ={4.7,0.6,9.6,1.2}

Grahic Jump Location
Fig. 14

Optimization of the mathematical example based on Eq. (19): (a) history of accepted random walks and (b) history of cost 4N1+N2

Grahic Jump Location
Fig. 15

Family of system response prediction PDFs at the solution of Eq. (19) of (N1,N2)=(5,10): (a)Pθ={4.7,0.3,8.1,1.0}, (b) Pθ={5.7,0.9,9.0,0.4}, and (c) Pθ={4.9,0.5,9.4,1.2}

Grahic Jump Location
Fig. 16

Structural dynamics challenge problem [30]: (a) level 1 (for testing), (b) level 2 (for testing), and (c) level 3 (for system prediction)

Grahic Jump Location
Fig. 17

Optimization of the multilevel problem based on Eq. (21)

Grahic Jump Location
Fig. 18

Family of system response prediction PDFs at (N1,N2,N3,N4)=(11,9,6,2): (a) k={5600,10433,8638}, (b) k={4483,9112,9987}, and (c) k={5776,9812,9393}

Grahic Jump Location
Fig. 19

Optimization of the multilevel problem based on Eq. (22)

Grahic Jump Location
Fig. 20

Family of system response prediction PDFs at (N1,N2,N3,N4)=(11,10,6,3): (a) k={4492,11183,9116}, (b) k={5074,8760,7812}, and (c) k={5276,9883,9518}

Tables

Errata

Discussions

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In