Research Papers

Model Validation of Functional Responses Across Experimental Regions Using Functional Regression Extensions to the CORA Objective Rating System

[+] Author and Article Information
Scott M. Storm

Department of Operational Sciences,
Air Force Institute of Technology,
Wright-Patterson AFB, OH 45433
e-mail: scott.storm@us.af.mil

Raymond R. Hill

Department of Operational Sciences,
Air Force Institute of Technology,
Wright-Patterson AFB, OH 45433
e-mail: raymond.hill@afit.edu

Joseph J. Pignatiello

Department of Operational Sciences,
Air Force Institute of Technology,
Wright-Patterson AFB, OH 45433
e-mail: joseph.pignatiello@afit.edu

G. Geoffrey Vining

Department of Statistics,
Virginia Tech,
Blacksburg, VA 24061
e-mail: vining@vt.edu

Edward D. White

Department of Mathematics and Statistics,
Air Force Institute of Technology,
Wright-Patterson AFB, OH 45433
e-mail: edward.white@afit.edu

Manuscript received August 14, 2017; final manuscript received February 1, 2018; published online March 5, 2018. Assoc. Editor: Jeffrey E. Bischoff.

J. Verif. Valid. Uncert 2(4), 041004 (Mar 05, 2018) (9 pages) Paper No: VVUQ-17-1029; doi: 10.1115/1.4039303 History: Received August 14, 2017; Revised February 01, 2018

As we continue to model more complex systems, the validation of dynamical responses has come to the forefront of modeling and simulation. One form of dynamic response is when the output is a function of time. The proper evaluation of functional data over an array of desired input parameters is critical to achieving a robust validation assessment of a simulation model. We extend the correlation analysis (CORA) objective rating system to validate functional data across experimental regions. Functional regression analysis is used to generate surrogate estimations of the system response functions at points within the region where experimental observations are absent. These CORA scores provide a measure of disagreement at each desired parameter configuration. An overall score for model validity is achieved using a weighted linear combination of the individual CORA scores. Finally, an improved CORA size scoring metric is introduced.

Copyright © 2017 by American Society of Mechanical Engineers
Your Session has timed out. Please sign back in to continue.


ISO, 2014, “Road Vehicles—Objective Rating Metric for Non-Ambiguous Signals,” International Organization for Standardization, Geneva, Switzerland, Standard No. ISO/TS 18571:2014. https://www.iso.org/standard/62937.html
Storm, S. , Hill, R. R. , and Pignatiello, J. J. , 2013, “ A Response Surface Methodology for Modeling Time Series Response Data,” Qual. Reliab. Eng. Int., 29(5), pp. 771–778. [CrossRef]
Balci, O. , 1994, “ Validation, Verification, and Testing Techniques Throughout the Life Cycle of a Simulation Study,” Ann. Oper. Res., 53(1), pp. 121–173. [CrossRef]
Davis, P. K. , 1992, “Generalizing Concepts and Methods of Verification, Validation, and Accreditation (VV&A) for Military Simulations,” RAND Corporation, Santa Monica, CA, Technical Report No. RAND/R-4249-ACQ. https://www.rand.org/pubs/reports/R4249.html
Sargent, R. G. , 2010, “ Verification and Validation of Simulation Models,” IEEE Winter Simulation Conference, Washington, DC, Dec. 9–12, pp. 166–183.
Law, A. M. , 2007, Simulation Modeling and Analysis, 4th ed., McGraw-Hill, New York.
Shannon, R. E. , 1975, Systems Simulation: The Art and Science, Prentice Hall, Englewood Cliffs, NJ.
Balci, O. , and Sargent, R. G. , 1981, “ A Methodology for Cost-Risk Analysis in the Statistical Validation of Simulation Models,” Commun. ACM, 24(4), pp. 190–197. [CrossRef]
Naylor, T. H. , and Finger, J. M. , 1967, “ Verification of Computer Simulation Models,” Manage. Sci., 14(2), pp. B92–B101. [CrossRef]
Garratt, M. , 1974, “ Statistical Validation of Simulation Models,” Summer Computer Simulation Conference, Houston, TX, pp. 915–926.
Fishman, G. S. , and Kiviat, P. J. , 1967, “ The Analysis of Simulation-Generated Time Series,” Manage. Sci., 13(7), pp. 525–557. [CrossRef]
Gallant, A. , Gerig, T. M. , and Evans, J. , 1974, “ Time Series Realizations Obtained According to an Experimental Design,” J. Am. Stat. Assoc., 69(347), pp. 639–645. [CrossRef]
Sarin, H. , Kokkolaras, M. , Hulbert, G. , Papalambros, P. , Barbat, S. , and Yang, R.-J. , 2010, “ Comparing Time Histories for Validation of Simulation Models: Error Measures and Metrics,” ASME J. Dyn. Syst. Meas. Control, 132(6), p. 061401. [CrossRef]
Van Horn, R. L. , 1971, “ Validation of Simulation Results,” Manage. Sci., 17(5), pp. 247–258. [CrossRef]
Balci, O. , 1998, “ Verification, Validation, and Testing,” Handbook of Simulation, Vol. 10, J. Banks , ed., Wiley, New York, pp. 335–393. [CrossRef]
Hills, R. G. , and Trucano, T. G. , 1999, “Statistical Validation of Engineering and Scientific Models: Background,” Sandia National Laboratories, Albuquerque, NM, Report No. SAND99-1256. https://inis.iaea.org/search/search.aspx?orig_q=RN:30047122
Buranathiti, T. , Cao, J. , Chen, W. , Baghdasaryan, L. , and Xia, Z. C. , 2006, “ Approaches for Model Validation: Methodology and Illustration on a Sheet Metal Flanging Process,” ASME J. Manuf. Sci. Eng., 128(2), pp. 588–597. [CrossRef]
Chen, W. , Baghdasaryan, L. , Buranathiti, T. , and Cao, J. , 2004, “ Model Validation Via Uncertainty Propagation and Data Transformations,” AIAA J., 42(7), pp. 1406–1415. [CrossRef]
Ghanem, R. G. , Doostan, A. , and Red-Horse, J. , 2008, “ A Probabilistic Construction of Model Validation,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2585–2595. [CrossRef]
Hills, R. G. , 2006, “ Model Validation: Model Parameter and Measurement Uncertainty,” ASME J. Heat Transfer, 128(4), pp. 339–351. [CrossRef]
Rebba, R. , and Mahadevan, S. , 2008, “ Computational Methods for Model Reliability Assessment,” Reliab. Eng. Syst. Saf., 93(8), pp. 1197–1207. [CrossRef]
Oberkampf, W. L. , and Barone, M. F. , 2006, “ Measures of Agreement Between Computation and Experiment: Validation Metrics,” J. Comput. Phys., 217(1), pp. 5–36. [CrossRef]
Ferson, S. , Oberkampf, W. L. , and Ginzburg, L. , 2008, “ Model Validation and Predictive Capability for the Thermal Challenge Problem,” Comput. Methods Appl. Mech. Eng., 197(29–32), pp. 2408–2430. [CrossRef]
Chen, W. , Xiong, Y. , Tsui, K.-L. , and Wang, S. , 2008, “ A Design-Driven Validation Approach Using Bayesian Prediction Models,” ASME J. Mech. Des., 130(2), p. 021101. [CrossRef]
Jiang, X. , and Mahadevan, S. , 2008, “ Bayesian Wavelet Method for Multivariate Model Assessment of Dynamic Systems,” J. Sound Vib., 312(4–5), pp. 694–712. [CrossRef]
Jiang, X. , and Mahadevan, S. , 2008, “ Bayesian Validation Assessment of Multivariate Computational Models,” J. Appl. Stat., 35(1), pp. 49–65. [CrossRef]
Rebba, R. , Huang, S. , Liu, Y. , and Mahadevan, S. , 2005, “ Statistical Validation of Simulation Models,” Int. J. Mater. Prod. Technol., 25(1–3), pp. 164–181.
Rebba, R. , and Mahadevan, S. , 2006, “ Model Predictive Capability Assessment Under Uncertainty,” AIAA J., 44(10), pp. 2376–2384. [CrossRef]
Rebba, R. , Mahadevan, S. , and Huang, S. , 2006, “ Validation and Error Estimation of Computational Models,” Reliab. Eng. Syst. Saf., 91(10–11), pp. 1390–1397. [CrossRef]
Rebba, R. , and Mahadevan, S. , 2006, “ Validation of Models With Multivariate Output,” Reliab. Eng. Syst. Saf., 91(8), pp. 861–871. [CrossRef]
Wang, S. , Chen, W. , and Tsui, K.-L. , 2009, “ Bayesian Validation of Computer Models,” Technometrics, 51(4), pp. 439–451. [CrossRef]
Kleijnen, J. P. , 1992, “ Regression Metamodels for Simulation With Common Random Numbers: Comparison of Validation Tests and Confidence Intervals,” Manage. Sci., 38(8), pp. 1164–1185. [CrossRef]
Kleijnen, J. P. , 1995, “ Verification and Validation of Simulation Models,” Eur. J. Oper. Res., 82(1), pp. 145–162. [CrossRef]
Kleijnen, J. P. , 1995, “ Sensitivity Analysis and Optimization of System Dynamics Models: Regression Analysis and Statistical Design of Experiments,” Syst. Dyn. Rev., 11(4), pp. 275–288. [CrossRef]
Kleijnen, J. P. , Feelders, A. J. , and Cheng, R. C. , 1998, “ Bootstrapping and Validation of Metamodels in Simulation,” IEEE Winter Simulation Conference, Washington, DC, Dec. 13–16, pp. 701–706.
Kleijnen, J. P. , and Sargent, R. G. , 2000, “ A Methodology for Fitting and Validating Metamodels in Simulation,” Eur. J. Oper. Res., 120(1), pp. 14–29. [CrossRef]
Kleijnen, J. P. , and Deflandre, D. , 2006, “ Validation of Regression Metamodels in Simulation: Bootstrap Approach,” Eur. J. Oper. Res., 170(1), pp. 120–131. [CrossRef]
Bendat, J. S. , and Piersol, A. G. , 1980, Engineering Applications of Correlation and Spectral Analysis, Vol. 315, Wiley-Interscience, New York, p. 1.
Jenkins, G. M. , 1961, “ General Considerations in the Analysis of Spectra,” Technometrics, 3(2), pp. 133–166. [CrossRef]
Donelly, B. , Morgan, R. M. , and Eppinger, R. H. , 1983, “ Durability, Repeatability and Reproducibility of the NHTSA Side Impact Dummy,” SAE Paper No. 831624.
Russell, D. M. , 1997, “ Error Measures for Comparing Transient Data—Part I: Development of a Comprehensive Error Measure,” 68th Shock and Vibration Symposium, Hunt Valley, MD, Nov. 3–6, pp. 175–184. http://roadsafellc.com/NCHRP22-24/Literature/Papers/Metrics/Russell-ErrorMeasures.pdf
Sprague, M. , and Geers, T. , 2004, “ A Spectral-Element Method for Modeling Cavitation in Transient Fluid-Structure Interaction,” Int. J. Numer. Methods Eng., 60(15), pp. 2467–2499. [CrossRef]
Gehre, C. , Gades, H. , and Wernicke, P. , 2009, “ Objective Rating of Signals Using Test and Simulation Responses,” 21st International Technical Conference on the Enhanced Safety of Vehicles Conference (ESV), Stuttgart, Germany, June 15–18, pp. 15–18.
Zhan, Z. , Fu, Y. , and Yang, R.-J. , 2011, “Enhanced Error Assessment of Response Time Histories (EEARTH) Metric and Calibration Process,” SAE Paper No. 2011-01-0245.
Jiang, X. , Yang, R.-J. , Barbat, S. , and Weerappuli, P. , 2009, “ Bayesian Probabilistic PCA Approach for Model Validation of Dynamic Systems,” SAE Int. J. Mater. Manuf., 2(1), pp. 555–563. [CrossRef]
Lamb, D. , Castanier, M. , Pan, H. , Kokkolaras, M. , and Hulbert, G. , 2012, “Model Validation for Simulations of Vehicle Systems,” Ford Motor Company, Dearborn, MI, DTIC Document No. A566158. http://www.dtic.mil/docs/citations/ADA566037
Zhan, Z. , Fu, Y. , Yang, R.-J. , and Peng, Y. , 2011, “ An Enhanced Bayesian Based Model Validation Method for Dynamic Systems,” ASME J. Mech. Des., 133(4), p. 041005. [CrossRef]
Jiang, X. , and Mahadevan, S. , 2011, “ Wavelet Spectrum Analysis Approach to Model Validation of Dynamic Systems,” Mech. Syst. Signal Process., 25(2), pp. 575–590. [CrossRef]
Cheng, Z. , Pellettiere, J. A. , and Wright, N. L. , 2006, “ Wavelet-Based Test-Simulation Correlation Analysis for the Validation of Biodynamical Modeling,” Conference and Exposition on Structural Dynamics, St Louis, MO, Jan. 30–Feb. 2, pp. 2124–2132. https://pdfs.semanticscholar.org/ef0c/bc3e2f9f09e0a87a6414ffe36a4a93216f60.pdf
Morris, J. S. , 2015, “ Functional Regression,” Annu. Rev. Stat. Appl., 2(1), pp. 321–359. [CrossRef]
Faraway, J. J. , 1997, “ Regression Analysis for a Functional Response,” Technometrics, 39(3), pp. 254–261. [CrossRef]
Faraway, J. J. , 2000, “Modeling Reach Motions Using Functional Regression Analysis,” SAE Paper No. 2000-01-2175.
Untaroiu, C. , Shin, J. , and Lu, Y.-C. , 2013, “ Assessment of a Dummy Model in Crash Simulations Using Rating Methods,” Int. J. Autom. Technol., 14(3), pp. 395–405. [CrossRef]
Thunert, C. , 2012, “Cora Release 3.6 User's Manual,” Partnership for Dummy Technology and Biomechanics, Ingolstadt, Germany.
Montgomery, D. C. , 2009, Design and Analysis of Experiments, 7th ed., Wiley, Hoboken, NJ.
Myers, R. H. , Montgomery, D. C. , and Anderson-Cook, C. M. , 2009, Response Surface Methodology: Process and Product Optimization Using Designed Experiments, 3rd ed., Wiley, Hoboken, NJ.
Gehre, C. , and Stahlschmidt, S. , 2011, “ Assessment of Dummy Models by Using Objective Rating Methods,” 22nd International Technical Conference on the Enhanced Safety of Vehicles, Washington, DC, June 13–16.


Grahic Jump Location
Fig. 1

Structure for CORA objective rating system [43]

Grahic Jump Location
Fig. 2

Evaluation with constant corridor widths [43]

Grahic Jump Location
Fig. 3

Extended CORA methodology for model validation over experimental regions

Grahic Jump Location
Fig. 4

Notional example of using CORA ratings to isolate poor model performance within experimental regions. Open circles represent estimated points.

Grahic Jump Location
Fig. 5

Different model and system responses with equal area

Grahic Jump Location
Fig. 6

Model and system responses with calculated size and area scores




Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In