0
Technical Brief

Exposing System and Model Disparity and Agreement Using Wavelets

[+] Author and Article Information
Andrew D. Atkinson

Department of Operational Sciences,
Air Force Institute of Technology,
Wright-Patterson AFB, OH 45433
e-mail: andrew.atkinson@afit.edu

Raymond R. Hill

Professor
Department of Operational Sciences,
Air Force Institute of Technology,
Wright-Patterson AFB, OH 45433
e-mail: raymond.hill@afit.edu

Joseph J. Pignatiello, Jr.

Professor
Department of Operational Sciences,
Air Force Institute of Technology,
Wright-Patterson AFB, OH 45433
e-mail: joseph.pignatiello@afit.edu

G. Geoffrey Vining

Professor
Department of Statistics,
Virginia Tech Blacksburg,
Blacksburg, VA 24061
e-mail: vining@vt.edu

Edward D. White

Professor
Department of Mathematics and Statistics,
Air Force Institute of Technology,
Wright-Patterson AFB, OH 45433
e-mail: edward.white@afit.edu

Eric Chicken

Professor
Department of Statistics,
Florida State University Tallahassee,
Tallahassee, FL 32306
e-mail: chicken@stat.fsu.edu

Manuscript received August 14, 2017; final manuscript received July 26, 2018; published online September 17, 2018. Assoc. Editor: Kevin Dowding. This work is in part a work of the U.S. Government. ASME disclaims all interest in the U.S. Government's contributions.

J. Verif. Valid. Uncert 3(2), 024501 (Sep 17, 2018) (6 pages) Paper No: VVUQ-17-1030; doi: 10.1115/1.4041265 History: Received August 14, 2017; Revised July 26, 2018

Model verification and validation (V&V) remain a critical step in the simulation model development process. A model requires verification to ensure that it has been correctly transitioned from a conceptual form to a computerized form. A model also requires validation to substantiate the accurate representation of the system it is meant to simulate. Validation assessments are complex when the system and model both generate high-dimensional functional output. To handle this complexity, this paper reviews several wavelet-based approaches for assessing models of this type and introduces a new concept for highlighting the areas of contrast and congruity between system and model data. This concept identifies individual wavelet coefficients that correspond to the areas of discrepancy between the system and model.

FIGURES IN THIS ARTICLE
<>
Copyright © 2018 by ASME
Topics: Wavelets , Signals
Your Session has timed out. Please sign back in to continue.

References

Balci, O. , 1997, “ Verification Validation and Accreditation of Simulation Models,” Winter Simulation Conference, pp. 135–141.
Balci, O. , 2013,Encyclopedia of Operations Research and Management Science, 3rd ed., Springer, New York, pp. 1618–1627.
Atkinson, A. , Hill, R. , Pignatiello, J. , Vining, G. , White, E. , and Chicken, E. , 2017, “ Dynamic Model Validation Metric Based on Wavelet Thresholded Signals,” J. Verif., Validation Uncertainty Quantif., 2(2), p. 021002. [CrossRef]
Atkinson, A. , Hill, R. , Pignatiello, J. , Vining, G. , White, E. , and Chicken, E. , 2017, “ Wavelet Anova Approach to Model Validation,” Simul. Modell. Pract. Theory, 78, pp. 18–27. [CrossRef]
Atkinson, A. , Hill, R. , Pignatiello, J. , Vining, G. , White, E. , and Chicken, E. , 2018, “ Wavelet Anova Bisection Method for Identifying Simulation Model Bias,” Simul. Modell. Pract. Theory, 80, pp. 66–74. [CrossRef]
Law, A. M. , 2013, Simulation Modeling and Analysis, 5th ed., McGraw-Hill, New York.
Schwer, L. , Mair, H. , and Crane, R. , 2006, Guide for Verification and Validation in Computational Solid Mechanics, The American Society of Mechanical Engineers, New York.
McHale, M. , Friedman, J. , and Karian, J. , 2009, “ Standard for Verification and Validation in Computational Fluid Dynamics and Heat Transfer,” The American Society of Mechanical Engineers, New York Standard No. VV20-2009. https://www.asme.org/products/codes-standards/v-v-20-2009-standard-verification-validation
Fishman, G. S. , and Kiviat, P. J. , 1967, “ The Analysis of Simulation-Generated Time Series,” Manage. Sci., 13(7), pp. 525–557. [CrossRef]
Kleijnen, J. P. , 1995, “ Verification and Validation of Simulation Models,” Eur. J. Oper. Res., 82(1), pp. 145–162. [CrossRef]
Robinson, S. , 2004, Simulation: The Practice of Model Development and Use, 1st ed., Wiley, Chichester, UK.
Sargent, R. G. , 2013, “ Verification and Validation of Simulation Models,” J. Simul., 7(1), pp. 12–24. [CrossRef]
Shannon, R. E. , 1975, Systems Simulation: The Art and Science, 1st ed., Prentice Hall, Englewood Cliffs, NJ.
Box, G. E. , Jenkins, G. M. , and Reinsel, G. C. , 2008, Time Series Analysis: Forecasting and Control, 4th ed., Wiley, New York.
Geers, T. L. , 1984, “ An Objective Error Measure for the Comparison of Calculated and Measured Transient Response Histories,” Shock Vib. Bull., 54(2), pp. 99–108. https://archive.org/details/DTIC_ADA148132
Russell, D. M. , 1997, “ Error Measures for Comparing Transient Data—Part I: Development of a Comprehensive Error Measure,” 68th Shock and Vibration Symposium, Hunt Valley, MD, Nov. 3–7, pp. 175–184.
Sargent, R. G. , 2003, “ Verification and Validation of Simulation Models,” Winter Simulation Conference, pp. 37–48.
Ramsay, J. O. , and Silverman, B. W. , 2005, Functional Data Analysis, 2nd ed., Springer Science Business Media, New York.
McKay, J. L. , Welch, T. D. , Vidakovic, B. , and Ting, L. H. , 2013, “ Statistically Significant Contrasts Between EMG Waveforms Revealed Using Wavelet-Based Functional Anova,” J. Neurophysiol., 109(2), pp. 591–602. [CrossRef] [PubMed]
Park, B. , and Schneeberger, J. , 2003, “ Microscopic Simulation Model Calibration and Validation: Case Study of VISSIM Simulation Model for a Coordinated Actuated Signal System,” Transp. Res. Rec.: J. Transp. Res. Board, 1856, pp. 185–192. [CrossRef]
Godley, S. T. , Triggs, T. J. , and Fildes, B. N. , 2002, “ Driving Simulator Validation for Speed Research,” Accid. Anal. Prev., 34(5), pp. 589–600. [CrossRef] [PubMed]
Sarin, H. , Kokkolaras, M. , Hulbert, G. , Papalambros, P. , Barbat, S. , and Yang, R.-J. , 2010, “ Comparing Time Histories for Validation of Simulation Models: Error Measures and Metrics,” ASME J. Dyn. Syst. Meas. Control, 132(6), p. 061401. [CrossRef]
Cheng, Z. , Pellettiere, J. , and Wright, N. , 2006, “ Wavelet-Based Test-Simulation Correlation Analysis for the Validation of Biodynamical Modeling,” 24th Conference and Exposition on Structural Dynamics, St Louis, MI, Jan 30–Feb. 2, pp. 2124–2132.
Harmon, S. , and Youngblood, S. M. , 2005, “ A Proposed Model for Simulation Validation Process Maturity,” J. Defense Model. Simul.: Appl., Methodol., Technol., 2(4), pp. 179–190.
Burrus, C. S. , Gopinath, R. A. , and Guo, H. , 1998, Introduction to Wavelets and Wavelet Transforms, 1st ed., Prentice Hall, Upper Saddle River, NJ.
Chui, C. K. , 1992, An Introduction to Wavelets, 1st ed., Academic Press, Boston, MA.
Najmi, A.-H. , 2012, Wavelets a Concise Guide, 1st ed., Johns Hopkins University Press, Baltimore, MD.
Ogden, T. , 1997, Essential Wavelets for Statistical Applications and Data Analysis, 1st ed., Birkhauser, Boston, MA.
Donoho, D. L. , and Johnstone, J. M. , 1994, “ Ideal Spatial Adaptation by Wavelet Shrinkage,” Biometrika, 81(3), pp. 425–455. [CrossRef]
Girimurugan, S. , Chicken, E. , Pignatiello , J. J., Jr. , and Zeisset, M. S. , 2013, Industrial and Systems Engineering Research Conference, San Juan, PR, May 18–22, pp. 3235–3244.
Girimurugan, S. B. , 2014, “ Nonlinear Multivariate Tests for High-Dimensional Data Using Wavelets With Applications in Genomics and Engineering,” Ph.D. thesis, Florida State University, Tallahassee, FL.
Bracey, M. J. , 2016, “ Dynamic Modeling of Thermal Management System With Exergy Based Optimization,” Master's thesis, Wright State University, Dayton, OH. https://etd.ohiolink.edu/!etd.send_file?accession=wright1503682474459341&disposition=inline

Figures

Grahic Jump Location
Fig. 1

System and model data, example 1

Grahic Jump Location
Fig. 2

System and model disparity, example 1

Grahic Jump Location
Fig. 3

Magnification of system and model data disparity, example 1

Grahic Jump Location
Fig. 4

System and model data, example 2

Grahic Jump Location
Fig. 5

System and model disparity, example 2

Grahic Jump Location
Fig. 6

Case study example, sample 19 temperature response

Grahic Jump Location
Fig. 7

Case study example, sample 19 temperature response exampled with wavelet methodology

Grahic Jump Location
Fig. 8

Case study example, sample 19 pressure response

Grahic Jump Location
Fig. 9

Case study example, sample 19 pressure response with wavelet methodology results

Tables

Errata

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In