Abstract

The engineering of complex systems, such as aircraft and spacecraft, involves large number of individuals within multiple organizations spanning multiple years. Since it is challenging to perform empirical studies directly on real organizations at scale, some researchers in systems engineering and design have begun relying on abstracted model worlds that aim to be representative of the reference socio-technical system, but only preserve some aspects of it. However, there is a lack of corresponding knowledge on how to design representative model worlds for socio-technical research. Our objective is to create such knowledge through a reflective case study of the development of a model world. This “inner” study examines how two factors influence interdisciplinary communication during a concurrent design process. The reference real world system is a mission design laboratory (MDL) at NASA, and the model world is a simplified engine design problem in an undergraduate classroom environment. Our analysis focuses on the thought process followed, the key model world design decisions made, and a critical assessment of the extent to which communication phenomena in the model world (engine experiment) are representative of the real world (NASA’s MDL). We find that the engine experiment preserves some but not all of the communication patterns of interest, and we present case-specific lessons learned for achieving and increasing representativeness in this type of study. More generally, we find that representativeness depends not on matching subjects, tasks, and context separately, but rather on the behavior that emerges from the interplay of these three dimensions.

References

1.
Szajnfarber
,
Z.
,
Grogan
,
P. T.
,
Panchal
,
J. H.
, and
Gralla
,
E.
,
2020
, “
A Call for Consensus on the Use of Representative Model Worlds in Systems Engineering and Design
,”
Syst. Eng.
,
23
(
4
), pp.
436
442
. 10.1002/sys.21536
2.
Panchal
,
J. H.
, and
Szajnfarber
,
Z.
,
2017
, “
Experiments in Systems Engineering and Design Research
,”
Syst. Eng.
,
20
(
6
), pp.
529
541
. 10.1002/sys.21415
3.
Grogan
,
P. T.
, and
de Weck
,
O. L.
,
2016
, “
Collaboration and Complexity: An Experiment on the Effect of Multi-Actor Coupled Design
,”
Res. Eng. Design
,
27
(
3
), pp.
221
235
. 10.1007/s00163-016-0214-7
4.
Grogan
,
P. T.
, and
de Weck
,
O. L.
,
2016
, “
Collaborative Design in the Sustainable Infrastructure Planning Game
,”
2016 Spring Simulation Multi-Conference Annual Simulation Symposium
,
Pasadena, CA
.
5.
Chaudhari
,
A. M.
,
Bilionis
,
I.
, and
Panchal
,
J. H.
,
2020
, “
Descriptive Models of Sequential Decisions in Engineering Design: An Experimental Study
,”
ASME J. Mech. Des.
,
142
(
8
), p.
081704
. 10.1115/1.4045605
6.
Gralla
,
E.
,
Goentzel
,
J.
, and
Fine
,
C.
,
2016
, “
Problem Formulation and Solution Mechanisms: A Behavioral Study of Humanitarian Transportation Planning
,”
Product. Operat. Manage.
,
25
(
1
), pp.
22
35
. 10.1111/poms.12496
7.
Azhar
,
A.
,
Gralla
,
E. L.
,
Tobias
,
C.
, and
Herrmann
,
J. W.
,
2016
, “
Identification of Subproblems in Complex Design Problems: A Study of Facility Design
,”
ASME 2016 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Charlotte, NC
.
8.
Chaudhari
,
A. M.
,
Sha
,
Z.
, and
Panchal
,
J. H.
,
2018
, “
Analyzing Participant Behaviors in Design Crowdsourcing Contests Using Causal Inference on Field Data
,”
ASME J. Mech. Des.
,
140
(
9
), p.
091401
. 10.1115/1.4040166
9.
Gralla
,
E.
, and
Szajnfarber
,
Z.
,
2018
, “
Games and Exercises for Teaching and Research: Exploring How Learning Varies Based on Fidelity and Participant Experience
,”
Volume 2B: 44th Design Automation Conference
,
Quebec City, Canada
, American Society of Mechanical Engineers, p. V02BT03A026.
10.
Kruskal
,
W.
, and
Mosteller
,
F.
,
1979
, “
Representative Sampling, III: The Current Statistical Literature
,”
Int. Stat. Rev.
,
47
(
3
), p.
245
. 10.2307/1402647
11.
Yin
,
R. K.
,
2009
,
Case Study Research: Design and Methods
(
Applied social research methods series
),
Sage Publications
,
Beverly Hills, CA
.
12.
Brunswik
,
E.
,
1949
, “
Systematic and Representative Design of Psychological Experiments. with Results in Physical and Social Perception
,”
Proceedings of the [First] Berkeley Symposium on Mathematical Statistics and Probability
,
Berkeley, CA
,
University of California Press
, pp.
143
202
.
13.
Sargent
,
R. G.
,
2013
, “
Verification and Validation of Simulation Models
,”
J. Simul.
,
7
(
1
), pp.
12
24
. 10.1057/jos.2012.20
14.
Zeigler
,
B. P.
,
Praehofer
,
H.
, and
Kim
,
T.
,
2000
,
Theory of Modeling and Simulation
, 2nd ed,
Academic Press
,
New York
.
15.
Raser
,
J. R.
,
1969
,
Simulation and Society: An Exploration of Scientific Gaming
,
Allyn and Bacon
,
Boston
.
16.
Anandan
,
S.
,
Teegavarapu
,
S.
, and
Summers
,
J. D.
,
2006
, “
Issues of Similarity in Engineering Design
,”
Volume 1: 32nd Design Automation Conference, ASME, Parts A and B
,
Philadelphia, PA
.
17.
Durand
,
F.
,
Helms
,
M. E.
,
Tsenn
,
J.
,
McAdams
,
D. A.
, and
Linsey
,
J. S.
,
2015
, “
In Search of Effective Design Problems for Design Research
,”
Volume 7: 27th International Conference on Design Theory and Methodology
,
Boston, MA
, American Society of Mechanical Engineers.
18.
Kumar
,
V.
, and
Mocko
,
G.
,
2016
, “
Similarity of Engineering Design Problems to Enable Reuse in Design Research Experiments
,”
Volume 7: 28th International Conference on Design Theory and Methodology
,
Charlotte, NC
, American Society of Mechanical Engineers.
19.
Sosa
,
R.
,
2018
, “
Metrics to Select Design Tasks in Experimental Creativity Research
,”
Proce. Inst. Mech. Eng., Part C: J. Mech. Eng. Sci.
,
233
(
2
), pp.
440
450
. 10.1177/0954406218778305
20.
Levy
,
B.
,
Hilton
,
E.
,
Tomko
,
M.
, and
Linsey
,
J.
,
2017
, “
Investigating Problem Similarity Through Study of Between-subject and Within-subject Experiments
,”
Volume 7: 29th International Conference on Design Theory and Methodology
,
Cleveland, OH
, American Society of Mechanical Engineers.
21.
Cross
,
N.
,
2004
, “
Expertise in Design: An Overview
,”
Design Studies
,
25
(
5
), pp.
427
441
. 10.1016/j.destud.2004.06.002
22.
Cross
,
N.
, and
Cross
,
A. C.
,
1998
, “
Expertise in Engineering Design
,”
Res. Eng. Design
,
10
(
3
), pp.
141
149
. 10.1007/BF01607156
23.
Ahmed
,
S.
,
Wallace
,
K. M.
, and
Blessing
,
L. T.
,
2003
, “
Understanding the Differences Between How Novice and Experienced Designers Approach Design Tasks
,”
Res. Eng. Design
,
14
(
1
), pp.
1
11
. 10.1007/s00163-002-0023-z
24.
Crismond
,
D. P.
, and
Adams
,
R. S.
,
2012
, “
The Informed Design Teaching and Learning Matrix
,”
J. Eng. Education
,
101
(
4
), pp.
738
797
. 10.1002/j.2168-9830.2012.tb01127.x
25.
Atman
,
C. J.
,
Adams
,
R. S.
,
Cardella
,
M. E.
,
Turns
,
J.
,
Mosborg
,
S.
, and
Saleem
,
J.
,
2007
, “
Engineering Design Processes: A Comparison of Students and Expert Practitioners
,”
J. Eng. Education
,
96
(
4
), pp.
359
379
. 10.1002/j.2168-9830.2007.tb00945.x
26.
Cash
,
P. J.
,
Hicks
,
B. J.
, and
Culley
,
S. J.
,
2013
, “
A Comparison of Designer Activity Using Core Design Situations in the Laboratory and Practice
,”
Design Studies
,
34
(
5
), pp.
575
611
. 10.1016/j.destud.2013.03.002
27.
Hales
,
C.
,
2000
, “
Ten Critical Factors in the Design Process
,”
Proceedings of ASM/ASME Conference on Failure Prevention Through Education–Getting to the Root Cause
,
Strongsville, OH
, pp.
49
55
.
28.
Szajnfarber
,
Z.
, and
Gralla
,
E.
,
2017
, “
Qualitative Methods for Engineering Systems: Why We Need them and How to Use them
,”
Syst. Eng.
,
20
(
6
), pp.
497
511
. 10.1002/sys.21412
29.
Broniatowski
,
D. A.
, and
Tucker
,
C.
,
2017
, “
Assessing Causal Claims About Complex Engineered Systems with Quantitative Data: Internal, External, and Construct Validity
,”
Syst. Eng.
,
20
(
6
), pp.
483
496
. 10.1002/sys.21414
30.
Sherwood
,
B.
, and
McCleese
,
D.
,
2013
, “
JPL Innovation Foundry
,”
Acta Astronaut.
,
89
, pp.
236
247
. 10.1016/j.actaastro.2013.04.020
31.
Eppinger
,
S. D.
,
2002
,
Patterns of product development interactions
. ESD Working Papers ESD-WP-2003-01.05-ESD Internal Symposium.
32.
Bucciarelli
,
L. L.
,
1994
,
Designing Engineers
,
MIT Press
,
Cambridge, MA
.
33.
Sosa
,
M. E.
,
Eppinger
,
S. D.
, and
Rowles
,
C. M.
,
2004
, “
The Misalignment of Product Architecture and Organizational Structure in Complex Product Development
,”
Manage. Sci.
,
50
(
12
), pp.
1674
1689
. 10.1287/mnsc.1040.0289
34.
DeChurch
,
L. A.
, and
Mesmer-Magnus
,
J. R.
,
2010
, “
The Cognitive Underpinnings of Effective Teamwork: A Meta-analysis.
,”
J. Appl. Psychol.
,
95
(
1
), pp.
32
53
. 10.1037/a0017328
35.
DeChurch
,
L. A.
, and
Mesmer-Magnus
,
J. R.
,
2010
, “
Measuring Shared Team Mental Models: A Meta-analysis.
,”
Group Dyn.: Theory, Res., Practice
,
14
(
1
), pp.
1
14
. 10.1037/a0017455
36.
Kozlowski
,
S. W.
, and
Ilgen
,
D. R.
,
2006
, “
Enhancing the Effectiveness of Work Groups and Teams
,”
Psychol. Sci. Public Interest
,
7
(
3
), pp.
77
124
. 10.1111/j.1529-1006.2006.00030.x
37.
Eppinger
,
S. D.
,
2001
, “
Innovation At the Speed of Information
,”
Harvard Bus. Rev.
,
79
(
1
), pp.
149
158
.
38.
Sbea
,
G. P.
, and
Guzzo
,
R. A.
,
1987
, “
Group Effectiveness: What Really Matters?
,”
Sloan Manage. Rev. (1986–1998)
,
28
(
3
), p.
25
.
39.
Langville
,
A. N.
, and
Meyer
,
C. D.
,
2005
, “
A Survey of Eigenvector Methods for Web Information Retrieval
,”
SIAM Rev.
,
47
(
1
), pp.
135
161
. 10.1137/S0036144503424786
40.
Hagberg
,
A.
,
Swart
,
P.
, and
S Chult
,
D.
,
2008
,
Exploring network structure, dynamics, and function using networkx
. Tech. rep.,
Los Alamos National Lab.(LANL)
,
Los Alamos, NM (United States)
.
41.
Waskom
,
M.
,
Botvinnik
,
O.
,
O’Kane
,
D.
,
Hobson
,
P.
,
Lukauskas
,
S.
,
Gemperline
,
D. C.
,
Augspurger
,
T.
,
Halchenko
,
Y.
,
Cole
,
J. B.
,
Warmenhoven
,
J.
,
de Ruiter
,
J.
,
Pye
,
C.
,
Hoyer
,
S.
,
Vanderplas
,
J.
,
Villalba
,
S.
,
Kunter
,
G.
,
Quintero
,
E.
,
Bachant
,
P.
,
Martin
,
M.
,
Meyer
,
K.
,
Miles
,
A.
,
Ram
,
Y.
,
Yarkoni
,
T.
,
Williams
,
M. L.
,
Evans
,
C.
,
Fitzgerald
,
C.
,
Brian
,
Fonnesbeck
, and
Qalieh
,
A.
,
2017
, mwaskom/seaborn: v0.8.1 (September 2017).
42.
Ball
,
L. J.
,
Onarheim
,
B.
, and
Christensen
,
B. T.
,
2010
, “
Design Requirements, Epistemic Uncertainty and Solution Development Strategies in Software Design
,”
Design Studies
,
31
(
6
), pp.
567
589
. 10.1016/j.destud.2010.09.003
43.
Hirschi
,
N.
, and
Frey
,
D.
,
2002
, “
Cognition and Complexity: An Experiment on the Effect of Coupling in Parameter Design
,”
Res. Eng. Design
,
13
(
3
), pp.
123
131
. 10.1007/s00163-002-0011-3
44.
Dorst
,
K.
,
2008
, “
Design Research: a Revolution-waiting-to-happen
,”
Design Studies
,
29
(
1
), pp.
4
11
. 10.1016/j.destud.2007.12.001
45.
Tversky
,
A.
, and
Kahneman
,
D.
,
1974
, “
Judgment Under Uncertainty: Heuristics and Biases
,”
Science
,
185
(
4157
), pp.
1124
1131
. 10.1126/science.185.4157.1124
46.
Yu
,
B. Y.
,
Honda
,
T.
,
Sharqawy
,
M.
, and
Yang
,
M.
,
2016
, “
Human Behavior and Domain Knowledge in Parameter Design of Complex Systems
,”
Design Studies
,
45
(Part B)
, pp.
242
267
. 10.1016/j.destud.2016.04.005
47.
Diamond
,
L.
, and
Lerch
,
F. J.
,
1992
, “
Fading Frames: Data Presentation and Framing Effects
,”
Decis. Sci.
,
23
(
5
), pp.
1050
1071
. 10.1111/j.1540-5915.1992.tb00435.x
48.
Marzilli Ericson
,
K. M.
, and
Fuster
,
A.
,
2014
, “
The Endowment Effect
,”
Annu. Rev. Econ.
,
6
(
1
), pp.
555
579
. 10.1146/annurev-economics-080213-041320
49.
Levitt
,
S. D.
, and
List
,
J. A.
,
2007
, “
What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?
,”
J. Econ. Perspect.
,
21
(
2
), pp.
153
174
. 10.1257/jep.21.2.153
50.
List
,
J. A.
,
2011
, “
Why Economists Should Conduct Field Experiments and 14 Tips for Pulling One Off
,”
J. Econ. Perspect.
,
25
(
3
), pp.
3
16
. 10.1257/jep.25.3.3
51.
Tenopir
,
C.
, and
King
,
D. W.
,
2004
,
Communication Patterns of Engineers
,
John Wiley & Sons
,
Hoboken, NJ
.
52.
Allen
,
T. J.
,
1970
, “
Communication Networks in R & D Laboratories
,”
R&D Manage.
,
1
(
1
), pp.
14
21
.
53.
Harris
,
D.
,
2001
, “
Supporting Human Communication in Network-Based Systems Engineering
,”
Syst. Eng.
,
4
(
3
), pp.
213
221
. 10.1002/sys.1017
54.
McDonough III
,
E. F.
,
2000
, “
Investigation of Factors Contributing to the Success of Cross-functional Teams
,”
J. Product Innov. Manage.: Int. Publication Product Develop. Manage. Associat.
,
17
(
3
), pp.
221
235
. 10.1111/1540-5885.1730221
55.
Flager
,
F.
,
Gerber
,
D. J.
, and
Kallman
,
B.
,
2014
, “
Measuring the Impact of Scale and Coupling on Solution Quality for Building Design Problems
,”
Design Studies
,
35
(
2
), pp.
180
199
. 10.1016/j.destud.2013.11.001
56.
Grogan
,
P. T.
, and
de Weck
,
O. L.
,
2016
, “
Collaboration and Complexity: An Experiment on the Effect of Multi-actor Coupled Design
,”
Res. Eng. Design
,
27
(
3
), pp.
221
235
. 10.1007/s00163-016-0214-7
57.
Bashir
,
H. A.
,
AlZebdeh
,
K.
, and
Abdo
,
J.
,
2009
, “
An Eigenvalue Based Approach for Assessing the Decomposability of Interdependent Design Project Tasks
,”
Concurrent Eng.
,
17
(
1
), pp.
35
42
. 10.1177/1063293X09102248
58.
Sudweeks
,
F.
, and
Rafaeli
,
S.
,
1996
,
How Do You Get a Hundred Strangers to Agree?
(
Computer Networking and Scholarly Communication in the Twenty-first Century
),
New York
,
State University of New York
. pp.
115
136
.
59.
Abarbanel
,
B.
, and
McNeely
,
W.
,
1996
, “
Fly Thru the Boeing 777
,”
ACM SIGGRAPH 96 Visual Proceedings: The Art and Interdisciplinary Programs of SIGGRAPH’96
,
New Orleans, LA
, ACM, p.
124
.
60.
Cohen
,
J.
,
1992
, “
Statistical Power Analysis
,”
Current Dir. Psychol. Sci.
,
1
(
3
), pp.
98
101
. 10.1111/1467-8721.ep10768783
61.
Green
,
S. B.
,
1991
, “
How Many Subjects Does it Take to Do a Regression Analysis
,”
Multivariate Behav. Res.
,
26
(
3
), pp.
499
510
. 10.1207/s15327906mbr2603_7
62.
Kraemer
,
H. C.
, and
Blasey
,
C.
,
2015
,
How Many Subjects?: Statistical Power Analysis in Research
,
Sage Publications
.
You do not currently have access to this content.