Abstract

In an increasingly interconnected & cyber-physical world, complexity is often cited as the root cause of adverse project outcomes, including cost-overruns and schedule delays. This realization has prompted calls for better complexity management, which hinges on the ability to recognize and measure complexity early in the design process. However, while numerous complexity measures (CMs) have been promulgated, there is limited agreement about “how” complexity should be measured and what a good measure should entail. In this paper, we propose a framework for benchmarking CMs in terms of how well they are able to detect systematic variation along key aspects of complexity growth. Specifically, the literature is consistent in expecting that complexity growth is correlated with increases in size, number of interconnections, and randomness of the system architecture. Therefore, to neutrally compare six representative CMs, we synthetically create a set of system architectures that systematically vary across each dimension. We find that none of the measures are able to detect changes in all three dimensions simultaneously, though several are consistent in their response to one or two. We also find that there is a dichotomy in the literature regarding the archetype of systems that are considered as complex: CMs developed by researchers focused on physics-based (e.g., aircraft) tend to emphasize interconnectedness and structure whereas flow-based (e.g., the power grid) focus on size. Our findings emphasize the need for more careful validation across proposed measures. Our framework provides a path to enable shared progress towards the goal of better complexity management.

References

1.
Bashir
,
H. A.
, and
Thomson
,
V.
,
1999
, “
Estimating Design Complexity
,”
J. Eng. Des.
,
10
(
3
), pp.
247
257
.
2.
Braha
,
D.
, and
Maimon
,
O.
,
1998
, “
The Measurement of a Design Structural and Functional Complexity
,”
IEEE Trans. Syst. Man Cybern. Part A Syst. Humans
,
28
(
4
), pp.
527
535
.
3.
Phukan
,
A.
,
Kalava
,
M.
, and
Prabhu
,
V.
,
2005
, “
Complexity Metrics for Manufacturing Control Architectures Based on Software and Information Flow
,”
Comput. Ind. Eng.
,
49
(
1
), pp.
1
20
.
4.
Salman
,
N.
, and
Dogru
,
A.
,
2004
, “
Design Effort Estimation Using Complexity Metrics
,”
J. Integr. Des. Process Sci.
,
8
(
3
), pp.
83
88
.
5.
Tamaskar
,
S.
,
Neema
,
K.
, and
DeLaurentis
,
D.
,
2014
, “
Framework for Measuring Complexity of Aerospace Systems
,”
Res. Eng. Des.
,
25
(
2
), pp.
125
137
.
6.
Pich
,
M. T.
,
Loch
,
C. H.
, and
de Meyer
,
A.
,
2002
, “
On Uncertainty, Ambiguity, and Complexity in Project Management
,”
Manage. Sci.
,
48
(
8
), pp.
1008
1023
.
7.
Ameri
,
F.
,
Summers
,
J. D.
,
Mocko
,
G. M.
, and
Porter
,
M.
,
2008
, “
Engineering Design Complexity: An Investigation of Methods and Measures
,”
Res. Eng. Des.
,
19
(
2–3
), pp.
161
179
.
8.
Sinha
,
K.
, and
de Weck
,
O. L.
,
2016
, “
Empirical Validation of Structural Complexity Metric and Complexity Management for Engineering Systems
,”
Syst. Eng.
,
19
(
3
), pp.
193
206
.
9.
Government Accountability Office
,
2018
, “
Integration and Test Challenges Have Delayed Launch and Threaten to Push Costs Over Cap
,” United States Government Accountability Office, GAO-18-273.
10.
Government Accountability Office
,
2020
, “
Technical Challenges Have Caused Schedule Strain and May Increase Costs
,” Government Accountability Office, GAO-20-224.
11.
Sheard
,
S. A.
, and
Mostashari
,
A.
,
2010
, “
7.3.1 A Complexity Typology for Systems Engineering
,”
INCOSE Int. Symp.
,
20
(
1
), pp.
933
945
.
12.
Summers
,
J. D.
, and
Shah
,
J. J.
,
2010
, “
Mechanical Engineering Design Complexity Metrics: Size, Coupling, and Solvability
,”
ASME J. Mech. Des.
,
132
(
2
), p.
021004
.
13.
Advancing the Design and Modeling of Complex Systems
,” https://www.darpa.mil/news-events/2015-11-20, Accessed July 9, 2021.
14.
de Weck
,
O. L.
,
Roos
,
D.
, and
Magee
,
C. L.
,
2011
,
Engineering Systems: Meeting Human Needs in a Complex Technological World
,
MIT Press
,
Cambridge, MA
.
15.
INCOSE
,
2015
,
INCOSE Systems Engineering Handbook : A Guide for System Life Cycle Processes and Activities
,
John Wiley & Sons, Inc.
,
San Diego, CA
.
16.
Lee
,
K.
,
Moses
,
M.
, and
Chirikjian
,
G. S.
,
2008
, “
Robotic Self-Replication in Structured Environments: Physical Demonstrations and Complexity Measures
,”
Int. J. Rob. Res.
,
27
(
3–4
), pp.
387
401
.
17.
McCabe
,
T. J.
,
1976
, “
A Complexity Measure
,”
IEEE Trans. Softw. Eng.
,
SE-2
(
4
), pp.
308
320
.
18.
Hölttä
,
K. M. M.
, and
Otto
,
K. N.
,
2005
, “
Incorporating Design Effort Complexity Measures in Product Architectural Design and Assessment
,”
Des. Stud.
,
26
(
5
), pp.
463
485
.
19.
Simon
,
H.
,
1962
, “
The Architecture of Complexity
,”
Proc. Am. Philos. Soc.
,
106
(
6
), pp.
467
482
.
20.
Crawley
,
E.
,
De Weck
,
O.
,
Magee
,
C.
,
Moses
,
J.
,
Seering
,
W.
,
Schindall
,
J.
,
Wallace
,
D.
, and
Whitney
,
D.
,
2004
,
The Influence of Architecture in Engineering Systems
,
Massachusetts Institute of Technology
,
Cambridge, MA
.
21.
Arena
,
M. V.
,
Younossi
,
O.
,
Brancato
,
K.
,
Blickstein
,
I.
, and
Grammich
,
C. A.
,
2008
,
Why Has the Cost of Fixed-Wing Aircraft Risen? A Macroscopic Examination of the Trends in U.S. Military Aircraft Costs Over the Past Several Decades
,
Rand Corporation
,
Santa Monica, CA
.
22.
Lloyd
,
S.
,
2001
, “
Measures of Complexity: A Nonexhaustive List
,”
IEEE Control System Magazine
.
23.
Carey
,
K.
,
2016
, “
A Complexity Primer for Systems Engineers
,” INCOSE Working Group, INCOSE.
24.
Moses
,
J.
,
2004
,
Foundational Issues in Engineering Systems: A Framing Paper
,
Massachusetts Institute of Technology
,
Cambridge, MA
.
25.
Halstead
,
M. H.
,
1977
,
Elements of Software Science
,
Elsevier Science Inc.
,
New York
.
26.
Broniatowski
,
D. A.
, and
Moses
,
J.
,
2016
, “
Measuring Flexibility, Descriptive Complexity, and Rework Potential in Generic System Architectures: Metrics for Generic System Architectures
,”
Syst. Eng.
,
19
(
3
), pp.
207
221
.
27.
Baldwin
,
C. Y.
, and
Clark
,
K. B.
,
2000
,
Design Rules
,
The MIT Press
,
Cambridge, MA
.
28.
Sussman
,
J. M.
,
2002
,
Collected Views on Complexity in Systems
,
Massachusetts Institute of Technology
,
Cambridge, MA
.
29.
Suh
,
N.
,
2005
, “
Complexity in Engineering
,”
CIRP Ann.
,
54
(
2
), pp.
46
63
.
30.
Lindemann
,
U.
,
2009
,
Structural Complexity Management: An Approach for the Field of Product Design
,
Springer
,
Berlin
.
31.
Kossiakoff
,
A.
,
Sweet
,
S. N.
,
Seymour
,
S. J.
, and
Biemer
,
S.
,
2011
,
Systems Engineering: Principles and Practice
, 2nd ed.,
Wiley-Interscience
,
Hoboken, NJ
.
32.
ElMaraghy
,
W.
,
ElMaraghy
,
H.
,
Tomiyama
,
T.
, and
Monostori
,
L.
,
2012
, “
Complexity in Engineering Design and Manufacturing
,”
CIRP Ann.
,
61
(
2
), pp.
793
814
.
33.
Kolmogorov
,
A. N.
,
1983
, “
Combinatorial Foundations of Information Theory and the Calculus of Probabilities
,”
Russ. Math. Surv.
,
38
(
4
), pp.
29
40
.
34.
Kolmogorov
,
A. N.
,
1965
, “
Three Approaches to the Quantitative Definition of Information
,”
Probl. Inf. Transm.
,
1
(
1
), pp.
3
11
.
35.
Hu
,
S. J.
,
Zhu
,
X.
,
Wang
,
H.
, and
Koren
,
Y.
,
2008
, “
Product Variety and Manufacturing Complexity in Assembly Systems and Supply Chains
,”
CIRP Ann.
,
57
(
1
), pp.
45
48
.
36.
Zhu
,
X.
,
Hu
,
S. J.
,
Koren
,
Y.
, and
Marin
,
S.
,
2008
, “
Modeling of Manufacturing Complexity in Mixed-Model Assembly Lines
,”
ASME J. Manuf. Sci. Eng.
,
130
(
5
), p.
051013
.
37.
Zaeh
,
M. F.
,
Wiesbeck
,
M.
,
Stork
,
S.
, and
Schubö
,
A.
,
2009
, “
A Multi-dimensional Measure for Determining the Complexity of Manual Assembly Operations
,”
Prod. Eng.
,
3
(
4–5
), pp.
489
496
.
38.
Boushaala
,
A. A.
,
2010
, “
Project Complexity Indices Based on Topology Features
,”
World Acad. Sci. Eng. Technol.
,
70
, pp.
49
54
.
39.
Shouman
,
M. A.
,
Ghafagy
,
A. Z.
,
Zaghloul
,
M. A.
, and
Boushaala
,
A. A.
,
1999
, “
New Heuristics for Scheduling Single Constrained Resource Projects
,”
Alexandria Eng. J.
,
38
(
3
), pp.
161
177
.
40.
Badiru
,
A. B.
,
1988
, “
Towards the Standardization of Performance Measures for Project Scheduling Heuristics
,”
IEEE Trans. Eng. Manage.
,
35
(
2
), pp.
82
89
.
41.
Caprace
,
J.-D.
, and
Rigo
,
P.
,
2012
, “
A Real-Time Assessment of the Ship Design Complexity
,”
Comput. Aided Des.
,
44
(
3
), pp.
203
208
.
42.
Sinha
,
K.
, and
Suh
,
E. S.
,
2018
, “
Pareto-Optimization of Complex System Architecture for Structural Complexity and Modularity
,”
Res. Eng. Des.
,
29
(
1
), pp.
123
141
.
43.
Orfi
,
N.
,
Terpenny
,
J.
, and
Sahin-Sariisik
,
A.
,
2011
, “
Harnessing Product Complexity: Step 1—Establishing Product Complexity Dimensions and Indicators
,”
Eng. Econ.
,
56
(
1
), pp.
59
79
.
44.
Keane
,
R. G.
,
Deschamps
,
L.
, and
Maguire
,
S.
,
2015
, “
Reducing Detail Design and Construction Work Content by Cost-Effective Decisions in Early-Stage Naval Ship Design
,”
J. Ship. Prod. Des.
,
31
(
3
), pp.
1
14
.
45.
Jacobs
,
M. A.
,
2013
, “
Complexity: Toward an Empirical Measure
,”
Technovation
,
33
(
4–5
), pp.
111
118
.
46.
Mezić
,
I.
,
Fonoberov
,
V. A.
,
Fonoberova
,
M.
, and
Sahai
,
T.
,
2019
, “
Spectral Complexity of Directed Graphs and Application to Structural Decomposition
,”
Complexity
,
2019
(
Jan.
), pp.
1
18
.
47.
Anderson
,
R. J.
, and
Sturges
,
R. H.
,
2013
, “
System Behaviors and Measures: Logical Complexity and State Complexity in Naval Weapons Elevators
,”
Complex Syst.
,
22
(
3
), pp.
247
309
.
48.
Sinha
,
K.
,
Suh
,
E. S.
, and
de Weck
,
O.
,
2018
, “
Integrative Complexity: An Alternative Measure for System Modularity
,”
ASME J. Mech. Des.
,
140
(
5
), p.
051101
.
49.
El-Haik
,
B.
, and
Yang
,
K.
,
1999
, “
The Component of Complexity in Engineering Design
,”
IIE Trans.
,
31
(
10
), pp.
925
934
.
50.
Zhang
,
X.
, and
Thomson
,
V.
,
2016
, “
The Impact and Mitigation of Complexity During Product Design
,”
Int. J. Des. Nat. Ecodyn.
,
11
(
4
), pp.
553
562
.
51.
Zhang
,
X.
, and
Thomson
,
V.
,
2018
, “
A Knowledge-Based Measure of Product Complexity
,”
Comput. Ind. Eng.
,
115
, pp.
80
87
.
52.
Sipser
,
M.
,
1992
, “
The History and Status of the P Versus NP Question
,”
Proceedings of the Twenty-Fourth Annual ACM Symposium on Theory of Computing—STOC’92
,
Victoria, British Columbia
,
May 4–6
, pp.
603
618
.
53.
Fortnow
,
L.
, and
Homer
,
S.
,
2002
, “
A Short History of Computational Complexity
,”
Bulletin of the EATCS
,
80
, pp.
27
.
54.
Prather
,
R. E.
,
1984
, “
An Axiomatic Theory of Software Complexity Measure
,”
Comput. J.
,
27
(
4
), pp.
340
347
.
55.
Shannon
,
C. E.
,
1948
, “
A Mathematical Theory of Communication
,”
Bell Syst. Tech. J.
,
27
(
3
), pp.
379
423
.
56.
Suh
,
N. P.
,
1998
, “
Axiomatic Design Theory for Systems
,”
Res. Eng. Des.
,
10
, p.
21
.
57.
Griffin
,
A.
,
1993
, “
Metrics for Measuring Product Development Cycle Time
,”
J. Prod. Innov. Manage.
,
10
(
2
), pp.
112
125
.
58.
Chen
,
L.
, and
Li
,
S.
,
2005
, “
Analysis of Decomposability and Complexity for Design Problems in the Context of Decomposition
,”
ASME J. Mech. Des.
,
127
(
4
), pp.
545
557
.
59.
Bonilla
,
F.
,
Holzer
,
T.
, and
Sarkani
,
S.
,
2020
, “
Complexity Measure for Engineering Systems Incorporating System States and Behavior
,”
IEEE Syst. J.
(Early Access
), pp.
1
12
.
60.
Sosa
,
M. E.
,
Mihm
,
J.
, and
Browning
,
T. R.
,
2013
, “
Linking Cyclicality and Product Quality
,”
Manuf. Serv. Oper. Manage.
,
15
(
3
), pp.
473
491
.
61.
Luo
,
J.
, and
Magee
,
C. L.
,
2011
, “
Detecting Evolving Patterns of Self-Organizing Networks by Flow Hierarchy Measurement
,”
Complexity
,
16
(
6
), pp.
53
61
.
62.
Pahl
,
G.
,
Wallace
,
K.
, and
Blessing
,
L.
,
2007
,
Engineering Design: A Systematic Approach
, 3rd ed.,
Springer
,
London
.
63.
Levitt
,
R. E.
,
Thomsen
,
J.
,
Christiansen
,
T. R.
,
Kunz
,
J. C.
,
Jin
,
Y.
, and
Nass
,
C.
,
1999
, “
Simulating Project Work Processes and Organizations: Toward a Micro-Contingency Theory of Organizational Design
,”
Manage. Sci.
,
45
(
11
), pp.
1479
1495
.
64.
Perrow
,
C.
,
1999
,
Normal Accidents: Living With High-Risk Technologies
,
Princeton University Press
,
Princeton, NJ
.
65.
Gomes
,
V. M.
,
Paiva
,
J. R. B.
,
Reis
,
M. R. C.
,
Wainer
,
G. A.
, and
Calixto
,
W. P.
,
2019
, “
Mechanism for Measuring System Complexity Applying Sensitivity Analysis
,”
Complexity
,
2019
(
Apr.
), pp.
1
12
.
66.
Maier
,
M. W.
, and
Rechtin
,
E.
,
2000
,
The Art of Systems Architecting
, 2nd ed.,
CRC Press
,
Boca Raton, FL
.
67.
Crawley
,
E.
,
Cameron
,
B.
, and
Selva
,
D.
,
2015
,
System Architecture: Strategy and Product Development for Complex Systems
, 1st ed,
Pearson
,
Hoboken, NJ
.
68.
Rasmussen
,
J.
,
1985
, “
The Role of Hierarchical Knowledge Representation in Decision Making and System Management
,”
IEEE Trans. Syst. Man Cybern.
,
SMC-15
(
2
), pp.
234
243
.
69.
Shafiei-Monfared
,
S.
, and
Jenab
,
K.
,
2012
, “
A Novel Approach for Complexity Measure Analysis in Design Projects
,”
J. Eng. Des.
,
23
(
3
), pp.
185
194
.
70.
Vidal
,
L.-A.
,
Marle
,
F.
, and
Bocquet
,
J.-C.
,
2011
, “
Measuring Project Complexity Using the Analytic Hierarchy Process
,”
Int. J. Project Manage.
,
29
(
6
), pp.
718
727
.
71.
Luo
,
J.
, and
Wood
,
K. L.
,
2017
, “
The Growing Complexity in Invention Process
,”
Res. Eng. Des.
,
28
(
4
), pp.
421
435
.
72.
Wang
,
H.
,
Zhu
,
X.
,
Wang
,
H.
,
Hu
,
S. J.
,
Lin
,
Z.
, and
Chen
,
G.
,
2011
, “
Multi-objective Optimization of Product Variety and Manufacturing Complexity in Mixed-Model Assembly Systems
,”
J. Manuf. Syst.
,
30
(
1
), pp.
16
27
.
73.
Allaire
,
D.
,
He
,
Q.
,
Deyst
,
J.
, and
Willcox
,
K.
,
2012
, “
An Information-Theoretic Metric of System Complexity With Application to Engineering System Design
,”
ASME J. Mech. Des.
,
134
(
10
), p.
100906
.
74.
Sinha
,
K.
,
2014
, “
Structural Complexity and Its Implications for Design of Cyber—Physical Systems
,”
dissertation
,
Massachusetts Institute of Technology
,
Cambridge, MA
.
75.
Moses
,
J.
,
2010
, “Flexibility and Its Relation to Complexity and Architecture,”
Complex Systems Design & Management
,
M.
Aiguier
,
F.
Bretaudeau
, and
D.
Krob
, eds.,
Springer
,
Berlin
, pp.
197
206
.
76.
Mathieson
,
J. L.
,
Wallace
,
B. A.
, and
Summers
,
J. D.
,
2013
, “
Assembly Time Modelling Through Connective Complexity Metrics
,”
Int. J. Comput. Integr. Manuf.
,
26
(
10
), pp.
955
967
.
77.
Min
,
G.
,
Suh
,
E. S.
, and
Hölttä-Otto
,
K.
,
2016
, “
System Architecture, Level of Decomposition, and Structural Complexity: Analysis and Observations
,”
ASME J. Mech. Des.
,
138
(
2
), p.
021102
.
78.
Sinha
,
S.
,
Kumar
,
B.
, and
Thomson
,
A.
,
2011
, “
Complexity Measurement of a Project Activity
,”
Int. J. Ind. Syst. Eng.
,
8
(
4
), pp.
432
448
.
79.
Yu
,
S. B.
, and
Efstathiou
,
J.
,
2006
, “
Complexity in Rework Cells: Theory, Analysis and Comparison
,”
J. Oper. Res. Soc.
,
57
(
5
), pp.
593
602
.
80.
Kumari
,
M.
, and
Kulkarni
,
M. S.
,
2019
, “
Single-Measure and Multi-Measure Approach of Predictive Manufacturing Control: A Comparative Study
,”
Comput. Ind. Eng.
,
127
(
Jan.
), pp.
182
195
.
81.
Bhattacharjee
,
T. K.
, and
Sahu
,
S.
,
1990
, “
Complexity of Single Model Assembly Line Balancing Problems
,”
Eng. Costs Prod. Econ.
,
18
(
3
), pp.
203
214
.
82.
Sivadasan
,
S.
,
Efstathiou
,
J.
,
Calinescu
,
A.
, and
Huatuco
,
L. H.
,
2006
, “
Advances on Measuring the Operational Complexity of Supplier–Customer Systems
,”
Eur. J. Oper. Res.
,
171
(
1
), pp.
208
226
.
83.
Weyuker
,
E. J.
,
1988
, “
Evaluating Software Complexity Measures
,”
IEEE Trans. Softw. Eng.
,
14
(
9
), pp.
1357
1365
.
84.
Hölttä-Otto
,
K.
,
Chiriac
,
N. A.
,
Lysy
,
D.
, and
Suk Suh
,
E.
,
2012
, “
Comparative Analysis of Coupling Modularity Metrics
,”
J. Eng. Des.
,
23
(
10–11
), pp.
790
806
.
85.
Steward
,
D. V.
,
1981
, “
The Design Structure System: A Method for Managing the Design of Complex Systems
,”
IEEE Trans. Eng. Manage.
,
EM-28
(
3
), pp.
71
74
.
86.
Browning
,
T. R.
,
2016
, “
Design Structure Matrix Extensions and Innovations: A Survey and New Opportunities
,”
IEEE Trans. Eng. Manage.
,
63
(
1
), pp.
27
52
.
87.
Steven
,
D. E.
, and
Browning
,
T. R.
,
2010
,
Design Structure Matrix Methods and Applications
,
The MIT Press
,
Cambridge, MA
.
88.
Sharman
,
D. M.
, and
Yassine
,
A. A.
,
2004
, “
Characterizing Complex Product Architectures
,”
Syst. Eng.
,
7
(
1
), pp.
35
60
.
89.
Tilstra
,
A. H.
,
Seepersad
,
C. C.
, and
Wood
,
K. L.
,
2012
, “
A High-Definition Design Structure Matrix (HDDSM) for the Quantitative Assessment of Product Architecture
,”
J. Eng. Des.
,
23
(
10–11
), pp.
767
789
.
90.
Guo
,
F.
, and
Gershenson
,
J. K.
,
2007
, “
Discovering Relationships Between Modularity and Cost
,”
J. Intell. Manuf.
,
18
, pp.
143
157
.
91.
Ulrich
,
K.
,
1995
, “
The Role of Product Architecture in the Manufacturing Firm
,”
Res. Policy
,
24
(
3
), pp.
419
440
.
92.
Browning
,
T. R.
,
2001
, “
Applying the Design Structure Matrix to System Decomposition and Integration Problems: A Review and New Directions
,”
IEEE Trans. Eng. Manage.
,
48
(
3
), pp.
292
306
.
93.
Huang
,
C.-C.
, and
Kusiak
,
A.
,
1998
, “
Modularity in Design of Products and Systems
,”
IEEE Trans. Syst. Man Cybern. Part A Syst. Humans
,
28
(
1
), pp.
66
77
.
94.
Sharman
,
D. M.
, and
Yassine
,
A. A.
,
2007
, “
Architectural Valuation Using the Design Structure Matrix and Real Options Theory
,”
Concurr. Eng.
,
15
(
2
), pp.
157
173
.
95.
Suh
,
N. P.
,
1999
, “
A Theory of Complexity, Periodicity and the Design Axioms
,”
Res. Eng. Des.
,
11
, p.
16
.
You do not currently have access to this content.