Abstract

Design variety metrics measure how much a design space is explored. This article proposes that a generalized class of entropy metrics based on Sharma–Mittal entropy offers advantages over existing methods to measure design variety. We show that an exemplar metric from Sharma–Mittal entropy, namely, the Herfindahl–Hirschman index for design (HHID) has the following desirable advantages over existing metrics: (a) more accuracy: it better aligns with human ratings compared to existing and commonly used tree-based metrics for two new datasets; (b) higher sensitivity: it has higher sensitivity compared to existing methods when distinguishing between the variety of sets; (c) allows efficient optimization: it is a submodular function, which enables one to optimize design variety using a polynomial time greedy algorithm; and (d) generalizes to multiple metrics: many existing metrics can be derived by changing the parameters of this metric, which allows a researcher to fit the metric to better represent variety for new domains. This article also contributes a procedure for comparing metrics used to measure variety via constructing ground truth datasets from pairwise comparisons. Overall, our results shed light on some qualities that good design variety metrics should possess and the nontrivial challenges associated with collecting the data needed to measure those qualities.

References

1.
Amabile
,
T. M.
,
1996
,
Creativity in Context: Update to the Social Psychology of Creativity
,
Hachette
,
New York
.
2.
Sternberg
,
R. J.
,
1999
,
Handbook of Creativity
,
Cambridge University Press
,
Cambridge, UK
.
3.
Mumford
,
M. D.
, and
Gustafson
,
S. B.
,
1988
, “
Creativity Syndrome: Integration, Application, and Innovation.
,”
Psychol. Bull.
,
103
(
1
), p.
27
. 10.1037/0033-2909.103.1.27
4.
Baer
,
J.
,
2014
,
Creativity and Divergent Thinking: A Task-Specific Approach
,
Psychology Press
,
East Sussex, UK
.
5.
Torrance
,
E. P.
,
1972
, “
Predictive Validity of the Torrance Tests of Creative Thinking
,”
J. Creat. Behav.
,
6
(
4
), pp.
236
262
. 10.1002/j.2162-6057.1972.tb00936.x
6.
Acar
,
S.
, and
Runco
,
M. A.
,
2017
, “
Latency Predicts Category Switch in Divergent Thinking.
,”
Psychol. Aesthetics Creat. Arts
,
11
(
1
), p.
43
. 10.1037/aca0000091
7.
Beitz
,
W.
, and
Pahl
,
G.
,
1996
, “
Engineering Design: A Systematic Approach
,”
MRS Bull.
,
71
, pp.
63
124
.
8.
Shah
,
J. J.
,
Smith
,
S. M.
, and
Vargas-Hernandez
,
N.
,
2003
, “
Metrics for Measuring Ideation Effectiveness
,”
Design Stud.
,
24
(
2
), pp.
111
134
. 10.1016/S0142-694X(02)00034-0
9.
Pahl
,
G.
, and
Beitz
,
W.
,
2013
,
Engineering Design: A Systematic Approach
,
Springer Science & Business Media
,
New York
.
10.
Dorst
,
K.
, and
Cross
,
N.
,
2001
, “
Creativity in the Design Process: Co-evolution of Problem–Solution
,”
Design Stud.
,
22
(
5
), pp.
425
437
. 10.1016/S0142-694X(01)00009-6
11.
Henderson
,
D.
,
Helm
,
K.
,
Jablokow
,
K.
,
McKilligan
,
S.
,
Daly
,
S.
, and
Silk
,
E.
,
2017
, “
A Comparison of Variety Metrics in Engineering Design
,”
ASME 2017 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Cleveland, OH
,
August
.
12.
Maher
,
M. L.
,
Poon
,
J.
, and
Boulanger
,
S.
,
1996
, “Formalising Design Exploration as Co-Evolution,”
Advances in Formal Design Methods for CAD
,
J. S.
Gero
, and
F.
Sudweeks
, eds.,
The International Federation for Information Processing, Springer
,
Boston, MA
, pp.
3
30
.
13.
Linsey
,
J. S.
,
Clauss
,
E.
,
Kurtoglu
,
T.
,
Murphy
,
J.
,
Wood
,
K.
, and
Markman
,
A.
,
2011
, “
An Experimental Study of Group Idea Generation Techniques: Understanding the Roles of Idea Representation and Viewing Methods
,”
ASME J. Mech. Des.
,
133
(
3
), p.
031008
. 10.1115/1.4003498
14.
Oman
,
S. K.
,
Tumer
,
I. Y.
,
Wood
,
K.
, and
Seepersad
,
C.
,
2013
, “
A Comparison of Creativity and Innovation Metrics and Sample Validation Through In-Class Design Projects
,”
Res. Engin. Des.
,
24
(
1
), pp.
65
92
. 10.1007/s00163-012-0138-9
15.
Nelson
,
B. A.
,
Wilson
,
J. O.
,
Rosen
,
D.
, and
Yen
,
J.
,
2009
, “
Refined Metrics for Measuring Ideation Effectiveness
,”
Des. Stud.
,
30
(
6
), pp.
737
743
. 10.1016/j.destud.2009.07.002
16.
Verhaegen
,
P.-A.
,
Vandevenne
,
D.
,
Peeters
,
J.
, and
Duflou
,
J. R.
,
2013
, “
Refinements to the Variety Metric for Idea Evaluation
,”
Des. Stud.
,
34
(
2
), pp.
243
263
. 10.1016/j.destud.2012.08.003
17.
Linsey
,
J. S.
,
2007
, “
Design-by-Analogy and Representation in Innovative Engineering Concept Generation
,” Ph.D. thesis,
University of Texas
,
Austin, TX
.
18.
Sluis-Thiescheffer
,
W.
,
Bekker
,
T.
,
Eggen
,
B.
,
Vermeeren
,
A.
, and
De Ridder
,
H.
,
2016
, “
Measuring and Comparing Novelty for Design Solutions Generated by Young Children Through Different Design Methods
,”
Des. Stud.
,
43
, pp.
48
73
. 10.1016/j.destud.2016.01.001
19.
Peeters
,
J.
,
Verhaegen
,
P.-A.
,
Vandevenne
,
D.
, and
Duflou
,
J.
,
2010
, “
Refined Metrics for Measuring Novelty in Ideation
,”
IDMME Virtual Concept Res. Inter. Des.
,
30
, pp.
20
22
.
20.
Fuge
,
M.
,
Stroud
,
J.
, and
Agogino
,
A.
,
2013
, “
Automatically Inferring Metrics for Design Creativity
,”
Proceedings of the ASME 2013 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. Volume 5: 25th International Conference on Design Theory and Methodology; ASME 2013 Power Transmission and Gearing Conference
,
Portland, OR
,
Aug. 4–7
, ASME, p.
V005T06A010
. https://doi.org/10.1115/DETC2013-12620.
21.
Shatz
,
D.
,
2004
,
Peer Review: A Critical Inquiry
,
Rowman & Littlefield
,
Lanham, MD
.
22.
Chulvi
,
V.
,
Mulet
,
E.
,
Chakrabarti
,
A.
,
López-Mesa
,
B.
, and
González-Cruz
,
C.
,
2012
, “
Comparison of the Degree of Creativity in the Design Outcomes Using Different Design Methods
,”
J. Engin. Des.
,
23
(
4
), pp.
241
269
. 10.1080/09544828.2011.624501
23.
Kline
,
P.
,
2014
,
The New Psychometrics: Science, Psychology and Measurement
,
Routledge
,
London, UK
.
24.
Twomey
,
M.
,
Wallis
,
L. A.
, and
Myers
,
J. E.
,
2007
, “
Limitations in Validating Emergency Department Triage Scales
,”
Emer. Med. J.
,
24
(
7
), pp.
477
479
. 10.1136/emj.2007.046383
25.
Hennessey
,
B. A.
, and
Amabile
,
T. M.
,
1999
, “Consensual Assessment,”
Encyclopedia of Creativity
,
M.
Runco
, and
S.
Pritzker
, eds.,
Elsevier Science
,
San Diego, CA
, pp.
347
359
.
26.
Harnad
,
S.
,
2008
, “
Validating Research Performance Metrics Against Peer Rankings
,”
Ethics Sci. Environ. Politics
,
8
(
1
), pp.
103
107
. 10.3354/esep00088
27.
Baer
,
J.
, and
Kaufman
,
J. C.
,
2019
, “Assessing Creativity Using the Consensual Assessment Technique,”
The Palgrave Handbook of Social Creativity Research. Palgrave Studies in Creativity and Culture
,
I.
Lebuda
and
V.
Glăveanu
, eds.,
Palgrave Macmillan
,
Cham, London, UK
.
28.
Amabile
,
T. M.
, and
Pillemer
,
J.
,
2012
, “
Perspectives on the Social Psychology of Creativity
,”
J. Creat. Behav.
,
46
(
1
), pp.
3
15
. 10.1002/jocb.001
29.
Schaefer
,
C. E.
, and
Anastasi
,
A.
,
1968
, “
A Biographical Inventory for Identifying Creativity in Adolescent Boys.
,”
J. Appl. Psychol.
,
52
(
1p1
), p.
42
. 10.1037/h0025328
30.
Taylor
,
C.
, and
Ellison
,
R.
,
1966
,
Alpha Biographical Inventory
,
Institute for Behavioral Research in Creativity
,
Salt Lake City, UT
.
31.
Cropley
,
A. J.
,
2000
, “
Defining and Measuring Creativity: Are Creativity Tests Worth Using?
,”
Roeper Rev.
,
23
(
2
), pp.
72
79
. 10.1080/02783190009554069
32.
Douglas
,
L.
,
Jillian
,
M.
,
Thomas
,
L.
, and
Eric
,
L.
,
2006
, “
Identifying Quality, Novel, and Creative Ideas: Constructs and Scales for Idea Evaluation1
,”
J. Assoc. Infor. Syst.
,
7
(
10
), p.
646
.
33.
Cross
,
N.
, and
Roy
,
R.
,
1989
,
Engineering Design Methods
, Vol.
4
,
Wiley
,
New York
.
34.
Dylla
,
N.
,
1991
, “
Thinking Methods and Procedures in Mechanical Design
,” Ph.D. thesis,
Mechanical Design, Technical, University of Munich
,
Munich, Germany
.
35.
Song
,
S.
, and
Agogino
,
A. M.
,
2004
, “
Insights on Designers’ Sketching Activities in New Product Design Teams
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Salt Lake City, UT
,
September
, Vol.
46962
, pp.
351
360
.
36.
Shah
,
J. J.
,
2005
, “
Identification, Measurement and Development of Design Skills in Engineering Education
,”
DS 35: Proceedings ICED 0n the 15th International Conference on Engineering Design
,
Melbourne, Australia
,
Aug. 15–18
, pp.
377
378
.
37.
Jagtap
,
S.
,
Larsson
,
A.
,
Hiort
,
V.
,
Olander
,
E.
, and
Warell
,
A.
,
2015
, “
Interdependency Between Average Novelty, Individual Average Novelty, and Variety
,”
Int. J. Des. Creat. Innovat.
,
3
(
1
), pp.
43
60
. 10.5585/iji.v3i2.50
38.
Jansson
,
D. G.
, and
Smith
,
S. M.
,
1991
, “
Design Fixation
,”
Des. Stud.
,
12
(
1
), pp.
3
11
. 10.1016/0142-694X(91)90003-F
39.
Kershaw
,
T. C.
, and
Ohlsson
,
S.
,
2004
, “
Multiple Causes of Difficulty in Insight: The Case of the Nine-Dot Problem.
,”
J. Exp. Psychol. Learn. Memory Cognition
,
30
(
1
), p.
3
. 10.1037/0278-7393.30.1.3
40.
Bonnardel
,
N.
,
2000
, “
Towards Understanding and Supporting Creativity in Design: Analogies in a Constrained Cognitive Environment
,”
Knowl. Based Syst.
,
13
(
7–8
), pp.
505
513
. 10.1016/S0950-7051(00)00067-8
41.
Herring
,
S. R.
,
Chang
,
C. -C.
,
Krantzler
,
J.
, and
Bailey
,
B. P.
,
2009
, “
Getting Inspired! Understanding How and Why Examples Are Used in Creative Design Practice
,”
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
,
Boston, MA
,
April
, pp.
87
96
.
42.
Wilson
,
J. O.
,
Rosen
,
D.
,
Nelson
,
B. A.
, and
Yen
,
J.
,
2010
, “
The Effects of Biological Examples in Idea Generation
,”
Design Stud.
,
31
(
2
), pp.
169
186
. 10.1016/j.destud.2009.10.003
43.
Srinivasan
,
V.
, and
Chakrabarti
,
A.
,
2010
, “
Investigating Novelty–Outcome Relationships in Engineering Design
,”
AI EDAM
,
24
(
2
), pp.
161
178
.
44.
Verhaegen
,
P.-A.
,
Vandevenne
,
D.
,
Peeters
,
J.
, and
Duflou
,
J. R.
,
2015
, “
A Variety Metric Accounting for Unbalanced Idea Space Distributions
,”
Procedia. Eng.
,
131
, pp.
175
183
. 10.1016/j.proeng.2015.12.368
45.
Viswanathan
,
V. K.
, and
Linsey
,
J. S.
,
2012
, “
Physical Models and Design Thinking: A Study of Functionality, Novelty and Variety of Ideas
,”
ASME J. Mech. Des.
,
134
(
9
), p.
091004
. 10.1115/1.4007148
46.
Thevenot
,
H. J.
, and
Simpson
,
T. W.
,
2007
, “
A Comprehensive Metric for Evaluating Component Commonality in a Product Family
,”
J. Engin. Des.
,
18
(
6
), pp.
577
598
. 10.1080/09544820601020014
47.
Kota
,
S.
,
Sethuraman
,
K.
, and
Miller
,
R.
,
2000
, “
A Metric for Evaluating Design Commonality in Product Families
,”
ASME J. Mech. Des.
,
122
(
4
), pp.
403
410
. 10.1115/1.1320820
48.
Jung
,
S.
, and
Simpson
,
T. W.
,
2016
, “
An Integrated Approach to Product Family Redesign Using Commonality and Variety Metrics
,”
Res. Engin. Des.
,
27
(
4
), pp.
391
412
. 10.1007/s00163-016-0224-5
49.
Chan
,
J.
,
Dang
,
S.
, and
Dow
,
S. P.
,
2016
, “
Comparing Different Sensemaking Approaches for Large-Scale Ideation
,”
Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
,
San Jose, CA
,
May
,
ACM
, pp.
2717
2728
.
50.
Chan
,
J.
,
Fu
,
K.
,
Schunn
,
C.
,
Cagan
,
J.
,
Wood
,
K.
, and
Kotovsky
,
K.
,
2011
, “
On the Benefits and Pitfalls of Analogies for Innovative Design: Ideation Performance Based on Analogical Distance, Commonness, and Modality of Examples
,”
ASME J. Mech. Des.
,
133
(
8
), p.
081004
. 10.1115/1.4004396
51.
Gini
,
C.
,
1912
, “Variabilità E Mutabilità,”
Reprinted in Memorie Di Metodologica Statistica
,
E.
Pizetti
, and
T.
Salvemini
, eds.,
Libreria Eredi Virgilio Veschi
,
Rome
.
52.
Simpson
,
E. H.
,
1949
, “
Measurement of Diversity
,”
Nature
,
163
(
4148
), p.
688
. 10.1038/163688a0
53.
Shannon
,
C. E.
,
1948
, “
A Mathematical Theory of Communication
,”
Bell. Syst. Tech. J.
,
27
(
3
), pp.
379
423
. 10.1002/j.1538-7305.1948.tb01338.x
54.
Jost
,
L.
,
2006
, “
Entropy and Diversity
,”
Oikos
,
113
(
2
), pp.
363
375
. 10.1111/j.2006.0030-1299.14714.x
55.
Hirschman
,
A. O.
,
1964
, “
The Paternity of an Index
,”
Amer. Econom. Rev.
,
54
(
5
), pp.
761
762
.
56.
Crupi
,
V.
,
2019
, “Measures of Biological Diversity: Overview and Unified Framework,”
From Assessing to Conserving Biodiversity. History, Philosophy and Theory of the Life Sciences
,
E.
Casetta
,
J.
Marques da Silva
, and
D.
Vecchi
, eds., vol.
24
,
Springer
,
Cham
.
57.
Rhoades
,
S. A.
,
1993
, “
The Herfindahl-Hirschman Index
,”
Fed. Res. Bull.
,
79
, p.
188
.
58.
Greenberg
,
J. H.
,
1956
, “
The Measurement of Linguistic Diversity
,”
Language
,
32
(
1
), pp.
109
115
. 10.2307/410659
59.
Pitt
,
R.
,
Pirtle
,
W. N. L.
, and
Metzger
,
A. N.
,
2019
, “
Academic Specialization, Double Majoring, and the Threat to Breadth in Academic Knowledge
,”
J. General Edu.
,
66
(
3-4
), pp.
166
191
. 10.5325/jgeneeduc.66.3-4.0166
60.
Nelson
,
J. D.
,
Crupi
,
V.
,
Meder
,
B.
,
Cevolani
,
G.
, and
Tentori
,
K.
,
2017
, “
A Unified Model of Entropy and the Value of Information
,”
Decision Making
,
7
, pp.
119
148
.
61.
Masi
,
M.
,
2005
, “
A Step Beyond Tsallis and Rényi Entropies
,”
Phys. Lett. A.
,
338
(
3–5
), pp.
217
224
. 10.1016/j.physleta.2005.01.094
62.
Ahmed
,
F.
,
Ramachandran
,
S. K.
,
Fuge
,
M.
,
Hunter
,
S.
, and
Miller
,
S.
,
2019
, “
Measuring and Optimizing Design Variety Using Herfindahl Index
,”
Proceedings of the ASME 2019 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. Volume 7: 31st International Conference on Design Theory and Methodology
,
Anaheim, CA
,
Aug. 18–21
, ASME, p.
V007T06A007
. https://doi.org/10.1115/DETC2019-97778
63.
Hurlbert
,
S. H.
,
1971
, “
The Nonconcept of Species Diversity: A Critique and Alternative Parameters
,”
Ecology
,
52
(
4
), pp.
577
586
. 10.2307/1934145
64.
Nemhauser
,
G. L.
,
Wolsey
,
L. A.
, and
Fisher
,
M. L.
,
1978
, “
An Analysis of Approximations for Maximizing Submodular Set Functionsi
,”
Math. Program.
,
14
(
1
), pp.
265
294
. 10.1007/BF01588971
65.
Feige
,
U.
,
Mirrokni
,
V. S.
, and
Vondrak
,
J.
,
2011
, “
Maximizing Non-Monotone Submodular Functions
,”
SIAM J. Comput.
,
40
(
4
), pp.
1133
1153
. 10.1137/090779346
66.
Krause
,
A.
, and
Golovin
,
D.
,
2014
, “Submodular Function Maximization,”
Tractability: Practical Approaches to Hard Problems
,
L.
Bordeaux
,
Y.
Hamadi
, and
P.
Kohli
, eds.,
Cambridge University Press
,
Cambridge, UK
, pp.
71
104
.
67.
Hoffmann
,
S.
,
2008
, “
Generalized Distribution Based Diversity Measurement: Survey and Unification
,”
Faculty of Economics and Management, Otto-von-Guericke University Magdeburg
,
Technical Report 08023
.
68.
Stobbe
,
P.
, and
Krause
,
A.
,
2010
, “
Efficient Minimization of Decomposable Submodular Functions
,”
Advances in Neural Information Processing Systems
,
Vancouver, Canada
,
December
, pp.
2208
2216
.
69.
Kendall
,
M.
,
1962
,
Rank Correlation Methods
(
Theory and Applications of Rank Order-Statistics
),
Hafner Pub. Co
,
New York City, NY
.
70.
Stewart
,
N.
,
Brown
,
G. D.
, and
Chater
,
N.
,
2005
, “
Absolute Identification by Relative Judgment.
,”
Psychol. Rev.
,
112
(
4
), p.
881
. 10.1037/0033-295X.112.4.881
71.
Ahmed
,
F.
,
Fuge
,
M.
, and
Gorbunov
,
L. D.
,
2016
, “
Discovering Diverse, High Quality Design Ideas From a Large Corpus
,”
Proceedings of the ASME 2016 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. Volume 7: 28th International Conference on Design Theory and Methodology
,
Charlotte, NC
,
Aug. 21–24
,
ASME
, p.
V007T06A008
. https://doi.org/10.1115/DETC2016-59926
72.
Ahmed
,
F.
, and
Fuge
,
M.
,
2018
, “
Ranking Ideas for Diversity and Quality
,”
ASME J. Mech. Des.
,
140
(
1
), p.
011101
. 10.1115/1.4038070
73.
Starkey
,
E. M.
,
Hunter
,
S. T.
, and
Miller
,
S. R.
,
2019
, “
Are Creativity and Self-Efficacy at Odds? An Exploration in Variations of Product Dissection in Engineering Education
,”
ASME J. Mech. Des.
,
141
(
1
), p.
012001
. 10.1115/1.4041173
74.
Toh
,
C. A.
, and
Miller
,
S. R.
,
2014
, “
The Impact of Example Modality and Physical Interactions on Design Creativity
,”
ASME J. Mech. Des.
,
136
(
9
), p.
091004
. 10.1115/1.4027639
75.
Ahmed
,
F.
,
Ramachandran
,
S. K.
,
Fuge
,
M.
,
Hunter
,
S.
, and
Miller
,
S.
,
2019
, “
Interpreting Idea Maps: Pairwise Comparisons Reveal What Makes Ideas Novel
,”
ASME J. Mech. Des.
,
141
(
2
), p.
021102
. 10.1115/1.4041856
76.
Ritchey
,
T.
,
2006
, “
Problem Structuring Using Computer-Aided Morphological Analysis
,”
J. Oper. Res. Soc.
,
57
(
7
), pp.
792
801
. 10.1057/palgrave.jors.2602177
77.
Pahl
,
G.
,
Beitz
,
W.
,
Feldhusen
,
J.
, and
Grote
,
K.
,
2007
,
Engineering Design: A Systematic Approach
(
Solid Mechanics and Its Applications
),
Springer
,
London
.
78.
George
,
D.
,
Linnerud
,
B.
, and
Mocko
,
G.
,
2014
, “
Integrated Idea Generation Method for Concept Generation Using Morphological and Options Matrices
,”
Proc. TMCE
,
Budapest, Hungary
,
May
, pp.
1
12
.
You do not currently have access to this content.