Three-dimensional spatial packaging of interconnected systems with physical interactions (SPI2) design plays a vital role in the functionality, operation, energy usage, and life cycle of practically all engineered systems, from chips to ships. SPI2 design problems are highly nonlinear, involving tightly constrained component placement, governed by coupled physical phenomena (thermal, hydraulic, electromagnetic, etc.), and involve energy and material transfer through intricate geometric interconnects. While many aspects of engineering system design have advanced rapidly in the last few decades through breakthroughs in computational support, SPI2 design has largely resisted automation and in practice requires at least some human-executed design steps. SPI2 system reasoning and design decisions can quickly exceed human cognitive abilities at even moderate complexity levels, thwarting efforts to accelerate design cycles and tackle increasingly complex systems. Existing design methods treat pieces of the SPI2 problem separately without a fundamental systems approach, are sometimes inefficient to evaluate various possible designs, and present barriers to effective adoption in practice. This article explores a vision of a holistic SPI2 design approach needed to develop next-generation automated design methods capable of rapidly producing viable SPI2 design candidates. We review several technical domains related to holistic SPI2 design, discuss existing knowledge gaps and practical challenges, examine exciting opportunities at the intersection of multiple domains that can enable comprehensive exploration of SPI2 design spaces, and present one viable two-stage SPI2 design automation framework. Holistic SPI2 design opens up a new direction of high industrial and societal relevance for the design research community.