The stability lobe diagram (SLD) is an important expression way of milling stability prediction result. The SLD obtained by only selecting the most flexible mode fails to predict the chatter if the milling process is dominated by multiple modes. To reveal the relationship between the SLD with multiple modes and the SLDs corresponding to each single mode, this paper studies the construction mechanism of the SLD with multiple modes by using a time domain method. First, the milling dynamic model of the tool with multiple modes is established. Then, the numerical method based on the Newton-Cotes rules is used to solve the milling dynamic model with multiple modes whose solution is in the form of the SLD. It shows that the SLD with multiple modes can be approximated by using the lowest envelope of the SLDs corresponding to each single mode. Finally, two study cases are adopted to verify the construction mechanism of the SLD with multiple modes. To verify the correctness of the SLD with multiple modes, a series of milling tests are carried out. The experimental results agree with the simulation results, which mean the proposed time domain method can reveal the construction mechanism of the SLD with multiple modes.