The flywheel energy storage system (FESS) is a closely coupled electric-magnetic-mechanical multiphysics system. It has complex nonlinear characteristics, which is difficult to be described in conventional models of the permanent magnet synchronous motor (PMSM) and active magnetic bearings (AMB). A novel nonlinear dynamic model is developed based on the alternative concept. Using back propagation (BP) neural network as a bridge, alternative mapping functions can be built from parametric calculation data of the finite element method (FEM) models. These functions are implemented in a system level simulation of the FESS. As a serial of linear equations, the alternative mapping function can precisely reproduce the electric-magnetic-mechanical characteristics in a satisfied speed and robust. Study of the cogging torque in the PMSM shows a good coincidence with the theory prediction. The current and displacement stiffness coefficients of the AMB are not constants as conventional linear models but change in different winding current and rotor positions. The influence parameters to the critical speed frequency and vibration amplitude are comprehensively studied, including the rotor mass, moment of inertial, eccentric distance, and the mass centroid offset. An operation boundary of the FESS is summarized to describe the feasible power load in different rotor rotation speed and PMSM winding current.