For a active magnetic bearing-supported flywheel rotor system with significant gyroscopic effect, its high nutation mode frequency greatly affects its stability at high rotational speeds. Therefore, it is necessary to develop some advanced control methods which can be used to directly control the dynamics of rotor modes. For such a rotor, since significant decoupling between conical modes and parallel modes, traditional PID controllers are not suitable any more. A new method called modal decoupling control is proposed in this paper. This method is based on mathematic models of flywheel rotor system which can stabilize the active magnetic bearing-supported flywheel rotor system at high rotational speeds. The pricciple of the modal decoupling control is introduced, and the decoupling control ability and effectiveness are analysed numerically. It is shown that the proposed modal decoupling control can separately regulate each modes’ stiffness and damping through decoupling between conical modes and parallel modes, and obviously improve the dynamic behaviours and capacity of anti-interference of active magnetic bearing-supported flywheel rotor system with significant gyroscopic effect.