Due to the structural limitations in the flexible rotor system supported by Active Magnetic Bearings， displacement sensor cannot be installed inside the bearings， which results in the non-collocation effect. The non-collocation of rotor system will not only decrease the vibration performance of rotor system， but also cause system instability. In this paper， the dynamic model of the active magnetic bearing-flexible rotor system is established by the finite element method. Through the dynamics characteristic，open loop transfer function， frequency response， and root locus method， the features of the non-collocation system are studied.The specific positions of the unstable mode eigenvalues are analyzed by the multi-input and multi-output state equation. Solutions including reducing the controller gain and adaptive plug-in notch filter are proposed to suppress the unstable modes because of non-collocation effect. Experiments are designed in a flexible rotor system test rig. The experiments’ results show that the rotor system’s stability is influenced significantly by non-collocation effect. The method proposed in the passage can suppress the instability of bending modes caused by different potential effects to a certain extent.