The vibration signal of faulty gears simultaneously includes amplitude modulation（AM） and frequency modulation（FM）components. Based on the mathematic modulation model of fault gears，square amplitude demodulation and the frequency demodulation according to the relation of FM with the Bessel function of the first kind are respectively used to establish system of nonlinear equations about AM and FM parameters. Least-square optimization algorithm is then applied to solve these parameters.An accurate separation method of AM and FM is proposed，where discrete spectrum correction technique is employed to improve the correctness. Simulation results show that this method possesses high accuracy and strong immunity from interference，and is superior to Hilbert demodulation and the demodulation based on the energy operator in accuracy. Experimental analyses demonstrate modulation signals separated from different gear faults have obvious features and differences，which provides a diagnosis method for fault gearbox.