Coexisting Periodic solutions and their stability for a nonlinear torsional vibration model of a planetary gear train with gear backlashes are studied by using the method of PNF. Some modifications are made for PNF to make it suit the non-smooth and nonlinear planetary gear train in this paper. All of the coexisting periodic solutions of the system with a certain dimensionless rotational speed are carried out by using the method of modified PNF and stability of each periodic solution is studied at the same time. The planetary gear train’s bifurcation characteristic about stability of the periodic trajectory is studied by using PNF at different speed. The study results reveal that a nonlinear planetary gear train with certain parameters may have several coexisting periodic trajectories, some of them may be stable and some of the may be unstable. The system’s motion state will change into chaos in the way of period doubling bifurcation as speed increase.