Gear system is the most widely used transmission system in engineering practice. It is of great significance to ensure the stability of the transmission process. Due to manufacturing，installation error and tooth deformation，it is inevitable that the gear will produce vibration，impact and noise in the meshing process. Proper modification of the helical gear profile can effectively improve the meshing state and the stability of transmission. A time-varying mesh stiffness model considering mixed modification for helical gears is proposed based on the loaded tooth contact analysis method and the validity of the model is verified. Based on the proposed model，the influence of different modifications on the meshing characteristics，such as time-varying mesh stiffness and load distribution on the tooth surface is analyzed. The meshing dynamics model considering the time-varying mesh stiffness of mixed modified gear pair is established，and the influence of different modification types and amounts on the vibration response of gear rotor system is analyzed. The results show that the mixed modification can not only avoid the edge contact，but also greatly reduce the vibration of the gear rotor system in a specific frequency range. This paper can provide theoretical basis for dynamic design of modified helical gears.