Aiming at the self-excited vibration of rudder structure system in flight, vibration mechanic model for rudder structure system under unsteady aerodynamics with dry friction and clearance are established. Theoretical analysis method is used to study the mechanism of the self-excited vibration phenomenon in flight, and self-excited vibration with high-order harmonics is simulated by numerical integration method. The simulated phenomenon and law in this paper accord well with the recorded results in flight test.