As a generalization form of traditional Fourier transform to deal with non-stationary signals， the fractional Fourier transform often faces the same spectrum leakage problem as the traditional Fourier transform in analyzing stationary signals. In order to reduce the spectrum leakage error and improve the accuracy of fractional Fourier transform and time-frequency analysis， according to the characteristic that the Kaiser window can freely choose the width of main lobe and side lobe， this paper proposes an algorithm of fractional Fourier transform combined with Kaiser window. The function principle of Kaiser window in fractional Fourier transform is discussed. The theoretical formula of time-frequency representation for Kaiser window based fractional Fourier transform is then derived. The time-frequency distribution and time-varying structure parameter identification algorithm of non-stationary signals are obtained. The feasibility of the proposed method is verified by a simulated linear frequency modulation signal and a three-storey model frame excited by non-stationary inputs through a shake table in the laboratory. It is demonstrated that the identified instantaneous frequencies of the proposed method are in good agreement with the theoretical values and involved Kaiser window in fractional Fourier transform does increase the accuracy to identify the time-varying parameters with a certain robustness.