Aiming at the problem of low energy concentration caused by the fixed window effect in STFT, a fault diagnosis method of rolling bearing under variable speed conditions based on Adaptive Window Rotation Optimization Short-Time Fourier Transform (AWROSTFT) is proposed. Firstly, Variational Mode Decomposition (VMD) is used to reduce the noise of the original vibration signal. Particle Swarm Optimization (PSO) is used to solve the complex problem of VMD parameter selection. Secondly, a series of rotation operators are adaptively matched to the horizontal window in STFT by using the tangent idea so that the rotation direction of the window is close to or even equal to the instantaneous frequency modulation, and the energy concentration of time-frequency representation is improved. Finally, the instantaneous frequency extracted by the spectral peak detection method is divided by the frequency transformation curve. The result is matched with the fault characteristic coefficient of the bearing to realize the fault diagnosis of the rolling bearing under variable speed conditions. The results of simulation and experiment signals show that the proposed method can take into account the advantages of PSO-VMD and AWROSTFT. The angle between the signal and the window function is zero globally through the adaptive rotation window with the idea of tangency to improve the energy concentration, sharpen the time-frequency ridge line, and realize the fault diagnosis of rolling bearings under variable speed conditions.