In order to solve the problems existing in variational mode decomposition（VMD）that the number of mode decomposition layers is difficult to determine and the target mode is difficult to select，an improved signal decomposition method，i.e.，variational mode extraction（VME）is introduced. And a fault diagnosis method of rotating machinery based on S transform（ST）and VME is proposed. Based on a new criterion that the spectrum overlap between the desired mode and the residual signal is minimized，VME method decomposes the signal into two layers，i.e.，the desired mode and the residual signal. Aiming to obtain the desired mode containing abundant fault features，the signal is processed by S transform to determine the frequency band of the fault features. Consequently，the central frequency initial value of the desired mode of VME is obtained. In order to extract the fault characteristic frequency from the desired mode，the squared envelope spectrum（SES）analysis method is applied to process the desired mode. The analysis results of the simulated signal and actual vibration signals show that the proposed ST-VME method can successfully extract the valuable desired mode and the accurate fault characteristic frequency to achieve rotating machinery fault diagnosis. Besides，compared with VMD，the ST-VME method is more targeted and easier to implement.