A modal parameter identification method applicable to flutter test data is proposed based on optimized variational mode decomposition（VMD）. Firstly，the natural excitation technique（NExT）is employed to extract impulse response signal from the test data. Then，the decomposition parameters are optimized by using the prior information of the test data combined with the proposed new fitness function. Finally，the target signal is decomposed into multiple monocomponents that each contains an independent oscillation mode. The matrix pencil method is adopted to identify the modal parameters. Numerical simulations and the wind tunnel flutter test demonstrate the effectiveness of the proposed algorism in separating close modes of flutter test data. While associated with the flutter margin method，the optimized VMD can help provide an accurate flutter boundary prediction.