Aiming at the problems that the incipient weak fault characteristics of the planetary gearbox are difficult to be identified under strong background noise，and the mode number K and penalty factor α of variational mode decomposition（VMD）must be set in advance and cannot be adaptively determined，a modified variational mode decomposition（MVMD）algorithm is proposed.The MVMD method determines the mode number K by adaptive scale space spectrum segmentation of the fault signal，and establishes the maximum kurtosis criterion to automatically select the penalty factor α. On this basis，a novel method for incipient fault diagnosis of planetary gearbox based on the MVMD and adaptive non-convex overlap group shrinkage（MVMD-ANOGS）is proposed. The collected vibration signal of the planetary gearbox is decomposed by MVMD，and then a series of narrow-band modal components are obtained. The proposed ANOGS algorithm performs sparse denoising on the sensitive mode with the maximum kurtosis，thereby highlighting the fault impact characteristics of the mode signal. The envelope demodulation analysis is performed on the denoised mode，and the obvious fault characteristic frequencies are extracted from its envelope spectrum. The feasibility of the proposed method is validated using both the numerical simulation and practical experimental dates of planetary gearbox. Moreover，compared with the traditional VMD，EEMD and fast spectral kurtosis methods，the proposed method can extract clearly weak fault impact features and improve the incipient fault identification accuracy of planetary gearbox.