A fault feature extraction method based on group sparsity and improved iterative shrinkage threshold optimization in frequency domain（GSRF）is proposed to solve the issues in rolling bearing diagnosis about difficulty in mathematical model determination，sparse constraints and optimization algorithm selection. The vibration signals are converted into the frequency domain and the variables are divided by overlapping rules. The least square regression model with group bridge constraint is constructed to screen impact related variables accurately. The iterative reweighting coefficient is introduced to simplify the equation，so that the sparse signal in frequency domain can be solved by iterative shrinkage-thresholding algorithm. The envelope spectrum analysis of reconstructed sparse signal in time domain is carried out to extract the fault features. The experimental results show that the proposed algorithm is superior to the traditional group sparse LASSO combined with L21 norm constraint. GSRF can effectively extract weak fault features and achieve bearing fault diagnosis in the sparse domain.