To establish efficient design methods of a helicopter rotor with low vibration，blade tip shape design is conducted through the combination of surrogated-based optimization and aeroelastic analysis. The aeroelastic equations of rotor blades with unstraight tips are derived. The Kriging surrogate-based models for predicting rotor power，modal damping and vibratory loads are trained. By using the aerodynamic performance and aeroelastic stability as constrains and minimizing the hub vibratory loads as objective，optimization strategy is developed based on an adaptive infill sampling criteria. Taking a scaled model rotor as an example，its performance and vibratory loads are calculated. The aeroelastic model validity is obtained through the comparison with experimental results. Design optimization of the blade sweep，droop and twist distribution from 0.9R to 1.0R is performed. The result indicates that the accuracy of the surrogates of objective function and constraint functions can be improved simultaneously with this adaptive infill sampling criteria based on minimum distance between two training samples. Through the feasibility analysis for optimization results，an optimized blade with double swept and negative/positive anhedral tip is obtained，which results in a 25% reduction of the vibratory hub loads.