In practical engineering，the tuned mass damper（TMD）will exhibit nonlinear characteristics when the displacement is large，so it is of great significance to study the effect of its nonlinear characteristics on control performance. In this paper，the control performance of TMD under the condition of nonlinear fifth-order stiffness is studied. Based on the traditional linear design method，an improved design method of nonlinear TMD is proposed，and the analytical solution of the jump frequency which can be used to improve the design is obtained. Simulation results show that compared with the linear-based design method，the improved design method of nonlinear TMD can improve the control performance of TMD.