The present work proposes the damage localization method based on LU decomposition of the proportional flexibility matrix. Firstly, several low-frequency modal frequencies and mode shapes are identified based on the vibration responses; then the proportional flexibility matrices before and after damage are extracted respectively. Secondly, LU decomposition of the change of proportional flexibility matrix is implemented. Finally, the damage index is extracted based on the U matrix and curvature method to locate the damage. Simulations are implemented based on a 20-story shear building model; moreover, a 6-story lumped-mass frame model is built in the laboratory, then its shaking table tests and pulse tests are implemented also. Both numerical and experimental results of single and multiple damage cases indicate that the proposed method can locate the damage very well. With responses before and after damage, the method does not require a structural finite element model; thus, establishing a numerical model and model updating can be avoided. Besides, the flexibility matrix can be estimated with sufficient accuracy by only using several low-frequency modes which can be identified well. All of these lay a good foundation for engineering application of the proposed method.