At present，research on dynamic displacement measurement based on computer vision usually requires high-speed and high-resolution camera as well as ideal shooting environment to ensure the performance and accuracy of measurement. However，high cost of the camera，requirements of high image contrast and stable environment during the shooting process limit the wide application of the technology. In this paper，a robust multi-target displacement monitoring method without artificial target is proposed based on the spatio-temporal context algorithm and the optical flow algorithm. The dynamic multi-point displacements of the structure under the interference environment are synchronously measured by smart phone. The frequency sweep experiment of the cantilever sphere model is carried out to test the measurement effect in a certain frequency range. In the experiment，a smart phone is used to shoot the exciting sphere，and the complex background is preserved and the illumination is simulated. Then，the video is processed with the proposed method and the common characteristic optical flow algorithm，and the dynamic displacement results are obtained and compared with the measurement from the displacement meter. The results show that the proposed method has stronger anti-interference ability under the interference of illumination variation. In addition，the maximum displacement deviation of each monitoring point is within 5%.