In order to effectively monitor structure state under the circumstance of system input unavailable, an adaptive extended kalman filter with unknown inputs approach, referred to as AEKF-UI, is developed to identify the structural parameters, such as the stiffness, damping and nonlinear hysteretic parameters, as well as the unknown inputs. Further, a new adaptive technique based on improved particle swarm optimization is proposed to on-line track time-varying parameters, leading to structural damage identification, including the time, location and severity. Vibration experiment study was conducted using a base-isolated building model, nonlinear dynamic characteristics of which were described by the Bouc-Wen model. El Centro and Kobe earthquake excitations were used to drive the model by shaking table. Different damage scenarios were simulated and tested during the experiment, and then the AEFK-UI approach was used to track damages and identify unknown inputs based on the measured acceleration responses. The identification results correlate well with the referenced values. It is concluded that the proposed approach is capable of damage tracking and unknown input identification for the base-isolated structure.