Nonlinear detection is the first step in the process of nonlinear identification，and accurate decisions on whether there are nonlinearities or not in the tested structures are directly related to the choice of methods and theories for the following research and engineering application. Among existing methods，the Hilbert transform nonlinear detection method is widely used，because it has a clear and reliable theory basis，what’s more，other phases in the process of nonlinear identification could be achieved based on the Hilbert transform. However，if the Hilbert transform is calculated by numerical integral methods，obvious truncation errors will be introduced，which will affect the results of nonlinear detection and other tasks of nonlinear identification. Combined with complex analysis theory and rational approximation to compute the Hilbert transform can effectively avoid truncation errors. In this paper，using residual theory，the complex analyzed Hilbert transform is re-derived，and the particular situation in which the real axis exists poles is added. Simultaneously，the Hilbert transform nonlinear detection method is also re-derived，and its misuse in the definition of Hilbert transform is clarified. A novel nonlinear detection criterion is established by reconciling the theory of complex analyzed Hilbert transform with the theory of nonlinear detection based on the Hilbert transform. Numerical examples and experimental studies cover single and multi degrees of freedom systems，linear and nonlinear cases，continuous and discontinuous nonlinear types，which fully verify the accuracy of the re-derived calculation method of complex analyzed Hilbert transform and the reliability of the newly defined Hilbert transform nonlinear detection criterion.