In comparison with an in-plane tethered satellite system，it is more valuable to investigate an out-of-plane system. Dynamic characteristics and various applications of the system with roll motions have been concentrated. This paper studies out-of plane chaotic behaviors of the tethered satellite. Based on a rigid rod model，both the masses and geometric properties of the satellite rigid bodies and tether are considered. Atmospheric drag and solar pressure in the direction of the roll angle are introduced.Then a dimensionless dynamic equation is deduced. A necessary condition for out-of-plane chaos induced by heteroclinic orbits breaking near saddle points of the perturbed system is analytically acquired using the Melnikov function. A chaotic domain concerning system parameters is calculated. The effects of orbital and structural parameters on the domain are analyzed. Numerical cases regarding out-of-plane roll motions with chaotic/ quasi-periodic characteristics demonstrate the proposed criterion.