A reliable approximate technique is developed for the determination of the power spectral density (PSD) for the stochastic response of a Duffing oscillator comprising fractional derivative elements. The oscillator excitation is Gaussian white noise. For this purpose, stochastic averaging is applied to the system to obtain a first-order stochastic differential equation governing the evolution of the probability density of the oscillator response amplitude. This procedure leads to a Markovian modeling of the response amplitude, and the Fokker-Plank equation associated with this Markovian model is derived. Furthermore, it is shown that its stationary solution can be determined in a closed form. Moreover, the transfer function of a surrogate linear system endowed with the original fractional derivative term is used to estimate the PSD of the response for the nonlinear oscillator. Pertinent Monte Carlo simulations are included demonstrating the reliability of the proposed technique, even for strongly nonlinear oscillators.