We construct a field theory to describe energy averaged quantum statistical properties of systems which are chaotic in their classical limit. An expression for the generating function of general statistical correlators is presented in the form of a functional supermatrix non-linear σ-model where the effective action involves the evolution operator of the classical dynamics. Low-lying degrees of freedom of the field theory are shown to reflect the irreversible classical dynamics describing relaxation of phase space distributions. The validity of this approach is investigated over a wide range of energy scales. As well as recovering the universal long-time behavior characteristic of random matrix ensembles, this approach accounts correctly for the short-time limit yielding results which agree with the diagonal approximation of periodic orbit theory.