We study the characteristic polynomials Z(U, θ) of matrices U in the Circular Unitary Ensemble (CUE) of Random Matrix Theory. Exact expressions for any matrix size N are derived for the moments of |Z| and Z/Z*, and from these we obtain the asymptotics of the value distributions and cumulants of the real and imaginary parts of log Z as N-->∞. In the limit, we show that these two distributions are independent and Gaussian. Costin and Lebowitz  previously found the Gaussian limit distribution for Im log Z using a different approach, and our result for the cumulants proves a conjecture made by them in this case. We also calculate the leading order N-->∞ asymptotics of the moments of |Z| and Z/Z*. These CUE results are then compared with what is known about the Riemann zeta function ζ (s) on its critical line Re s= 1/2, assuming the Riemann hypothesis. Equating the mean density of the non-trivial zeros of the zeta function at a height T up the critical line with the mean density of the matrix eigenvalues gives a connection between N and T. Invoking this connection, our CUE results coincide with a theorem of Selberg for the value distribution of log ζ(1/2+iT) in the limit T-->∞. They are also in close agreement with numerical data computed by Odlyzko  for large but finite T. This leads us to a conjecture for the moments of |ζ(1/2+it) |. Finally, we generalize our random matrix results to the Circular Orthogonal (COE) and Circular Symplectic (CSE) Ensembles.