Holonomy algebras arise naturally in the classical description of Yang-Mills fields and gravity, and it has been suggested, at a heuristic level, that they may also play an important role in a non-perturbative treatment of the quantum theory. The aim of this paper is to provide a mathematical basis for this proposal. The quantum holonomy algebra is constructed, and, in the case of real connections, given the structure of a certain C-star algebra. A proper representation theory is then provided using the Gel'fand spectral theory. A corollory of these general results is a precise formulation of the ``loop transform'' proposed by Rovelli and Smolin. Several explicit representations of the holonomy algebra are constructed. The general theory developed here implies that the domain space of quantum states can always be taken to be the space of maximal ideals of the C-star algebra. The structure of this space is investigated and it is shown how observables labelled by ``strips'' arise naturally.