We study random magnetic systems emphasizing the concept of gauge invariance and gauge-invariant disorder (frustration) introduced by Toulouse and Anderson. We formulate our models in a gauge-invariant manner and introduce gauge-invariant correlation functions to isolate the effects of gauge-invariant disorder. Specifically, we study the Ising and X-Y models in two and three dimensions in a frozen distribution of frustrations. Using duality transformations, we obtain expressions for the energetics of frustrations and their effect on correlations. We study simple configurations of frustrations quantitatively. In addition we reformulate the quenching procedure in terms of frustrations.