We discuss the behaviour of the JHKL'M' colors of L and T dwarfs, based on photometric and spectroscopic data obtained at the United Kingdom Infrared Telescope in Hawaii. We have measured the first accurate M' photometry for L and T dwarfs. The K-M' colors of T dwarfs are much bluer than predicted by published models, suggesting that CO may be more abundant than expected, as has been found spectroscopically for the T6 dwarf Gl 229B. We also find that K-L' increases monotonically through most of the M, L, and T subclasses, but it is approximately constant between types L6 and T5, due to the onset of methane absorption at the blue edge of the L' bandpass. The JHK colors of L dwarfs show significant scatter, suggesting variations in the amount and properties of photospheric dust, and indicating that it may not be possible to associate a unique effective temperature with a given L spectral type. The H-K colors of the later T dwarfs also show some scatter which we suggest is due to variations in pressure-induced molecular hydrogen opacity, which is sensitive to gravity (or age for a brown dwarf) and metallicity.