A number of Anomalous X-ray Pulsars (AXPs) have recently been detected in the optical/IR wavelengths. We use their inferred brightness to place general constraints on any model for this emission within the magnetar framework. We find that neutron-star surface emission cannot account for the observations and that the emission must be magnetospheric in origin. We propose a model for the optical/IR emission in which a distribution of energetic electrons in the neutron-star magnetosphere emits synchrotron radiation. This model can naturally reproduce the observed brightness and the rising spectra of AXPs as well as the observed pulsations at the stellar spin frequency and the correlation of the IR flux with their bursting activity.