Based on the modified fractional order Fourier transform, we introduce a modified fractional correlation operation (MFCO). We propose two systems to implement the MFCO experimentally. These are analyzed theoretically to conclude that the MFCO of two objects equals the conventional correlation between the two objects modified by multiplying each by a quadratic phase factor that contains the parameter that determines the fractional order. The MFCO is applied to a rectangular slit analytically, numerically, and experimentally. We show that the MFCO results in superior correlation peaks compared to those of the conventional correlation operation.