Non-classical virtual knots may have non-isomorphic upper and lower quandles. We exploit this property to define the quandle difference invariant, which can detect non-classicality by comparing the numbers of homomorphisms into a finite quandle from a virtual knot's upper and lower quandles. The invariants for small-order finite quandles detect non-classicality in several interesting virtual knots. We compute the difference invariant with the six smallest connected quandles for all non-evenly intersticed Gauss codes with 3 and 4 crossings. For non-evenly intersticed Gauss codes with 4 crossings, the difference invariant detects non-classicality in 86% of codes which have non-trivial upper or lower counting invariant values.