Sets preserved by a large subgroup of the special linear group

Let $E$ be a subset of the the affine plane over a finite field $\mathbb{F}_q$. We bound the size of the subgroup of $SL_2(\mathbb{F}_q)$ that preserves $E$. As a consequence, we show that if $E$ has size $\ll q^\alpha$ and is preserved by $\gg q^\beta$ elements of $SL_2(\mathbb{F}_q)$ with $\beta\geq 3\alpha/2$, then $E$ is contained in a line. This result is sharp in general, and will be proved by using a point-line incidence bound in $\mathbb{F}_q^3$.