Abelian varieties genuinely of $\mathrm{GL}_n$-type

A simple abelian variety $A$ defined over a number field $k$ is called of $\mathrm{GL}_n$-type if there exists a number field of degree $2\dim(A)/n$ which is a subalgebra of $\mathrm{End}^0(A)$. We say that $A$ is genuinely of $\mathrm{GL}_n$-type if its base change $A_{\overline{k}}$ contains no isogeny factor of $\mathrm{GL}_m$-type for $m<n$. This generalizes the classical notion of abelian variety of $\mathrm{GL}_2$-type without potential complex multiplication introduced by Ribet. We develop a theory of building blocks, inner twists and nebentypes for these varieties. When the center of $\mathrm{End}^0(A)$ is totally real, Chi, Banaszak, Gajda, and Krasoń have attached to $A$ a compatible system of Galois representations of degree $n$ which is either symplectic or orthogonal. We extend their results under the weaker assumption that the center of $\mathrm{End}^0(A_{\overline{k}})$ be totally real. We conclude the article by showing an explicit family of abelian fourfolds genuinely of $\mathrm{GL}_4$-type. This involves the construction of a family of genus 2 curves defined over a quadratic field whose Jacobian has trivial endomorphism ring and is isogenous to its Galois conjugate.