A recent study has revealed that the low carrier density electron gas (2DEG) induced at the interface of EuO and (111) KTaO$_3$ exhibits a broken symmetry phase with a strong in-plane anisotropy of the resistivity. We present a minimal tight binding model of this (111) 2DEG, including the large spin-orbit coupling from the Ta ions, which reveals a hexagonal Fermi surface with a highly enhanced 2$k_F$ electronic susceptibility. We argue that repulsive electronic interactions, together with a ferromagnetic EuO substrate, favor a magnetic stripe instability leading to a partially gapped Fermi surface. Such a stripe state, or its vestigial nematicity, could explain the observed transport anisotropy. We propose a $k\cdot p$ theory for the low energy $j=3/2$ states, which captures the key results from our tight-binding study, and further reveals the intertwined dipolar and octupolar modulations underlying this magnetic stripe order. We conclude by speculating on the relation of this stripe order to the superconductivity seen in this material.