We reexamine the vacuum structure of three-dimensional quantum chromodynamics (QCD$_3$) with gauge group $SU(N)$, $N_f$ fundamental quark flavors, and a level-$k$ Chern-Simons term. This analysis can be reliably carried out in the large-$N$, fixed $N_f, k$ limit of the theory, up to certain assumptions that we spell out explicitly. At leading order in the large-$N$ expansion we find $N_f + 1$ distinct, exactly degenerate vacuum superselection sectors with different patterns of flavor-symmetry breaking. The associated massless Nambu-Goldstone bosons are generically accompanied by topological Chern-Simons theories. This set of vacua contains many candidate phases previously proposed for QCD$_3$. At subleading order in the large-$N$ expansion, the exact degeneracy between the different superselection sectors is lifted, leading to a multitude of metastable vacua. If we dial the quark masses, different metastable vacua can become the true vacuum of the theory, leading to a sequence of first-order phase transitions. This intricate large-$N$ dynamics can be captured by the previously proposed bosonic dual theories for QCD$_3$, provided these bosonic duals are furnished with a suitable scalar potential. Interestingly, this potential must include terms beyond quartic order in the scalar fields.