We experimentally investigate the rear-wall-expansion cavity, a recent hot flameholder configuration for supersonic combustor in a scramjet engine. Two cavities with different rear-wall-expansion ratios and a conventional cavity are separately employed to stabilize the ethylene jet combustion under the Ma = 2.92 air inflow. Flame luminosity, CH spontaneous emission and OH-PLIF (planar laser-induced fluorescence) are captured to compare the three cavities in terms of flame holding capability and flame structure. The results show that the cavity rear-wall height can drastically affect the flame stabilization and the existence of geometric limit is proven. Under a high equivalence ratio, a rear-wall-expansion cavity helps to relieve the intensive reaction in the combustor and thus prevents combustion mode transition.