Here, we show through a statistical analysis that rogue waves in broadband non-breaking seas are spatially asymmetric. In addition to the top-down asymmetry due to nonlinear effects, we show that the two troughs adjacent to the rogue wave crest are generally of different depths, which is unlike the conventional picture of rogue waves with symmetric fore and aft troughs often obtained from model equations. The rogue-wave trough asymmetry is demonstrated for sea states 4 to 6 on Douglas Sea Scale. Considering the deepest trough leads to approximately 10% increase in the calculation of the mean rogue-wave height compared to previous results for rogue waves reported with symmetric troughs. This implies that estimates of rogue-wave trough-crest amplitudes based on model equations should be re-assessed upward for all realistic oceanic conditions.