The Dicke model with a weak dissipation channel is realized by coupling a Bose-Einstein condensate to an optical cavity with ultranarrow bandwidth. We explore the dynamical critical properties of the Hepp-Lieb-Dicke phase transition by performing quenches across the phase boundary. We observe hysteresis in the transition between a homogeneous phase and a self-organized collective phase with an enclosed loop area showing power-law scaling with respect to the quench time, which suggests an interpretation within a general framework introduced by Kibble and Zurek. The observed hysteretic dynamics is well reproduced by numerically solving the mean-field equation derived from a generalized Dicke Hamiltonian. Our work promotes the understanding of nonequilibrium physics in open many-body systems with infinite range interactions.