Complexity originates from the tendency of large dynamical systems to organize themselves into a critical state, with avalanches or "punctuations" of all sizes. In the critical state, events which would otherwise be uncoupled become correlated. The apparent, historical contingency in many sciences, including geology, biology, and economics, finds a natural interpretation as a self-organized critical phenomenon. These ideas are discussed in the context of simple mathematical models of sandpiles and biological evolution. Insights are gained not only from numerical simulations but also from rigorous mathematical analysis.