Rogue waves is the name given by oceanographers to isolated large amplitude waves, that occur more frequently than expected for normal, Gaussian distributed, statistical events. Rogue waves are ubiquitous in nature and appear in a variety of different contexts. Besides water waves, they have been recently reported in liquid Helium, in nonlinear optics, microwave cavities, etc. The first part of the review is dedicated to rogue waves in the oceans and to their laboratory counterpart with experiments performed in water basins. Most of the work and interpretation of the experimental results will be based on the nonlinear Schrödinger equation, an universal model, that rules the dynamics of weakly nonlinear, narrow band surface gravity waves. Then, we present examples of rogue waves occurring in different physical contexts and we discuss the related anomalous statistics of the wave amplitude, which deviates from the Gaussian behavior that were expected for random waves. The third part of the review is dedicated to optical rogue waves, with examples taken from the supercontinuum generation in photonic crystal fibers, laser fiber systems and two-dimensional spatiotemporal systems. In particular, the extreme waves observed in a two-dimensional spatially extended optical cavity allow us to introduce a description based on two essential conditions for the generation of rogue waves: nonlinear coupling and nonlocal coupling. The first requirement is needed in order to introduce an elementary size, such as that of the solitons or breathers, whereas the second requirement implies inhomogeneity, a mechanism needed to produce the events of mutual collisions and mutual amplification between the elementary solitons or wavepackets. The concepts of “granularity” and “inhomogeneity” as joint generators of optical rogue waves are introduced on the basis of a linear experiment. By extending these concepts to other systems, rogue waves can be classified as phenomena occurring in the presence of many uncorrelated “grains” of activity inhomogeneously distributed in large spatial domains, the “grains” being of linear or nonlinear origin, as in the case of wavepackets or solitons.