Digital video coding is over 60 years old and the first major video coding standard - H.261 - is over 25 years old, yet today there are more patents than ever related to, or evaluated as essential to video coding standards. This paper examines the historical development of video coding standards, from the perspective of when the significant contributions for video coding technology were made, what performance can be attributed to those contributions and when original patents were filed for those contributions. These patents have now expired, so the main video coding tools, which provide the significant majority of coding performance, are now royalty-free. The deployment of video coding tools in a standard involves several related developments. The tools themselves have evolved over time to become more adaptive, taking advantage of the increased complexity afforded by advances in semiconductor technology. In most cases, the improvement in performance for any given tool has been incremental, although significant improvement has occurred in aggregate across all tools. The adaptivity must be mirrored by the encoder and decoder, and advances have been made in reducing the overhead of signaling adaptive modes and parameters. Efficient syntax has been developed to provide such signaling. Furthermore, efficient ways of implementing the tools with limited precision, simple mathematical operators have been developed. Correspondingly, categories of patents related to video coding can be defined. Without discussing active patents, this paper provides the timeline of the developments of video coding and lays out the landscape of patents related to video coding. This provides a foundation on which royalty free video codec design can take place.