Surface impurities and contamination often seriously degrade the properties of two-dimensional materials such as graphene. To remove contamination, thermal annealing is commonly used. We present a comparative analysis of annealing treatments in air and in vacuum, both ex situ and "pre-situ", where an ultra-high vacuum treatment chamber is directly connected to an aberration-corrected scanning transmission electron microscope. While ex situ treatments do remove contamination, it is challenging to obtain atomically clean surfaces after ambient transfer. However, pre-situ cleaning with radiative or laser heating appears reliable and well suited to clean graphene without undue damage to its structure.