Understanding how transcription factors (TFs) regulate mammalian gene expression in space and time is a central topic in biology. To activate a gene, a TF has first to diffuse in the available space of the nucleus until it reaches a target DNA sequence or protein (target site). This eventually results in the recruitment of the whole transcriptional machinery. All these processes take place in the mammalian nucleoplasm, a highly organized and dynamic environment, in which some complexes transiently assemble and break apart, whereas others appear more stable. This diversity of dynamic behaviors arises from the number of biomolecules that make up the nucleoplasm and their pairwise interactions. Indeed, interactions energies that span several orders of magnitude, from covalent bounds to transient and dynamic interactions can shape nuclear landscapes. Thus, the nuclear environment determines how frequently and how fast a TF contacts its target site, and indirectly gene expression. How exactly transient interactions are involved in the regulation of TF diffusion is unclear, but are reflected by live cell imaging techniques such as fluorescence correlation spectroscopy, fluorescence recovery after photobleaching or single-particle tracking. Overall, the macroscopic result of these microscopic interactions is almost always anomalous diffusion, a phenomenon widely studied and modeled. Here, we review the connections between the anomalous diffusion of a TF and the microscopic organization of the nucleus, including recently described topologically associated domains and dynamic phase-separated compartments. We propose that anomalous diffusion found in single particle tracking (SPT) data result from weak and transient interactions with dynamic nuclear substructures, and that SPT data analysis would benefit form a better description of such structures.