Matter distribution around clusters is highly anisotropic from their being the nodes of the cosmic web. Clusters' shape and the number of filaments they are connected to, i.e., their connectivity, should reflect their level of anisotropic matter distribution and must be, in principle, related to their physical properties. We investigate the influence of the dynamical state and the formation history on both the morphology and local connectivity of about 2400 groups and clusters of galaxies from the large hydrodynamical simulation IllustrisTNG at z=0. We find that the mass of groups and clusters mainly influences the geometry of the matter distribution: massive halos are significantly more elliptical, and more connected to the cosmic web than low-mass ones. Beyond the mass-driven effect, ellipticity and connectivity are correlated and are imprints of the growth rate of groups and clusters. Both anisotropy measures appear to trace different dynamical states, such that unrelaxed groups and clusters are more elliptical and more connected than relaxed ones. This relation between matter anisotropies and dynamical state is the sign of different accretion histories. Relaxed groups and clusters are mostly formed long time ago, and slowly accreting matter at the present time. They are rather spherical and weakly connected to their environment, mostly because they had enough time to relax and, hence, lost the connection with their preferential directions of accretion and merging. In contrast, late-formed unrelaxed objects are highly anisotropic with large connectivities and ellipticities. These groups and clusters are in formation phase and must be strongly affected by the infalling of materials from filaments.