The pentagram map was introduced by R. Schwartz in 1992 and is now one of the most renowned discrete integrable systems. In the present paper we prove that this map, as well as all its known integrable multidimensional generalizations, can be seen as refactorization-type mappings in the Poisson-Lie group of pseudo-difference operators. This brings the pentagram map into the rich framework of Poisson-Lie groups, both describing new structures and simplifying and revealing the origin of its known properties. In particular, for multidimensional pentagram maps the Poisson-Lie group setting provides new Lax forms with a spectral parameter and, more importantly, invariant Poisson structures in all dimensions, the existence of which has been an open problem since the introduction of those maps. Furthermore, for the classical pentagram map our approach naturally yields its combinatorial description in terms of weighted directed networks and cluster algebras.