Virasoro constraints on the KP hierarchy, arising in matrix models, are studied by reexpressing them in terms of dressing operators of the hierarchy. There exists a one-parameter family of Virasoro representations on the KP hierarchy (depending on a number J which can be identified as the conformal weight of an abstract bc system). The respective full invariance algebra is the ``Borel'' subalgebra of W∞(J), which we describe as an extension of the ``wedge'', or higher spin, algebra Bλ = J-J2 by the L2 Virasoro generator. Reductions of these structures of the N-KdV hierarchies are performed explicitly.Permanent address: Theory Division, P.N. Lebedev Physics Institute, SU-117 924 Moscow, USSR.