The IRS 13E complex is an unusual concentration of massive, early-type stars at a projected distance of ~0.13 pc from the Milky Way's central supermassive black hole Sagittarius A* (Sgr A*). Because of their similar proper motion and their common nature as massive, young stars, it has recently been suggested that IRS 13E may be the remnant of a massive stellar cluster containing an intermediate-mass black hole (IMBH) that binds its members gravitationally in the tidal field of Sgr A*. Here, we present an analysis of the proper motions in the IRS 13E environment that combines the currently best available data with a time line of 10 years. We find that an IMBH in IRS 13E must have a minimum mass of ~104 Msolar in order to bind the source complex gravitationally. This high-mass limit in combination with the absence so far of compelling evidence for a nonthermal radio and X-ray source in IRS 13E make it appear unlikely that an IMBH exists in IRS 13E that is sufficiently massive to bind the system gravitationally.