We have obtained deep optical images with the Very Large Telescope at ESO of the first well-localized short-duration gamma-ray burst, GRB 050509B. From V and R imaging, initiated ~2 days after the GRB trigger and lasting up to three weeks, we detect no variable object inside the small Swift XRT X-ray error circle down to 2 σ limits of V=26.5 and R=25.1. The X-ray error circle includes a giant elliptical galaxy at z=0.225, which has been proposed as the likely host of this GRB. Our limits indicate that if the GRB originated at z=0.225, any supernova-like event accompanying the GRB would have to be over 100 times fainter than normal Type Ia SNe or Type Ic hypernovae, 5 times fainter than the faintest known Ia or Ic SNe, and fainter than the faintest known Type II SNe. Moreover, we use the optical limits to constrain the energetics of the GRB outflow. Simple models indicate that unless the intrinsic energy in the outflow from GRB 050509B was <<1051 ergs, there was very little radioactive material with efficient decay timescales for generating a large luminosity. These limits strongly constrain progenitor models for this short GRB.Based on observations collected at the European Southern Observatory, Paranal, Chile (ESO program 075.D-0261).