We show under what conditions the complex computing general Ext-groups carries the structure of a cyclic operad such that Ext becomes a Batalin-Vilkovisky algebra. This is achieved by transferring cyclic cohomology theories for the dual of a (left) Hopf algebroid to the complex in question, which asks for the notion of contramodules introduced along with comodules by Eilenberg-Moore half a century ago. Another crucial ingredient is an explicit formula for the inverse of the Hopf-Galois map on the dual, by which we illustrate recent categorical results and answer a long-standing open question. As an application, we prove that the Hochschild cohomology of an associative algebra A is Batalin-Vilkovisky if A itself is a contramodule over its enveloping algebra A \otimes A^op. This is, for example, the case for symmetric algebras and Frobenius algebras with semisimple Nakayama automorphism. We also recover the construction for Hopf algebras.