Differential categories provide an axiomatization of the basics of differentiation and categorical models of differential linear logic. As differentiation is an important tool throughout quantum mechanics and quantum information, it makes sense to study applications of the theory of differential categories to categorical quantum foundations. In categorical quantum foundations, compact closed categories (and therefore traced symmetric monoidal categories) are one of the main objects of study, in particular the category of finite-dimensional Hilbert spaces FHilb. In this paper, we will explain why the only differential category structure on FHilb is the trivial one. This follows from a sort of in-compatibility between the trace of FHilb and possible differential category structure. That said, there are interesting non-trivial examples of traced/compact closed differential categories, which we also discuss. The goal of this paper is to introduce differential categories to the broader categorical quantum foundation community and hopefully open the door to further work in combining these two fields. While the main result of this paper may seem somewhat "negative" in achieving this goal, we discuss interesting potential applications of differential categories to categorical quantum foundations.