We propose a framework for unifying the sl(N) Khovanov-Rozansky homology (for all N) with the knot Floer homology. We argue that this unification should be accomplished by a triply graded homology theory which categorifies the HOMFLY polynomial. Moreover, this theory should have an additional formal structure of a family of differentials. Roughly speaking, the triply graded theory by itself captures the large N behavior of the sl(N) homology, and differentials capture non-stable behavior for small N, including knot Floer homology. The differentials themselves should come from another variant of sl(N) homology, namely the deformations of it studied by Gornik, building on work of Lee. While we do not give a mathematical definition of the triply graded theory, the rich formal structure we propose is powerful enough to make many non-trivial predictions about the existing knot homologies that can then be checked directly. We include many examples where we can exhibit a likely candidate for the triply graded theory, and these demonstrate the internal consistency of our axioms. We conclude with a detailed study of torus knots, developing a picture which gives new predictions even for the original sl(2) Khovanov homology.