We present a general theoretical framework for finding the time-optimal unitary evolution of the quantum systems when the Hamiltonian is subject to arbitrary constraints. Quantum brachistochrone (QB) is such a framework based on the variational principle, whose drawback is that it deals with equality constraints only. While inequality constraints can be reduced to equality ones in some situations, there are situations where they cannot, especially when a drift field is present in the Hamiltonian. The drift which we cannot control appears in a wide range of systems. We first develop a framework based on Pontryagin's maximum principle (MP) in order to deal with inequality constraints as well. The new framework contains QB as a special case, and their detailed correspondence is given. Second, using this framework, we discuss general relations among the drift, the singular controls, and the inequality constraints. The singular controls are those that satisfy MP trivially so as to cause a trouble in determining the optimal protocol. Third, to overcome this issue, we derive an additional necessary condition for a singular protocol to be optimal by applying the generalized Legendre-Clebsch condition. This condition in particular reveals the physical meaning of singular controls. Finally, we demonstrate how our framework and results work in some examples.