Analysis of long transients and detection of early warning signals of extinction in a class of predator-prey models exhibiting bistable behavior

In this paper, we develop a method of analyzing long transient dynamics in a class of predator-prey models with two species of predators competing explicitly for their common prey, where the prey evolves on a faster timescale than the predators. In a parameter regime near a {\em{singular zero-Hopf bifurcation}} of the coexistence equilibrium state, we assume that the system under study exhibits bistability between a periodic attractor that bifurcates from the singular Hopf point and another attractor, which could be a periodic attractor or a point attractor, such that the invariant manifolds of the coexistence equilibrium point play central roles in organizing the dynamics. To find whether a solution that starts in a vicinity of the coexistence equilibrium approaches the periodic attractor or the other attractor, we reduce the equations to a suitable normal form, and examine the basin boundary near the singular Hopf point. A key component of our study includes an analysis of the long transient dynamics, characterized by their rapid oscillations with a slow variation in amplitude, by applying a moving average technique. We obtain a set of necessary and sufficient conditions on the initial values of a solution near the coexistence equilibrium to determine whether it lies in the basin of attraction of the periodic attractor. As a result of our analysis, we devise a method of identifying early warning signals, significantly in advance, of a future crisis that could lead to extinction of one of the predators. The analysis is applied to the predator-prey model considered in [\emph{Discrete and Continuous Dynamical Systems - B} 2021, 26(10), pp. 5251-5279] and we find that our theory is in good agreement with the numerical simulations carried out for this model.