Self-diffusion-driven pattern formation in prey–predator system with complex habitat under fear effect

In the present work, we explore the influence of habitat complexity on the activities of prey and predator of a spatio-temporal system by incorporating self-diffusion. First, we modify the Rosenzweig–MacArthur predator–prey model by incorporating the effects of habitat complexity on the carrying capacity and fear effect of prey and predator functional response. We establish conditions for the existence and stability of all feasible equilibrium points of the non-spatial model, and later we prove the existence of Hopf and transcritical bifurcations in different parametric phase-planes analytically and numerically. The stability of the spatial system is studied, and we discuss the conditions for Turing instability. Selecting suitable control parameter from the Turing space, the existence conditions for stable patterns are derived using the amplitude equations. Results obtained from theoretical analysis of the amplitude equations are justified by numerical simulation results near the critical parameter value. Further, from numerical simulation, we illustrate the effect of diffusion of the dynamical system in the spatial domain by different pattern formations. Thus, our model clearly shows that the fear effect of prey and predator's functional response makes an anti-predator behaviour including habitat complexity which helps the prey to survive in the spatio-temporal domain through diffusive process.