Dynamic Analysis of Three-Species System with Fear Effect on Prey and Allee Effect on Top Predator

Abstract

Fear-induced behavioral responses and low-density reproductive limitations can affect the stability and persistence of interacting populations. This study investigates a three-species system consisting of prey, middle predator, and top predator, incorporating a fear effect on prey growth and an Allee effect on top predator reproduction. The existence and local stability of biologically feasible equilibria are analyzed using the Jacobian matrix and Routh–Hurwitz criterion. Numerical continuation with MATCONT is employed to identify bifurcations associated with ecological and predation parameters, while numerical simulations are performed using the adaptive fourth–fifth-order Runge–Kutta method. For the selected parameter set, two positive coexistence equilibria are obtained, with the first unstable and the second locally asymptotically stable. The continuation analysis reveals limit-point bifurcations for both the Allee-effect and fear-effect parameters. Further analysis shows that variation in the middle-predator predation rate m₁ produces two limit-point bifurcations, whereas variation in the top-predator predation rate m₂ generates one limit point and a subcritical Hopf bifurcation. The latter indicates the emergence of an unstable periodic orbit near m₂ = 2.232969. Numerical simulations confirm convergence toward the stable coexistence equilibrium with damped oscillatory transients. These results demonstrate that fear, Allee effects, and predation rates can alter the number and stability of coexistence equilibria and induce both equilibrium and oscillatory transitions in the three-species system.