Dynamic Analysis of a Predator–Prey Model with Group Defense in Prey and Cooperative Hunting in Predators

Abstract

This article discusses a single-prey and single-predator model by incorporating two behavioral mechanisms, namely group defense in prey modeled through a Holling type IV response function and cooperative hunting in predators represented by a predation rate dependent on predator density. Through system analysis, up to four equilibrium points are obtained mathematically. Among these, three equilibria are biologically feasible under typical parameter values, corresponding to total extinction, predator extinction, and coexistence states. The total extinction equilibrium is always unstable, while the stability of the predator extinction and coexistence equilibria depends on the predator attack rate. Numerical simulations in the form of phase portraits were obtained by varying the parameters related to the predator attack rate. The simulation results show various dynamic behaviors, including predator extinction, asymptotically stable coexistence between prey and predators, and bistability. Numerical continuation analysis identifies a subcritical Hopf bifurcation at α=0.4595, confirmed by a positive first Lyapunov coefficient, as well as a saddle-node at α= 0.0478 and transcritical bifurcations α=3.0505 that alter equilibrium structure and stability. These findings demonstrate how prey group defense and predator cooperation can generate bistability and abrupt transitions between extinction and coexistence, accompanied by damped oscillatory dynamics near critical parameter values.