IDSs are conventionally grouped into two families: signature-based and anomaly-based. Signature-based systems detect known attacks reliably but struggle against new or variant ones[1]. Anomaly-based systems take the opposite approach, modeling normal traffic so that deviations flag unknown attacks, at the cost of a higher false-alarm rate [11]. Machine learning has pushed detection accuracy considerably further, with neural networks, support vector machines, and deep-learning models all applied to this problem [3]. Even so, these approaches generally treat the IDS as a passive observer and largely overlook how an attacker's adaptive behavior affects the defense's long-term performance [12]. Many of them also focus narrowly on detection metrics, giving only limited attention to the operational costs that false alarms and detection errors actually impose [13]. Game theory occupies a distinct place in cybersecurity research because it models defenders and attackers strategically rather than in isolation: each side's behavior is analyzed as a response to the other's strategy [6]. Security games of this kind have found wide application in critical-infrastructure protection, network security, and cyber defense more broadly [7]. Much of this work relies on static, complete-information games, where both players are assumed to know the full game structure and payoffs [14], an assumption that rarely survives contact with real networks. Repeated games have been proposed to capture longer-term interactions, but many still assume perfect, error-free detection [15], which again leaves little room to study how false alarms and detection errors actually shape attacker behavior.

Combining game theory with reinforcement learning and multi-agent learning to model dynamic security games is an active research area. Here, the defender and the attacker are both cast as learning agents that update their strategies based on environmental feedback [6], opening the door to studying convergence, instability, and adaptive competition directly. A number of studies look at adaptive attackers that alter their attack patterns to evade detection [2], but few examine systematically how the attacker's learning speed shapes the security boundary. As before, most of this work reports point-wise numerical results rather than a regime analysis that maps secure, competitive, and insecure regions across the parameter space [7].

Taken together, this body of work has advanced the use of game theory and adaptive learning in network security, but gaps remain: detection uncertainty is rarely built explicitly into game-theoretic models, the attacker's relative learning speed is seldom examined for its effect on security stability, and the lack of a comprehensive regime analysis makes it hard to pin down exactly when an IDS can hold a secure state against an intelligent attacker. The framework proposed here is aimed at closing these gaps, offering a systematic, multi-agent-learning-based analysis of the IDS-attacker interaction under realistic detection uncertainty.

We model the IDS–attacker interaction as a two-player repeated game in which each side behaves as an autonomous, rational agent. Player one, the IDS, monitors the network and decides whether to activate its defensive mechanisms; player two, the attacker, chooses strategies to extract maximum benefit from its attacks. At every round of the game, the IDS chooses between:

I. Normal: Routine network monitoring, with no active defense engaged.

II. Active defense: Engaging advanced detection and defense mechanisms at some operational cost.

The attacker, in turn, chooses between:

I. No attack: Holding off.

II. Attack: Launching an attack on the network.

The game repeats over time, with each agent updating its decision based on the system's previous state and the feedback it receives; this is what allows the model to capture long-term, adaptive interaction [14]. The payoffs mirror the real costs and benefits faced by each side. A successful attack earns the attacker a benefit G and costs the defender a loss L. Engaging active defense costs the IDS an operational amount Cₑ, and mounting an attack costs the attacker Cₐ. When an attack is caught, the attacker pays a penalty P, and the IDS collects a matching reward. Together, these terms balance deterrence, operational cost, and potential attack payoff, a structure common throughout the security-game literature [6].

Unlike much of the prior work, which treats detection as complete and error-free, this study builds the IDS's inherent uncertainty directly into the model through two parameters:

I. TPR: The probability that an attack is correctly identified when defense is active.

II. FPR: The probability of a false alarm when no attack has actually occurred.

When active defense is engaged, but no attack actually takes place, the IDS bears an extra cost, with probability equal to the FPR, standing in for wasted resources, service disruption, or the need for human intervention. When an attack does occur while defense is active, it is detected and blocked with probability TPR and succeeds with probability (1 − TPR). Framing the payoffs this way, in expectation, makes it possible to study the system's long-run behavior under realistically imperfect detection [11]. Both agents follow memory-one strategies: each decision depends only on what happened in the previous round, so the game's state space reduces to the four combinations of the IDS's and attacker's prior actions. Such a structure keeps the model tractable while still letting it capture reactive, adaptive behavior, an assumption widely used in the repeated-games literature [16].

Both agents update their behavior through gradual learning to maximize expected long-term payoff. To capture asymmetries in adaptation speed, we introduce a control parameter ω, defined as the ratio of the attacker's learning speed to the IDS's. Large ω values describe attackers that outpace the IDS in responding to environmental change, while small values reflect a defender with the learning advantage [17]. Put together, this study addresses the long-term behavior of a repeated security game played by learning agents, with payoffs shaped by detection uncertainty and asymmetric learning speeds: specifically, under what conditions the IDS can hold a secure state against an adaptive attacker, and how the security boundary moves across the parameter space.

As noted above, the system's state at step t is set by the combination of actions taken in the previous step, giving four possible states:

S = {(N, NA), (N, A), (D, NA), (D, A)}, (1)

Here N and D denote normal monitoring and active defense, respectively, while NA and A denote no attack and attack. The IDS's memory-one policy is given by the probability vector

p = [p1, p2, p3, p4], (2)

where pᵢ = Pr (D | sᵢ) is the probability of activating defense in state sᵢ, and the attacker's policy is defined analogously as

q = [q1, q2, q3, q4]. (3)

with qᵢ = Pr (A | sᵢ). To keep dimensionality manageable while still reflecting realistic security behavior, we restrict the attacker's policy class: in the state offering a favorable, near-guaranteed opportunity, the attacker attacks almost always, and in the high-risk state it almost never does. Concretely, two components of q are held fixed, and two are learnable, and symmetrically, only a subset of the IDS's parameters is learnable as well. Such a design keeps the learning process stable and makes the resulting analysis easier to interpret. Because the policies are memory-one, the game's dynamics reduce to a first-order Markov chain over the state space S: given p and q, the transition matrix M (p, q) follows directly, and the stationary distribution π satisfies

π = πM, ∑ πi = 1. (4)

We compute π numerically by power iteration until it converges, and then obtain the expected long-term payoffs of the IDS and the attacker as expectations over the state distribution:

U (p, q) = ∑ π r (s ;p, q), (5)

D i D i

UA(p, q) = ∑ πi rA(si;p, q), (6)

Here rD and rA denote the stage payoffs, which already incorporate imperfect detection through TPR and FPR along with the operational and false-alarm costs introduced in Section 3.

The IDS seeks to maximize UD while the attacker seeks to maximize UA, giving a two-objective game in which each agent nudges its policy parameters in the direction that improves its own long-term payoff:

θD → θD + αD∇θ UD, (7)

θA → θA + αA∇θ UA. (8)

Differentiating U analytically is awkward once the stationary distribution and the TPR/FPR-dependent payoffs are involved, so we approximate the gradient numerically via finite differences:

∂U U(θk + ε) − U(θk − ε)

≈ , (9)

∂θk 2ε where ε is a small step size used for the approximation.

To capture fast-adapting attackers, we set the attacker's learning rate relative to the IDS's as

αA = ω αD, ω ∈ (0,1], (10)

As ω grows, the attacker reacts to change more quickly, and the competition intensifies. This single parameter drives the regime, or phase-diagram, analysis presented later.

To keep the numerics stable and the policies valid probabilities, we clip the parameters to [ε, 1 − ε] after every update:

pi → clip(pi, ϵ, 1 − ϵ), qi → clip(qi, ϵ, 1 − ϵ), (11)

Clipping keeps probabilities from getting stuck at 0 or 1 and avoids numerical problems when computing the stationary distribution. The fixed parameters within the restricted policy class are similarly re-constrained at every iteration so that the intended policy structure is preserved.

Convergence is tracked by monitoring how much the stationary distribution, or the payoffs, change across successive iterations. In practice, the implementation runs a fixed number of iterations to reach steady state, computing the long-term payoff from π at each learning step.

The overall learning process is presented below in algorithmic form.

Algorithm 1. Gradual learning of the IDS and the attacker within a repeated-game framework with imperfect detection.

1: Initialize p and q

2: Enforce probabilistic constraints on p and q

3: for t = 1 to T do

4: Compute stationary distribution π for Markov process M (p, q)

5: Compute long-term payoffs:

6: UD (p, q) → IDS payoff

7: UA (p, q) → Attacker payoff

8: Approximate IDS gradient using finite differences:

9: ∇p UD (p, q)

10: Update IDS strategy:

11: p → p + αD ∇p UD

12: Approximate attacker gradient using finite differences:

13: ∇q UA (p, q)

14: Set attacker learning rate:

15: αA = ω αD

16: Update attacker strategy:

17: q → q + αA ∇q UA

18: Apply clipping and policy constraints (if required)

19: end for

20: Set p* → p, q* → q

21: Return p*, q*, UD (p*, q*), UA (p*, q*)

Because ω and the TPR/FPR pair together govern the system's behavior, the game can settle into qualitatively different regimes depending on where these parameters sit. Running the algorithm above over a grid of (ω, TPR) values produces a regime map delineating the attacker-dominance, adaptive-competition, and secure regions, giving a systematic way to determine when the IDS, despite operational costs and detection errors, can still maintain deterrence against a learning attacker.

To evaluate the model, we simulated the IDS–attacker interaction as a two-agent repeated game, with both sides following memory-one strategies so that each round's decision depends only on the previous round's state. Payoffs were computed in expectation, folding in TPR and FPR so that detection uncertainty is built directly into the simulation. The payoff parameters, including the benefit of a successful attack (G), the defender's loss on a breach (L), the cost of active defense (Cd), the cost of mounting an attack (Ca), and the detection penalty (P), were varied over reasonable ranges for the analysis. Adaptive behavior was modeled by updating each strategy's selection probability gradually, based on cumulative payoff, and the control parameter ω governed the attacker's learning speed relative to the defender, covering scenarios where the attacker learns more slowly, at the same pace, or faster than the IDS. Simulations ran over a long time horizon so that behavioral stability and payoff convergence could both be assessed. We evaluated performance using the two agents' expected long-term payoffs together with a regime analysis over the parameter space: extracting each agent's average steady-state payoff and mapping out the attacker-dominance, adaptive-competition, and secure regions, which together indicate what conditions are needed for stable security.

We first checked the convergence behavior of both agents' policies and long-term payoffs to rule out any influence from transient learning effects on the final results. Fig. 1 shows that the IDS's long-term payoff, across different ω values, settles into a steady state after an initial transient, evidence that the update mechanism, which relies on maximizing long-term payoff through the Markov chain's stationary distribution, is numerically stable. Fig. 2 shows the same pattern for the attacker's long-term payoff, which converges without notable oscillation in the later stages of learning. Together, these results indicate that the two agents' adaptive interaction settles into a stable equilibrium, making meaningful long-term analysis possible.

We also examined how the IDS's and the attacker's probabilistic policies converge. Figs. 3 and 4 show that the action-selection probabilities across the different game states settle at fixed values, confirming that the learned policies are stable, further evidence that the framework consistently captures stable behavior for both

Here we examine how the IDS's detection quality shapes the game's strategic dynamics, through a sensitivity analysis over the TPR and the learning-speed parameter ω. Table 1 reports only representative and boundary points, those marking a shift in behavioral regime, to keep the discussion focused. At low detection quality (TPR = 0.5, for instance), the IDS's expected payoff stays negative regardless of ω, while the attacker earns either a positive payoff or only a marginal loss. Such a pattern is a clear signature of attacker dominance: attacking remains strategically worthwhile, and active defense fails to deter. As the TPR rises further, the system passes through a transition region. Around TPR = 0.6 in Table 1, depending on ω, the regime shifts from attacker dominance into a mixed, arms-race-like state, where both players' payoffs are negative, yet neither has a clear incentive to disengage; strategic updates instead settle into an unstable, competitive equilibrium. This finding shows that improving detection quality alone does not guarantee security; the attacker's adaptive behavior still plays a decisive role. Once TPR crosses roughly 0.72, Table 1 shows the regime shifting qualitatively into the secure region: the IDS's expected payoff turns positive while the attacker's payoff drops substantially, the signature of stable deterrence, where attacking is no longer strategically attractive. The learning-speed parameter ω also matters for where these boundaries fall: at intermediate TPR values, Table 1 shows that changing ω alone can push the system between the mixed and secure regimes. Security stability, in other words, depends not just on raw detection quality but also on how the IDS's learning process weighs past experience. Overall, these results suggest that network security is not simply a direct function of the TPR but emerges from the interplay between detection quality and adaptive learning dynamics. For IDS design, the practical takeaway is that stable security requires crossing specific thresholds on both the detection and the learning dimensions, not just improving one in isolation.

To build a fuller, more systematic picture of the long-run IDS–attacker interaction, we carried out a regime analysis over the two-dimensional (TPR, ω) space, where ω captures the attacker's learning speed relative to the IDS, and TPR captures the IDS's intrinsic detection quality. This analysis lets us study the joint effect of the attacker's learning capability and the defense's detection accuracy on network-security stability. The system's behavior falls into three qualitative regimes: attacker-dominant, competitive (arms-race), and secure. Attacker dominance shows up mainly at low TPR, where the IDS cannot build effective deterrence and attacking still pays off economically; the defense fails against an adaptive attacker even though the IDS itself is learning. As detection quality improves, the system moves through a transition region that behaves like an arms race. Both agents keep updating their strategies; neither gains a lasting edge, and network security ends up hinging heavily on the learning parameters and operational conditions. Table 2 quantifies these transition thresholds and shows that the TPR needed to enter the secure regime rises with ω. For small ω, a slowly adapting attacker, the IDS can secure the system at a fairly moderate TPR. As ω grows and the attacker reacts more quickly to past experience, the minimum TPR needed to stay secure climbs substantially. This coupling between ω and TPR is a key design insight: raising detection accuracy alone, without regard for the attacker's learning capability (or vice versa), does not guarantee stable security. Network security instead depends on a careful balance between detection quality and both players' learning dynamics. More broadly, the regime analysis here shows that the long-term IDS-attacker interaction can be mapped as distinct qualitative regions in the (TPR, ω) space, a framework that not only explains the underlying behavior but also yields practical thresholds for designing and tuning adaptive IDSs, linking game-theoretic analysis directly to network-security engineering.

Complementing the regime analysis above, this section looks at the IDS's long-term payoff map over the (TPR, ω) space, which lets us weigh security stability against the economic efficiency of defense together, going beyond the purely regime-based view. The heat map in Fig. 6 shows the IDS's payoff generally improving as TPR rises; at low TPR, the payoff is mostly negative, reflecting an ineffective defense against an adaptive attacker. These low-payoff regions largely coincide with the attacker-dominant regime in Fig. 5; under these conditions, the IDS is doing poorly both from a security standpoint and a cost-benefit one. At intermediate TPR (matching the mixed/arms-race regime in Fig. 5), the IDS's payoff hovers near zero or turns slightly positive, a borderline situation in which defense and attack are roughly balanced and small changes in learning parameters or operational cost can tip the advantage either way. Practically speaking, this is where network security is most fragile and where IDS parameters need the most careful tuning.

Fig. 6 also reveals that being in the secure regime (per Fig. 5) does not always mean the IDS's net payoff is maximal. For some (ω, TPR) combinations, stable deterrence is achieved and the attacker's payoff drops sharply, yet the costs of active defense and adaptive learning still eat into the IDS's net payoff, a reminder that “secure” and “economically optimal” are not the same thing. That said, the regions of high IDS payoff in Fig. 6 line up structurally with the secure regime in Fig. 5, which supports the regime analysis: the IDS's payoff peaks mostly where detection quality is high, and the attacker's adaptation speed is kept in check. Payoff maps, then, work well as a quantitative complement to the qualitative regime picture. In short, designing an adaptive IDS has to look past stable security alone and weigh cost–benefit trade-offs at the same time. Pairing regime analysis with long-term payoff maps gives engineers a fuller basis for decision-making, one that can guide practical IDS configurations balanced between security and economic efficiency.

We compared the proposed framework against the classical approaches reported in [6] and [14]; Table 3 summarizes the comparison.

As Table 3 shows, the traditional static-game models in [6] and [14] treat the attacker as passive and predictable, while the framework proposed here captures the attacker's dynamic, adaptive behavior directly. Conventional approaches also tend to be overly optimistic about the security state; by explicitly accounting for the attacker's relative learning speed ω, the proposed model draws more realistic security boundaries.