Game-Theoretic Analysis of Adaptive Intrusion Detection Systems under Detection Uncertainty

Authors

  • Mohammad Malekraeisi * Department of Computer Engineering, Bi.C., Islamic Azad University, Birjand, Iran.
  • Ali Akbar Kiani Department of Computer Engineering, Bi.C., Islamic Azad University, Birjand, Iran.
  • Hamid Saadatfar Department of Computer Engineering, Faculty of Electrical and Computer Engineering, University of Birjand, Birjand, Iran. https://orcid.org/0000-0002-6130-8450

https://doi.org/10.22105/raise.vi.105

Abstract

As computer networks expand and cyberattacks grow more sophisticated, Building Intrusion Detection Systems (IDS) that can withstand intelligent, adaptive adversaries has become a core problem in network security. Much of the existing literature assumes attacker behavior that is static or fully specified in advance, ignoring the uncertainty inherent to any real detection process, including false alarms and missed detections. Such simplifications make it difficult to capture how a defensive system and a learning attacker actually behave against each other over time. We address this gap with a game-theoretic framework, built on repeated games, that models the ongoing interaction between an IDS and an intelligent attacker. Both players are represented as autonomous agents that follow memory-one strategies and adjust their behavior through gradual learning. To make the model more realistic, detection uncertainty enters the security payoffs directly through the True Positive Rate (TPR) and the False Positive Rate (FPR), and a separate control parameter captures how quickly the attacker adapts relative to the defender, so we can study fast-adapting attacker scenarios. Numerical simulations produce a phase diagram over the space of attacker learning speed and IDS detection capability, revealing three distinct regimes: attacker dominance, adaptive competition, and a secure state. Faster-learning attackers push the security boundary outward, so the IDS needs stronger detection capability to stay in the secure regime. These findings offer practical guidance for designing and tuning adaptive  IDS  in dynamic, hostile network environments.

Keywords:

Intrusion detection system, Network security, Adaptive attacker, Repeated games, Multi-agent learning

References

  1. [1] Liu, H., & Lang, B. (2019). Machine learning and deep learning methods for intrusion detection systems: A survey. Applied sciences, 9(20), 4396. https://doi.org/10.3390/app9204396
  2. [2] Ferrag, M. A., Maglaras, L., Moschoyiannis, S., & Janicke, H. (2020). Deep learning for cyber security intrusion detection: Approaches, datasets, and comparative study. Journal of information security and applications, 50, 102419. https://doi.org/10.1016/j.jisa.2019.102419
  3. [3] Khraisat, A., Gondal, I., Vamplew, P., & Kamruzzaman, J. (2019). Survey of intrusion detection systems: techniques, datasets and challenges. Cybersecurity, 2(1), 1–22. https://doi.org/10.1186/s42400-019-0038
  4. [4] Ayub, M. A., Johnson, W. A., Talbert, D. A., & Siraj, A. (2020). Model evasion attack on intrusion detection systems using adversarial machine learning. In 2020 54th annual conference on information sciences and systems (CISS) (pp. 1-6). IEEE. https://doi.org/10.1109/CISS48834.2020.1570617116
  5. [5] Ahmed, M., Mahmood, A. N., & Hu, J. (2016). A survey of network anomaly detection techniques. Journal of network and computer applications, 60, 19–31. https://doi.org/10.1016/j.jnca.2015.11.016
  6. [6] Manshaei, M. H., Zhu, Q., Alpcan, T., Bacşar, T., & Hubaux, J. P. (2013). Game theory meets network security and privacy. Acm Computing Surveys (CSUR), 45(3), 1-39. https://doi.org/10.1145/2480741.2480742
  7. [7] Zhang, H., Mi, Y., Liu, X., Zhang, Y., Wang, J., & Tan, J. (2023). A differential game approach for real-time security defense decision in scale-free networks. Computer networks, 224, 109635. https://doi.org/10.1016/j.comnet.2023.109635
  8. [8] Bai, R., Lin, H., Wu, X., Li, M., & Jia, W. (2025). On the computation of mixed strategies for security games with general defending requirements. Artificial intelligence, 341, 104297. https://doi.org/10.1016/j.artint.2025.104297
  9. [9] Chen, J., Xu, Z., Li, Y., Yu, C., Song, J., Yang, H., ... & Wu, Y. (2024). Accelerate multi-agent reinforcement learning in zero-sum games with subgame curriculum learning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 10, pp. 11320-11328). https://doi.org/10.1609/aaai.v38i10.29011
  10. [10] Lan, Y., & Hu, Q. (2023). Intrusion detection and response of network node attacks based on game theory. In Proceedings of the 2023 International Conference on Artificial Intelligence, Systems and Network Security (pp. 386-390).
  11. [11] Estévez-Pereira, J. J., Fernández, D., & Novoa, F. J. (2020). Network anomaly detection using machine learning techniques. In Proceedings (Vol. 54, No. 1, p. 8). MDPI. https://doi.org/10.3390/proceedings2020054008
  12. [12] Apruzzese, G., Andreolini, M., Ferretti, L., Marchetti, M., & Colajanni, M. (2022). Modeling realistic adversarial attacks against network intrusion detection systems. Digital threats: Research and practice (dtrap), 3(3), 1–19. https://doi.org/10.1145/3469659
  13. [13] Ahmad, Z., Shahid Khan, A., Wai Shiang, C., Abdullah, J., & Ahmad, F. (2021). Network intrusion detection system: A systematic study of machine learning and deep learning approaches. Transactions on emerging telecommunications technologies, 32(1), e4150. https://doi.org/10.1002/ett.4150
  14. [14] Ho, E., Rajagopalan, A., Skvortsov, A., Arulampalam, S., & Piraveenan, M. (2022). Game theory in defence applications: A review. Sensors, 22(3), 1032. https://doi.org/10.3390/s22031032
  15. [15] Wang, Z., Xu, S., Xu, G., Yin, Y., Zhang, M., & Sun, D. (2020). Game Theoretical Method for Anomaly-Based Intrusion Detection. Security and communication networks, 2020(1), 8824163. https://doi.org/10.1155/2020/8824163
  16. [16] FathimaAH, N., Khraisat, A., Ibrahim, S., & Li, G. (2025). Adaptive memory replay for network intrusion detection: Tackling data drift and catastrophic forgetting. Computer networks, 111712. https://doi.org/10.1016/j.comnet.2025.111712
  17. [17] Malloy, T., & Gonzalez, C. (2023). Learning to defend by attacking (and vice-versa): Transfer of learning in cybersecurity games. 2023 IEEE european symposium on security and privacy workshops (EuroS&PW) (pp. 458-464). IEEE. https://doi.org/10.1109/EuroSPW59978.2023.00056

Published

2025-12-20

Issue

Vol. 2 No. 4 (2025): Research Annals of Industrial and Systems Engineering

Section

Articles

How to Cite

Malekraeisi, M. ., Kiani, A. A. ., & Saadatfar, H. . (2025). Game-Theoretic Analysis of Adaptive Intrusion Detection Systems under Detection Uncertainty. Research Annals of Industrial and Systems Engineering, 2(4), 306-318. https://doi.org/10.22105/raise.vi.105

Similar Articles

1-10 of 39

You may also start an advanced similarity search for this article.