Reinforcement learning.

In recent years, biological attacks have emerged as a significant threat to national security, as even a small quantity of pathogenic agents can pose a widespread risk to public health. Experiences from terrorist incidents and bioterrorism simulation exercises conducted in countries such as the United States have demonstrated that effective preparedness against such threats requires robust prevention and control systems, as well as well-designed medical countermeasure supply chains [1]. Existing studies have primarily addressed response supply chain design through three-stage Defender–Attacker–Defender (DAD) models, in which strategic decisions regarding the location of pharmaceutical stockpiles are followed by operational decisions concerning drug distribution [1], [2]. Although robust optimization techniques have commonly been employed to solve these models [3], their applicability to large-scale problems is often limited by computational complexity. Furthermore, many studies simplify the transportation component and overlook practical considerations such as vehicle capacity constraints, delivery time windows, and uncertainties associated with demand and transportation routes. In practice, incorporating transportation decisions corresponds to the classical Vehicle Routing Problem (VRP), which is known to be computationally challenging [4], [5]. To address these limitations, the present study extends the traditional DAD framework by explicitly integrating the transportation layer, resulting in a DAD–VRP structure. Moreover, Reinforcement Learning (RL) is adopted as the solution approach to manage the increased complexity of the proposed model. RL algorithms, including policy-gradient methods, REINFORCE, Actor–Critic, and Q-learning, have been widely recognized as effective techniques for solving dynamic decision-making problems under uncertainty [6–9]. In this study, the REINFORCE algorithm is combined with a recurrent neural network architecture to identify efficient drug distribution routes. This integration enables the model to achieve superior performance compared with conventional optimization-based approaches [8].

The present study extends the three-stage DAD framework proposed in [10]. Within this framework, following decisions regarding warehouse location and inventory allocation, and after the occurrence of an attack, the third stage focuses on the distribution of medical countermeasures. In many previous studies, this stage has been simplified by assuming direct transportation of pharmaceuticals from warehouses to demand centers. However, in real-world emergency response operations, transportation activities are subject to various practical constraints, including limited vehicle capacities, delivery time requirements, and uncertainties in demand and transportation routes.

To better capture these operational realities, the transportation layer is incorporated into the conventional DAD framework, resulting in a new DAD–VRP structure. In addition to addressing demand coverage and inventory management, the proposed model explicitly considers transportation-related constraints such as vehicle capacity limitations, delivery scheduling requirements, and the prioritization of high-risk regions. Consequently, the resulting problem is formulated as a multi-stage nonlinear mixed-integer optimization model. Owing to its computational complexity, solving this model using conventional optimization approaches becomes challenging, thereby motivating the adoption of advanced solution techniques such as RL.

The development of the proposed model requires a set of assumptions that define the operational characteristics and scope of the problem. Accordingly, this section first presents the assumptions adopted in the study, followed by the notation and parameters used in the mathematical formulation. Subsequently, the problem is formulated as a mathematical optimization model that captures the interactions among facility location decisions, inventory allocation, attack scenarios, and transportation operations within the proposed DAD–VRP framework.

Given the inherent characteristics of pharmaceutical distribution problems in emergency situations, including nonlinearity, environmental dynamics, and uncertainty, conventional approaches such as linear programming and heuristic algorithms often exhibit limited effectiveness due to their restricted generalization capability and inability to adapt to real-time environmental changes. Consequently, this study adopts an RL approach, a branch of machine learning in which an agent learns through interaction with its environment and improves its decision-making process via trial-and-error experience [11], [12]. The objective of the agent is to maximize the cumulative reward over time while progressively learning an effective decision policy from observed outcomes.

To implement this approach, a simulation environment was developed consisting of a central warehouse with limited storage capacity, a set of demand nodes, and a fleet of vehicles with predefined capacities. At each decision step, the state of the environment is represented by the remaining demand at each node, the current locations of the vehicles, their residual capacities, and the elapsed time since the beginning of the episode. Based on the observed state, the agent selects an action, which may include traveling to a demand node, returning to the warehouse, or remaining idle. The reward function is designed to encourage successful pharmaceutical delivery, shorter delivery times, and efficient resource utilization. Conversely, delayed responses, unmet demand, and increased operational costs result in negative rewards.

The REINFORCE algorithm is employed to train the agent in the proposed environment. As one of the fundamental policy-gradient methods, REINFORCE directly updates the policy without requiring an approximation of the value function, making it suitable for problems characterized by large state and action spaces [13]. At the end of each episode, the total operational cost, including inventory holding costs, transportation costs, and costs associated with casualties, is calculated and reported as the primary performance measure.

To maintain computational tractability, an initial implementation of the proposed model was conducted using a limited-scale problem instance. The input parameters used in the experiments are summarized in Table 2, where parameter values were selected to ensure a balanced representation of the problem environment. Furthermore, the pseudocode of the proposed solution procedure is presented in Fig. 1.

The proposed model was evaluated using the parameter settings presented in Table 2 and trained over 5,000 episodes. Since fixed values were not assigned to inventory holding costs and several other cost parameters, different runs of the model may produce different objective function values. Therefore, the results reported in this section correspond to a representative execution of the proposed framework.

During the initial training phase with 1,000 episodes, significant fluctuations were observed in the total operational cost, cumulative reward, number of casualties, and inventory holding cost. However, as the number of training episodes increased, the learning process gradually stabilized and converged toward improved solutions. When the number of episodes was increased to 5,000, convergence occurred more rapidly, and the magnitude of fluctuations was substantially reduced, as illustrated in Fig. 2.

Overall, the results indicate that increasing the number of training episodes enables the agent to learn more effective decision policies, leading to lower operational costs and fewer casualties. These findings demonstrate that the REINFORCE algorithm, despite its relatively simple structure, is capable of learning high-quality decision policies in complex and uncertain environments. The observed convergence behavior further confirms the suitability of RL for solving dynamic pharmaceutical distribution problems under bioterrorism-related emergency conditions. Finally, the optimal inventory allocation obtained for each node is reported in

Table 4 compares the performance of the proposed algorithm under two learning settings: with exploration and without exploration. In the absence of exploration, the algorithm converges more rapidly to a stable policy and achieves a lower total operational cost. However, the resulting policy exhibits limited diversity, increasing the likelihood of convergence to locally optimal solutions.

In contrast, incorporating exploration encourages a broader search of the solution space, resulting in more diverse decision policies and improved performance with respect to critical humanitarian measures, particularly casualty reduction. Although the total operational cost is slightly higher in this setting, the exploration mechanism enables the agent to identify more effective long-term strategies that would otherwise remain undiscovered.

These findings highlight the importance of maintaining a balanced exploration–exploitation trade-off in RL - based emergency response systems. While exploitation promotes rapid convergence and cost reduction, controlled exploration can ultimately produce more robust, adaptive, and socially desirable policies, especially in environments characterized by uncertainty and dynamic operational conditions.

Figs. 3-5 illustrate the routes generated for each vehicle during different stages of the training process. In these figures, Node 0 represents the central warehouse, while the highlighted routes indicate the paths selected by the RL agent. The routes shown in red correspond to the decisions chosen by the learned policy. Presenting the routes at different training stages provides insight into the evolution of the agent's behavior and demonstrates how routing decisions become progressively more structured and efficient as learning proceeds. A disaggregated analysis of the cost components throughout the training process revealed that transportation and casualty costs decreased significantly, while inventory holding costs remained relatively stable. This finding indicates that the RL algorithm employed in this study reduced casualties by increasing the initial inventory levels at selected nodes, thereby lowering the contribution of higher-risk cost components. Furthermore, during the final 100 episodes, approximately 72% of the total cost was attributed to inventory holding, 16% to casualties, and 11% to transportation. The algorithm's reward also improved from initially negative values to values approaching the optimal level, indicating a reduction in total cost and an improvement in decision-making quality. Overall, under uncertain conditions, the algorithm was able to learn a near-optimal policy that achieved a balance among inventory, transportation, and casualty costs. In addition, a comparison was conducted to identify the difference between the optimal solution and the solution obtained by the algorithm. All of these results are presented in Fig. 6.

This study presents a novel framework for responding to biological attacks through the integration of the DAD model and the VRP. By employing RL to optimize pharmaceutical distribution, the proposed problem was solved without requiring model linearization, and the corresponding results were obtained. The findings demonstrate the capability of the proposed framework to respond dynamically to environmental changes while maintaining an effective balance among different cost components. Furthermore, the results indicate that the RL algorithm employed in this study achieved satisfactory convergence behavior.

For future research, more advanced RL algorithms, such as PPO and DQN, may be applied and compared with the proposed approach. In addition, the model can be extended to larger-scale problem instances in order to further evaluate its effectiveness, scalability, and computational performance.

Conceptualization, M. S. and F. P.; methodology, M. S. and F. P.; validation, M. S.; formal analysis, M. S.; writing-creating the initial design, M. S.; writing-reviewing and editing, F. P.; monitoring, F. P.; project management, F. P.

No conflict of interest.