Introduction

A Hybrid Two-Stage DEA and Deep Learning Framework for Efficiency Evaluation of Iranian Stock Exchange Companies

Valizadeh

Omid

Aghakhani

Atefeh

Ghiyasi

Mojtaba

Joshani

Bahareh

Operations Research, Mashhad, Iran. Industrial Management, Mashhad, Iran. Industrial Engineering and Management, Shahrood, Iran. Industrial Management, shahrood, Iran.

This study investigates the efficiency of Iranian stock exchange-listed companies (2007–2023) using a hybrid approach integrating two-stage Data Envelopment Analysis (DEA) with Deep Learning (DL). Traditional DEA evaluates efficiency but struggles with nonlinear patterns and noisy data. By combining DEA with Long Short-Term Memory (LSTM) and TabNet models, this research addresses these limitations. Results reveal that LSTM outperforms TabNet in predicting efficiency scores (MSE: 0.0025 vs. 0.0203), demonstrating its superiority in capturing temporal dependencies in financial data. The hybrid framework enhances accuracy in identifying inefficiencies, optimizing resource allocation, and informing strategic decisions. This methodology bridges DEA’s multi-input/output assessment with Artificial Intelligence (AI)’s predictive power, offering transformative insights for financial analytics.

Two-stage data envelopment analysisDeep learningLong short-term memoryTabNet

1Introduction

In the competitive and dynamic world of financial markets, companies that are members of the stock exchange are always looking for new ways to improve performance, increase productivity, and gain a competitive advantage. Iranian Stock Exchange are companies whose shares are traded on the stock exchange. These companies enter the stock exchange market to finance themselves by selling shares to the public. The stock exchange is known as an official place for buying and selling stocks and other securities, where investors can take advantage of investment opportunities, and companies can also attract their financial resources. The history of the stock exchange in Iran dates back to 1967 when the Tehran Stock Exchange was established. The main purpose of establishing this exchange was to provide a platform for financing companies and also to provide transparency in transactions. The Tehran Stock Exchange initially started operating with limited transactions and a few small companies, but over time, with the growth of the Iranian economy and the expansion of companies, this market flourished. In this regard, it can be said that Data Envelopment Analysis (DEA) is a nonparametric method based on linear programming that was introduced in 1978 by Charnes, Cooper, and Rhodes (CCR) [1]. It can be said that Banker et al. [2]. The concept of returns to scale was considered in this method, and thus, the foundation of a set of performance evaluation methods was formed, which provides managers with appropriate and efficient tools for evaluating industrial, cultural, and economic units, which are called Decision-Making Units (DMUs) in the literature of DEA. The main goal of DEA is to measure different units in terms of efficiency by comparing the optimization of inputs and outputs and, if necessary, identifying inefficient units. After the development of various DEA models, this method was expanded in several fields including banking, education, health, and industry. This paper uses a two-stage DEA communication model to separate efficiency in a two-stage production process where the outputs of the first stage are the inputs of the second stage. In the meantime, Artificial Intelligence (AI) is a process that integrates network equipment, robotics, digital media production, and computers, as well as business practices, technologies, and various daily operations [3]. AI systems and programs have become widespread in various industries and various parts of companies, and in current businesses, with increasing fierce competition, they have changed the way organizations operate and have also provided various opportunities for the marketing process [4]. AI, as one of the most advanced technologies of the present era, has been able to create a huge transformation in many areas. In the field of data analysis, and especially DEA, AI can use new tools to model, optimize, and predict the performance of various units. AI finds its applications in various contexts, in business scenarios. Experts and academics believe that AI is the future of our society. With the advancement of technology, the world has become a network of interconnected networks [5]. The implementation of technology leads to investment in AI to create market intelligence [6]. The applications of AI are not limited to marketing but are also widely used in other sectors, such as medicine, e-commerce, education, law, and manufacturing [7]. It can be said that AI and especially Deep Learning (DL) models, with their complex capabilities in data analysis and pattern recognition, have been able to help overcome some of the limitations and weaknesses of traditional data analysis methods such as DEA. DEA is mainly used to evaluate the efficiency of DMUs in various fields. One of the main weaknesses of this method is the inability to model complex nonlinear relationships and interactions of variables. DEA may also be sensitive to incomplete or noisy data and may not achieve adequate accuracy in simulating the real behavior of the system. It can be said that in traditional DEA, a single model is usually used to evaluate the efficiency of all units using a set of inputs and outputs. In this method, it is assumed that all inputs contribute simultaneously to the production of outputs and the efficiency of the entire process is evaluated. However, two-stage DEA, especially in situations where there are complex and multi-stage processes, can have significant advantages over traditional envelopment analysis. In this method, processes are separated into two or more stages, and the efficiency of each stage is evaluated separately; then, the results of each stage are evaluated separately. It can be said that DL models, using complex neural networks, can simulate and learn complex and nonlinear patterns in data. These capabilities can help to analyze the performance of units more accurately and comprehensively and improve the accuracy of forecasts and assessments. In addition, DL methods can address the analysis of large and complex data that are usually challenged in DEA and provide more optimal solutions. Combining DEA with AI, especially through the use of Machine Learning (ML) and DL, has provided a new and efficient approach to evaluating and optimizing the performance of DMUs. On the other hand, ML and DL enable the analysis and processing of huge volumes of financial, economic, and management data. These technologies can identify hidden patterns in data and provide accurate predictions about market trends, investment risks, and profitable opportunities. Combining the DEA method with AI can provide comprehensive insight into the performance of companies and improve decision-making strategies. In today's world, listed companies are recognized as key institutions in the global economy that use the capital market to finance and develop their activities. Analyzing the performance of these companies and evaluating their efficiency is of particular importance. One of the common methods for evaluating the efficiency and productivity of companies on the stock exchange is DEA. This method helps to compare different units in terms of optimal use of available resources and production of desired outputs. It can be said that DEA has many strengths, including simplicity of implementation, the ability to evaluate multiple inputs and outputs simultaneously, and the ability to simulate different decision-making conditions. However, this method also has limitations. For example, DEA relies on linear and stationary assumptions about the relationships between variables and is sensitive to noisy and incomplete data. DEA is also unable to simulate complex and nonlinear patterns that may exist in the performance of companies. In this context, AI, especially DL, can play a complementary and even alternative role in data analysis. DL models play a complementary and even alternative role in data analysis due to their ability to identify complex patterns and nonlinear relationships in data. DL models can perform better in financial and stock market analysis, especially for companies evaluating their performance, due to their ability to identify complex patterns and nonlinear relationships in data. These models can process large, unstructured and complex data and achieve more accurate predictions about the behavior and performance of companies. In general, combining DL with DEA can provide innovative solutions for analyzing the performance of listed companies and compensate for the weaknesses of traditional methods. The objective of this paper is to investigate efficiency decomposition in a two-stage production process. We will examine the role and application of DEA along with AI techniques in evaluating and optimizing the performance of Iranian stock exchange member companies from 2002 to 2007, and the benefits of using these methods in financial and managerial decision-making will be discussed.

2Literature Review 2.1The Role of Machine Learning in Evaluating the Efficiency of Listed Companies

ML, a subfield of AI, has significantly impacted financial data analysis and the evaluation of corporate efficiency in stock markets. By employing advanced algorithms, it surpasses traditional methods in generating more accurate predictions, enabling analysts to simulate future trends using historical company and market data. This, in turn, enhances investment decision-making. The application of ML in assessing corporate efficiency, particularly in financial markets, allows for more precise analyses and helps address the limitations of traditional approaches, which often rely solely on past data [8].

In this paper, Mehtab and Sen [9] present a very robust and accurate framework of stock price prediction that consists of an agglomeration of statistical, ML, and DL models. They use daily stock price data, collected at five minutes intervals of time, of a very well-known company that is listed in the National Stock Exchange (NSE) of India. Extensive results have been presented on the performance of these models.

Khan et al. [10], in this study, investigated the impact of social media data and financial news on stock market prediction accuracy over a ten-day period. By applying various ML algorithms, they improved model performance through feature selection and the removal of spam tweets. The results show that the highest prediction accuracy was achieved using social media data (80.53%) and financial news (75.16%). Furthermore, predicting the New York and Red Hat stock markets was more challenging. The findings also indicate that New York and IBM stocks are more influenced by social media, while London and Microsoft stocks are more affected by financial news. Among the classifiers, the random forest algorithm demonstrated the best performance, achieving an accuracy of 83.22%.

Sonkavde et al. [11] examine various algorithms, including supervised and unsupervised ML algorithms, ensemble methods, time series analysis algorithms, and DL models, for stock price prediction and classification tasks. It provides an overview of ML and DL models used in the financial sector, establishes a general framework for stock price prediction and classification, and conducts a comparative analysis between ML and DL models.

2.2Data Envelopment Analysis

DEA is a mathematical programming method used to evaluate the efficiency and compare the performance of a group of DMUs. Each DMU represents an organization or a unit that operates with multiple inputs and outputs. The goal of DEA is to identify efficient units and inefficient units in comparison to others. DEA uses linear programming to calculate the efficiency of units through various models. One of the key features of DEA is that it does not rely on a fixed or predefined benchmark for comparison. Instead, it evaluates units based on their relative performance and the ratio of inputs to outputs [12].

DEA is a method that separates efficient units from inefficient ones by measuring their efficiency. Pourmahmoud and Norouzi Bene [13], In this study, modeling has been applied to oil companies listed on the Tehran Stock Exchange. By combining DEA with discriminant analysis, the research introduces a new model that designs hyperplanes to distinguish between two groups (Successful and unsuccessful). The results indicate that this method can assist in predicting the status of companies and provide a strategy for identifying paths to success or avoiding failure.

Amin and Hajjami [14] deal with the role of alternative optimal solutions existing in the DEA models for cross-efficiency evaluation in portfolio selection. The paper shows that incorporating alternative optimal solutions for constructing a cross-efficiency matrix improves the result of the mean-variance portfolio selection method. This improvement means that building portfolios with lower risk and higher expected returns is possible when alternative optimal solutions are considered. The proposed method in this paper is applied to stock portfolio selection in the Tehran stock market.

3Relational Two-Stage Data Envelopment Analysis Model

DEA is commonly used to evaluate the efficiency of DMUs. However, in many practical systems, the production process is divided into multiple sequential stages, where the outputs of one stage serve as the inputs for the next stage. The two-stage DEA model takes this relationship into account and measures the efficiency of each stage independently while also calculating the overall efficiency of the system. The two-stage DEA model provides valuable insights for evaluating complex systems. It allows for the independent assessment of each stage’s efficiency, enabling a more accurate identification of inefficiencies. By considering both individual stages and the overall system, it aids decision-makers in improving processes and allocating resources more effectively. Its flexibility makes it well-suited for complex systems, offering a clearer understanding of the interdependencies between stages and enhancing the overall performance evaluation [15].

This model is used to assess the efficiency of DMUs that have a process divided into two sub-processes.

Ek = max ∑ ur Yrk,

s. t. ∑ = 1,

∑ ur Yrj − ∑ vi Xij ≤ 0, j = 1, . . . , n,

r=1 i=1

∑ wp Zpj − ∑ vi Xij ≤ 0, j = 1, . . . , n,

p=1 i=1 ∑ ur Yrj − ∑ wp Zpj ≤ 0, j = 1, . . . , n,

r=1 p=1

ur, vi, wp ≥ ε, r = 1, . . . , s, i = 1, . . . , m, p = 1, . . . , q.

∗ ∗ ∗

After the optimal multipliers ur ,vi , and wp are solved, the efficiencies are obtained subsequently as

Ek = ∑ ur Yrk,

1 ∗ ∗

Ek = ∑ wp Zpk/ ∑ vi Xik,

p=1 i=1

2 ∗ ∗

Ek = ∑ ur Yrk/ ∑ wp Zpk,

r=1 p=1

In this model:

I. Ek : Represents the overall efficiency of DMU.

II. ur : Are the weights for the outputs of the first stage.

III. vi : Are the weights for the inputs of the first stage.

IV. wp : Are the weights for the intermediate products Zpj, which are used as both outputs of the first sub-process and inputs of the second.

V. Yrk : Are the outputs in the second stage related to the overall production of the system.

VI. Zpj: Are intermediate products that are outputs in the first sub-process and inputs in the second.

VII. Xik : Represents the inputs in the first stage.

The constraints in the model ensure that the production process meets certain conditions, like the relationship between inputs and outputs and bounds for the efficiency measures.

The goal of the model is to maximize the overall efficiency E (k ), which is defined as a function of both the efficiencies of the first and second sub-processes. The equation decomposes the efficiency of the whole process into the product of the efficiencies of the individual stages.

I. The first step: It receives a set of inputs (X) and generates intermediate outputs (Z).

II. The second stage receives the intermediate outputs (Z) as input and produces the final output (Y).

III. The efficiency of the whole process is equal to the product of the efficiency of two steps.

EK = EK × EK.

3.1Long Short-Term Memory

Long Short-Term Memory (LSTM) is an advanced type of Recurrent Neural Network (RNN) specifically designed for processing sequential data and addressing the issues related to the forgetting of information over time. In traditional RNNs, models struggle to effectively retain and utilize long-term information, which becomes particularly problematic in complex, sequential tasks. LSTM overcomes this challenge through its unique structure, which includes input, forget, and output gates. Each gate dynamically determines which information should be stored, updated, or discarded in the memory. These features enable LSTM to maintain long-term dependencies within the data and leverage them for accurate and effective predictions [16]. LSTM has found widespread applications in areas such as natural language processing (e.g., machine translation, sentiment analysis, and language modeling), time series forecasting (Such as financial trend prediction), speech recognition, and the analysis of audio and visual data. In summary, LSTM, with its ability to preserve and process information over extended periods, is recognized as a powerful tool for modeling complex, time-dependent problems that require long-term memory retention [17].

3.2TabNet

TabNet is an advanced DL model specifically designed for processing tabular data, introduced by Google in 2020. Unlike many traditional DL methods that face challenges when dealing with structured data, TabNet has demonstrated remarkable performance in analyzing tabular data through its innovative attention mechanism. This attention mechanism allows TabNet to dynamically select important features and focus solely on them, effectively extracting useful information from the data. Instead of processing all data simultaneously, TabNet examines the features progressively at different stages, enabling the model to obtain more precise representations of the data. This process, using adaptive attention, helps the model learn complex relationships between features and make accurate predictions without the need for intricate feature engineering [18]. TabNet has shown exceptional performance in tasks such as classification, time series forecasting, risk analysis, and fraud detection. Its ability to handle complex datasets and its scalability makes it a powerful and efficient tool for problems involving structured data. Overall, TabNet, by combining the strengths of DL with high adaptability, provides a robust solution for analyzing and modeling complex tabular data [19].

4Analysis of Results

In this section, the efficiency scores of 130 stock exchange companies between 2006 and 2023 are first calculated using two-stage DEA, and then the efficiency scores are examined using DL.

4.1Two-Stage Data Envelopment Analysis

In the first stage, the efficiency scores of stock exchange companies from 2007 to 2013 (130 companies per fiscal year) are analyzed using the DEA method. DEA method is very sensitive to the selection of inputs and outputs, and there is no general agreement in this regard for the selection of these variables; in this study, by reviewing the relevant literature and interviewing experts, Two variables, capital and interest expense are classified as inputs, while two variables, sales and operating profit are categorized as intermediate outputs. Additionally, two variables, net profit and Market Value of Equity, are defined as outputs. Table 1 shows a statistical summary of the companies.

Table 1

Statistical summary of variables.

This table is shown in the PDF version of the article.

Table 2

Average efficiency of 130 stock exchange companies using two-stage data envelopment analysis.

This table is shown in the PDF version of the article.

4.2Combined Approach of Two-Stage Data Envelopment Analysis and Deep Learning

first, the input and target variables for DL were defined. Specifically, the performance scores of companies were selected as the target variable, and 29 financial variables were designated as inputs. The complete list of these variables is provided in Table 3. Subsequently, data cleaning was performed, which included handling missing values and removing outliers. The data was then split into training (80%) and testing (20%) sets to enable DL models to effectively analyze the correlation between input features and the target variable. After training and testing, three evaluation metrics MAE, MSE, and RMSE were employed to assess model performance (All analyses were conducted using Python 3.10.1.).

Table 3

List of inputs and target.

This table is shown in the PDF version of the article.

Total assets, total assets (Previous year), total Efficiency scores liabilities, capital, shareholders' equity, sales, sales (Previous year), operating profit, interest expense, taxes, net profit, annual stock return, market value of equity, stock trading volume, number of outstanding shares, ROA, ROE, MTB, GROWTH, ILLIQ, RETRISK, SIZE, LEV, WACC, VOL, RETURN, FVALUE, EVA

After training the LSTM and TabNet models, the evaluation metrics are presented in Table 4.

Table 4

Evaluating machine learning models.

This table is shown in the PDF version of the article.

The evaluation results of the models demonstrated that the LSTM model outperformed the TabNet model in predicting efficiency scores. The evaluation metrics included MSE with values of 0.0025 for LSTM compared to 0.0203 for TabNet, MAE with values of 0.0255 versus 0.0691 and RMSE with values of 0.0496 versus 0.1425. These results indicate higher accuracy and lower error in the predictions made by LSTM compared to TabNet, which is statistically and practically indicative of the significant superiority of this model. This superiority can be attributed to the inherent ability of LSTM to model long-term patterns and complex temporal dependencies in financial time-series data, which, in this study, comprised multi-year financial indicators such as Total Assets, Sales, Net Profit, ROA, ROE, and other key variables. The recurrent structure of LSTM, leveraging memory mechanisms, enables it to effectively learn temporal relationships and hidden trends in the data, which is critical for financial data characterized by a time-series nature.

In Fig. 1 and Fig. 2, an Actual vs. Predicted plot is presented for both methods.

Figure 1

Actual vs. Predicted for TabNet model.

This figure is shown in the PDF version of the article.

Figure 2

Actual vs. Predicted for long short-term memory model.

This figure is shown in the PDF version of the article.

6Conclusion

This study explored the integration of two-stage DEA with artificial intelligence, specifically DL, to evaluate the efficiency of Iranian stock exchange-listed companies from 2007 to 2023. The results demonstrated that the LSTM model outperformed TabNet in predicting efficiency scores, achieving lower errors (MSE: 0.0025 vs. 0.0203; MAE: 0.0255 vs. 0.0691; RMSE: 0.0496 vs. 0.1425). This superiority stems from LSTM’s ability to capture long-term temporal dependencies in financial time-series data, addressing nonlinear complexities often challenging for traditional DEA. By combining DEA’s strengths in multi-input/output efficiency assessment with DL’s pattern recognition capabilities, this hybrid approach offers a robust framework for identifying inefficiencies, optimizing resource allocation, and enhancing decision-making in dynamic financial markets.

However, the study has limitations, including its focus on a specific timeframe (2007–2023) and a fixed sample of 130 companies, which may affect generalizability. Future research could extend the analysis to broader periods, incorporate additional contextual variables (e.g., macroeconomic indicators), and test advanced DL architectures like Transformers. Practically, this methodology provides actionable insights for managers and investors to benchmark performance, mitigate risks, and leverage AI-driven analytics for strategic planning. Ultimately, the fusion of DEA and AI represents a transformative step toward data-driven efficiency evaluation in complex, multi-stage financial systems.

Conflict of Interest

The authors declare that they have no conflict of interest regarding the publication of this manuscript.

Data Availability

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Funding

This research received no external funding.

References 1

Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision making units. European journal of operational research, 2(6), 429–444. https://doi.org/10.1016/0377-2217(78)90138-8

Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some models for estimating technical and scale inefficiencies in data envelopment analysis. Management science, 30(9), 1078–1092. https://doi.org/10.1287/mnsc.30.9.1078

Sutton, R. S. (1998). Reinforcement learning, a Bradford book. MIT press. https://mitpress.mit.edu/9780262193986/reinforcement-learning/

Campbell, C., Sands, S., Ferraro, C., Tsao, H. Y. J., & Mavrommatis, A. (2020). From data to action: How marketers can leverage AI. Business horizons, 63(2), 227–243. https://doi.org/10.1016/j.bushor.2019.12.002

Jarek, K., & Mazurek, G. (2019). Marketing and artificial intelligence. Central european business review, 8(2), 46-55. https://www.ceeol.com/search/article-detail?id=775030

Stone, M., Aravopoulou, E., Ekinci, Y., Evans, G., Hobbs, M., Labib, A., … ., & Machtynger, L. (2020). Artificial intelligence (AI) in strategic marketing decision-making: a research agenda. The bottom line, 33(2), 183–200. https://doi.org/10.1108/BL-03-2020-0022

Pitt, C., Paschen, J., Kietzmann, J., Pitt, L. F., & Pala, E. (2023). Artificial intelligence, marketing, and the history of technology: Kranzberg’s laws as a conceptual lens. Australasian marketing journal, 31(1), 81–89. https://doi.org/10.1177/18393349211044175

Bouasabah, M. (2024). A performance analysis of machine learning algorithms in stock market prediction, compared to traditional indicators. Information systems and technological advances for sustainable development (pp. 264–271). Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-75329-9_29

Mehtab, S., & Sen, J. (2020). A time series analysis-based stock price prediction using machine learning and deep learning models. International journal of business forecasting and marketing intelligence, 6(4), 272–335. https://doi.org/10.1504/IJBFMI.2020.115691

Khan, W., Ghazanfar, M. A., Azam, M. A., Karami, A., Alyoubi, K. H., & Alfakeeh, A. S. (2022). Stock market prediction using machine learning classifiers and social media, news. Journal of ambient intelligence and humanized computing, 13(7), 3433–3456. https://doi.org/10.1007/s12652-020-01839-w

Sonkavde, G., Dharrao, D. S., Bongale, A. M., Deokate, S. T., Doreswamy, D., & Bhat, S. K. (2023). Forecasting stock market prices using machine learning and deep learning models: A systematic review, performance analysis and discussion of implications. International journal of financial studies, 11(3), 94. https://doi.org/10.3390/ijfs11030094

Jamshidi, M., Sanei, M., & Mahmoodirad, A. (2022). An uncertain allocation model for data envelopment analysis: A case in the Iranian stock market. Scientia iranica, 29(6), 3434–3454. https://doi.org/10.24200/sci.2020.55047.4046

Pourmahmoud, J., & D, N. B. (2022). Evaluation of general two-stage network systems in the presence of undesirable and non-discretionary data. Iranian journal of operations research, 13(2), 83–95. chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://iors.ir/journal/article-1-800-en.pdf

Amin, G. R., & Hajjami, M. (2021). Improving DEA cross-efficiency optimization in portfolio selection. Expert systems with applications, 168, 114280. https://doi.org/10.1016/j.eswa.2020.114280

Bencová, T., & Boháčiková, A. (2022). DEA in performance measurement of two-stage processes: Comparative overview of the literature. Economic studies, 31(5), 111-129. https://www.ceeol.com/search/article-detail?id=1048505

Lindemann, B., Müller, T., Vietz, H., Jazdi, N., & Weyrich, M. (2021). A survey on long short-term memory networks for time series prediction. Procedia cirp, 99, 650–655. https://doi.org/10.1016/j.procir.2021.03.088

Van Houdt, G., Mosquera, C., & Nápoles, G. (2020). A review on the long short-term memory model. Artificial intelligence review, 53(8), 5929–5955. https://doi.org/10.1007/s10462-020-09838-1

Arik, S. Ö., & Pfister, T. (2021). Tabnet: Attentive interpretable tabular learning. Proceedings of the AAAI conference on artificial intelligence (pp. 6679–6687). PKP Publishing Services Network. https://doi.org/10.1609/aaai.v35i8.16826

Shah, C., Du, Q., & Xu, Y. (2022). Enhanced TabNet: Attentive interpretable tabular learning for hyperspectral image classification. Remote sensing, 14(3), 716. https://doi.org/10.3390/rs14030716