Deep Learning Models for Price Prediction in Travertine Stone Mines: A Comparison of LSTM, Transformer, and Hybrid Models

  • Haniye MoazeniDepartment of Industrial Engineering, Qom University of Technology, Qom, Iran.
  • Amirhesam KamalpourDepartment of Statistics, Mathematics, and Computer Science, Allameh Tabataba’i University, Tehran, Iran.

Abstract

This study aims to develop and evaluate four deep learning models for forecasting the monthly prices offered by stone suppliers in Iran. Given the simultaneous influence of temporal factors (such as exchange rate, inflation, fuel price, and order volume) and static attributes (including block quality, brand reputation, and cooperation history), each model was designed with a dual-input structure to separately process sequential and non-sequential features, which are then integrated at a later stage. The implemented architectures include LSTM, Bi-LSTM, Transformer, and a hybrid Transformer+LSTM+CLS model. The models were trained and evaluated using data collected from five different mines over several months, ensuring robustness and generalizability across diverse supply sources and time periods. Model performance was assessed using key evaluation metrics including Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), the coefficient of determination (R²), and validation loss. The results indicate that the Transformer model achieved the highest accuracy, with the lowest prediction errors and the best generalization capability. The hybrid model also performed comparably well, making it a robust alternative for more complex forecasting tasks. In contrast, the Bi-LSTM model underperformed and is less recommended for this application. Overall, the findings highlight that combining attention-based architectures with sequential analysis provides an effective solution for price forecasting in data-driven and competitive industrial contexts.

Keywords:Price forecastingDeep learning modelsTransformer architectureTime series forecastingStone

suppliers, Data-driven.

1. Introduction

Copper blocks (raw stone blocks extracted from mines) are among the most important primary products in the building stone industry, playing a decisive role in the profitability of the supply chain, exports, and the development of domestic markets. The price of this product is influenced by multiple factors such as stone type, geographic location of the mine, transportation costs, extraction quality, and market conditions. The continuous fluctuations of these factors make accurate price forecasting a fundamental challenge in production planning, sales pricing, and investment decision-making. Traditional pricing methods are mostly based on expert experience and static analyses, which, due to data complexity and multiple influencing factors, can be inefficient or unreliable. Recent advances in machine learning algorithms and increased access to mining and market data have paved the way for developing intelligent price prediction models. Machine learning algorithms, with their ability to identify complex and nonlinear patterns in data, have achieved significant success in price forecasting across sectors such as metals, energy, and agricultural products. In this study, four advanced deep learning architectures were developed to predict the monthly prices offered by stone suppliers in the export market. The primary objective of designing these models was to simultaneously leverage temporal (time series) and non-temporal (static) information influencing pricing. To this end, the architectures were structured in a dual-branch manner, where time series data and static features were processed independently and subsequently fused at an intermediate stage. The models examined include Long Short-Term Memory (LSTM), Bidirectional LSTM (Bi-LSTM), Transformer architecture, and a hybrid Transformer-LSTM-CLS structure. Their performance was evaluated and compared using various statistical metrics. The findings of this research can significantly assist stakeholders and practitioners in the stone industry to make better economic decisions and reduce investment risks by leveraging historical data and intelligent analyses.

2. Literature Review

Accurate prediction of mineral prices, especially copper ore, is crucial for economic decisions and risk management in mining. Tapia Cortez et al. [1] noted traditional methods fail to model complex market dynamics well, proposing ML and chaos theory as superior, though economic interpretability remains challenging. Combining these approaches suits this study’s focus on ML for copper ore price prediction [1]. Shi used a hybrid CNN-LSTM to model nonlinear and temporal patterns in carbon trading prices, reducing error by 9% and speeding training by 40%. Despite data dependency and interpretability issues, deep learning shows promise for mineral price forecasting [2].

In geology, a 2020 Jiangxi study applied Random Forest and CNN to predict tungsten deposits; despite CNN’s accuracy, RF’s stability made it preferable, highlighting ML’s strength in complex geological data modeling relevant for copper ore price prediction [3]. Blockchain-based mineral land tokenization requires precise price forecasting. A 2023 study showed optimized deep neural networks improved mineral land price predictions despite high computational costs, aligning with this research’s goals [4]. A 2023 metal price study (lead, aluminum, copper, tin) found hybrid LSTM-GRU and LSTM-CNN models outperformed others in accuracy and stability, even amid COVID-19 volatility, supporting hybrid models for copper ore forecasting [5]. A 2021 review found ANN, SVM, and RF effective in modeling complex geological parameters but noted challenges like overfitting and high computation. Hybrid and pretrained models alongside industrial standards were recommended to improve adoption and accuracy [6].

In energy forecasting, advanced hybrids like 2022’s VMD-A-LSTM-SVR decomposed volatile coal prices for better accuracy. Similarly, Kailing Yang et al. [7] combined signal preprocessing with deep learning for improved carbon and natural gas price predictions, especially short-term [8]. Electricity markets benefit from deep models like GHTnet, which improved sharp price fluctuation predictions by 17.34% MAPE on New York power plant data [9]. A 2022 German study compared SARIMA(X), LSTM, CNN-LSTM, and hybrid VAR models for next-day electricity prices, finding LSTM best overall, VAR strong short-term, and their hybrid further improved accuracy, emphasizing combining traditional and advanced methods [10].

In the field of stone, few studies have examined and analyzed real mine data. One such study, conducted in 2016, used mine data for order allocation, but considered the block stone price based on historical data [11]. In copper price prediction, a 2021 study compared GEP, ANN, ANFIS, and ANFIS-ACO, with ANN performing best (RMSE=356.51, R²=98.1%), confirming AI’s ability to model complex metal markets [12]. A 2023 study proposed a CNN-LSTM-ARMA hybrid for financial time series forecasting under environmental change, outperforming individual models in accuracy and stability [13].

For stock prices, a 2020 hybrid ANN model accurately predicted Tehran stock exchange trends by optimizing key economic inputs [14]. Kiani Mavi et al. [15] used MLP with backpropagation for precise next-day stock prediction. Studies in 2024 found PSO-optimized LSTM-CNN hybrids superior in modeling financial factors [16]. Farmani and Jahangiri [17] showed PSO outperformed GA and GWO for LSTM tuning. An XGBoost-LSTM hybrid excelled in Bitcoin price forecasting, offering a potential framework for copper ore prediction [18].

Finally, a 2024 review showed ML methods achieve strong real estate price prediction but face interpretability and integration challenges. Combining qualitative/quantitative factors and dynamic models enhances accuracy and global market analysis, aiding stakeholders [19].

In summary, hybrid machine learning models incorporating signal preprocessing, deep neural networks, and optimization algorithms have significantly improved the accuracy and robustness of price forecasting across various domains, including mineral commodities, energy, financial assets, and real estate. These advancements pave the way for enhanced copper ore price prediction and support strategic decision-making and risk management in the mining sector.

Building upon existing literature that highlights the strong performance of deep learning models such as LSTM, Transformer, and hybrid architectures in forecasting metal and mineral prices, this study introduces a dual-input structure to simultaneously incorporate sequential and non-sequential features in predicting export stone prices. The evaluation of four models, LSTM, Bi-LSTM, Transformer, and a hybrid Transformer+LSTM+CLS, demonstrates the superiority of attention-based architectures and the hybrid model over traditional models like Bi-LSTM in terms of accuracy and generalization. This innovative approach, in contrast to previous research focusing mainly on industrial metals and financial markets, underscores the importance of combining sequential analysis with attention mechanisms to enhance deep learning performance in competitive, data-driven industries.

3. Methodology

3.1 Data and Preprocessing

The dataset used in this study comprises monthly records collected from five different stone mines, all of which have been continuous suppliers of raw materials to Almas Stone Factory, located in Qom, for the past ten years. Due to the large volume of data, a portion of the data related to five suppliers is presented in

Table 1. The model variables are divided into two main categories: The first includes temporal features such as exchange rate, diesel fuel price, inflation rate, and order volume; the second consists of static features including stone block quality, mine brand score, supplier cooperation history, and the current month's price.

This table is shown in the PDF version of the article.

Categorical features such as stone block quality and mine brand score were encoded using an ordinal encoding scheme to preserve their relative order. The mine identifier was one-hot encoded, enabling the model to differentiate between mines without introducing ordinal bias.

After normalization using MinMax scaling, we applied a sliding window technique to extract fixed-length input sequences. Specifically, for each mine independently, we constructed input windows of six months and predicted the price for the following month. This approach produced a sequence input capturing temporal trends and a static vector representing contemporaneous features at the prediction point.

We performed a time-based 80/20 split per mine, using the earliest 80% of each mine’s samples for training and the remaining 20% for validation. No data shuffling was performed to prevent temporal leakage. This ensured that all validation predictions were made on strictly unseen future data. By processing each mine separately and feeding the resulting windows in parallel, the models were able to learn both global patterns across mines and local variations within each mine. This strategy balances generalization and specificity, which is especially important in low-data forecasting contexts like this one.

3.2 Implementation Environment and General Settings

All models were implemented in Python using the TensorFlow, Keras, NumPy, and Pandas libraries. The Adam optimization algorithm was employed in all architectures with an initial learning rate of 0.001. Model performance was evaluated based on Mean Squared Error (MSE), Mean Absolute Error (MAE), and the coefficient of determination (R²). The training process was executed for 300 epochs with a batch size of 32. To improve convergence and reduce overfitting, a learning rate reduction strategy was applied: the learning rate was decreased by a factor of 0.1 if the validation loss did not improve for 10 consecutive epochs. Early stopping was not used; instead, the model with the lowest validation loss was saved.

4. Model Architectures

4.1 Model 1: Dual-Path LSTM Network

The first proposed model is based on the LSTM recurrent neural network. Due to its ability to capture long-term temporal dependencies, LSTM is one of the most effective architectures for time series analysis. In this model, time-series data, comprising six time steps with twelve features per step, are fed into a unidirectional LSTM layer with 32 units to learn temporal dependencies. In parallel, a separate branch processes non-sequential (static) features using a Dense layer with 32 neurons and ReLU activation. The outputs from both branches are concatenated and passed through a fully connected layer with 32 neurons, followed by a final output layer producing a single scalar prediction. This architecture integrates sequential trends and static contextual attributes into a unified forecasting framework.

4.2 Model 2: Bidirectional LSTM (Bi-LSTM)

The second architecture explores a Bidirectional LSTM (Bi-LSTM) layer in place of the standard LSTM. Bi-LSTMs process sequence data in both forward and backward directions, which can be beneficial in contexts where the full sequence is available and temporal dependencies are symmetric. However, in time-series forecasting, where only past information should influence future predictions, this approach can introduce modeling inconsistencies. In our case, the Bi-LSTM-based model used the same lookback structure and static feature processing as the previous architecture but replaced the LSTM with a Bi-LSTM layer containing 16 units.

4.3 Model 3: Transformer Based on Attention

The third model adopts a Transformer encoder architecture to capture complex temporal dependencies within fixed-length input sequences. The time-series branch begins with a Dense projection layer that maps the 12-feature input at each of the six time steps to a 32-dimensional latent space. To preserve the temporal order, sinusoidal positional encoding is added to the projected input. This sequence is then passed through a single Transformer encoder block consisting of multi-head self-attention (2 heads) and a position-wise feed-forward network (hidden size = 32), with residual connections and layer normalization. To reduce the sequence to a fixed-size representation, global average pooling is applied over the time dimension. A second branch processes the static features via a 32-unit Dense layer with ReLU activation. The outputs of both branches are concatenated and passed through a 64-unit Dense layer and a final linear output layer predicting the price. This model allows the attention mechanism to dynamically weight relevant time steps, making it particularly suitable for learning variable-length dependencies in economic time-series forecasting.

4.4 Model 4: Hybrid Transformer-LSTM-CLS Architecture

The fourth architecture combines LSTM and Transformer components to jointly capture short-term temporal patterns and long-range dependencies. The sequential input is first processed by a unidirectional LSTM layer (32 units) to extract local sequence representations. These are projected to a fixed embedding size (32 dimensions) and prepended with a learnable [CLS] token, a vector trained to summarize sequence-level information. Sinusoidal positional encoding is applied to the extended sequence to retain temporal ordering. This input is passed through a Transformer encoder block with multi-head self-attention (2 heads) and a feed-forward network (64 hidden units), using residual connections and layer normalization. The final output is derived from the CLS token, which acts as an aggregate representation of the sequence. A separate static feature branch processes contemporaneous non-sequential attributes through a 32-unit dense layer. The CLS representation and static vector are concatenated and passed through a 64-unit dense layer before producing a single scalar prediction. This hybrid design leverages the sequential modeling strength of LSTMs with the global context-awareness of Transformers, making it particularly suited to complex, low-signal forecasting scenarios.

5. Results and Model Evaluation

This section presents the predicted monthly prices for stone suppliers generated by the four machine learning models developed in this study. Each model aimed to forecast the prices for five different mines (Mine1 to Mine5), incorporating both time-series and static features. The predicted values are summarized as follows: I. The Transformer model delivered highly accurate predictions, particularly for Mine1, where the forecasted value reached 9,171,733 units. This demonstrates the model’s strong ability to identify complex patterns and long-range dependencies within the data.

II. The hybrid Transformer+LSTM+CLS model also performed well, especially in predicting prices for Mine2, with a value of 6,163,421 units. By combining the capabilities of both Transformer and LSTM architectures, this model achieved high prediction accuracy across various mines and effectively captured price fluctuations. III. The classical LSTM model yielded relatively balanced predictions, with a forecast of 3,278,542 units for Mine5. This suggests that the model performs reasonably well under simpler data structures and limited information.

IV. The Bi-LSTM model, although theoretically capable of analyzing sequences bidirectionally, produced relatively higher predictions for mines such as Mine3 and Mine4. This might indicate potential overestimation or a heightened sensitivity to data fluctuations. Table 2 shows the forecasted prices. Figs. 1-5 present the comparative charts of actual and predicted prices obtained through four different approaches for five mines. In the following, to determine the best approach, the performance of four predictive models, LSTM, Bi-LSTM, Transformer, and the hybrid Transformer+LSTM+CLS, is evaluated based on five key metrics: MAE, MSE, Root Mean Squared Error (RMSE), Coefficient of Determination (R²), and validation loss (val_loss). These criteria provide a comprehensive view of both the accuracy and the generalization capability of the models.

Table 2. Predicted prices of mines using four approaches.

This table is shown in the PDF version of the article.

This figure is shown in the PDF version of the article.

Figure 1. Comparison of actual and predicted prices for Mine 1.

This figure is shown in the PDF version of the article.

Figure 2. Comparison of actual and predicted prices for Mine 2.

This figure is shown in the PDF version of the article.

Figure 3. Comparison of actual and predicted prices for Mine 3.

This figure is shown in the PDF version of the article.

Figure 4. Comparison of actual and predicted prices for Mine 4.

This figure is shown in the PDF version of the article.

Figure 5. Comparison of actual and predicted prices for Mine 5.

The classical LSTM model, with 8,289 parameters, demonstrated acceptable performance. It achieved an MAE of approximately 329,760 indicating a moderate average prediction error. The RMSE of 403,002 and R² value of 0.9701 suggest that the model could explain over 97% of the variance in the price data. The validation loss of 0.00452 further supports its generalization capability. Overall, this model strikes a good balance between simplicity and predictive performance.

Despite having 15,073 parameters and the same training duration, the Bi-LSTM model underperformed relative to its unidirectional LSTM counterpart. It produced the highest MAE (479,072) and RMSE (590,329), and the lowest R² score (0.9359) among all models. These results likely stem not from overfitting or excess complexity, but from the Bi-LSTM’s bidirectional structure, which is misaligned with the nature of forecasting tasks. Since future time steps are not available at inference, training with bidirectional context can lead to temporal leakage or inconsistent learning dynamics. This makes Bi-LSTM architectures less suitable for time-series forecasting where causal direction is crucial.

The Transformer-based model, with 15,713 trainable parameters, demonstrated the strongest overall performance among the architectures evaluated. It achieved the lowest MAE (230,432) and RMSE (302,412), along with the highest R² score (0.9832), indicating high accuracy in predicting future prices. Its validation loss of 0.00121 suggests strong generalization to unseen data. The model’s attention mechanism likely played a central role in this performance, enabling it to effectively capture long-range dependencies and complex, non-linear relationships within the time-series data.

Table 3. Comparison of approaches based on evaluation criteria.

This table is shown in the PDF version of the article.

A comprehensive comparison of the four models reveals that each architecture has its strengths and limitations:

I. The Transformer model delivered the best results (MAE: 230,432; RMSE: 302,412; R²: 0.9832; val_loss:

0.00121), capturing long-term and non-linear dependencies through self-attention. Its architecture proved highly effective for the pricing task.

II. The hybrid Transformer+LSTM+CLS model closely matched the Transformer (MAE: 239,869; R²: 0.9823) by combining LSTM’s sequential modeling with Transformer attention. However, the added complexity (20,033 parameters) offered limited benefit, likely due to representational redundancy in a low-data setting. III. The LSTM model performed reasonably well (MAE: 329,760; R²: 0.9701), offering a strong balance between simplicity and accuracy. It remains a good option when computational or data constraints exist.

IV. The Bi-LSTM model, despite its theoretical ability to model bidirectional dependencies, performed the worst (MAE: 479,072; R²: 0.9359). This was not due to overfitting, but rather the model’s use of future context during training, which contradicts the causal nature of forecasting. While Bi-LSTM can be effective in other sequence tasks, it is unsuitable here unless modified to respect time order.

In conclusion, the Transformer model is recommended as the most effective approach for predicting stone suppliers' prices in this study. The hybrid model can also be considered for scenarios demanding more nuanced modeling of both long-range and local patterns. The LSTM remains a practical option for lightweight applications, whereas Bi-LSTM requires cautious use and further adaptation for effective deployment.

6. Conclusion and Future Research

In this study, four deep learning architectures, LSTM, Bi-LSTM, Transformer, and a hybrid Transformer+LSTM+CLS, were developed and evaluated to forecast monthly prices offered by stone suppliers in the export market. Each model followed a dual-branch design, enabling simultaneous processing of sequential (time-series) and static (supplier-specific) features, which were later fused to enhance prediction accuracy.

Evaluation results showed that the Transformer model outperformed all others, benefiting from its attention mechanism and ability to model long-range dependencies. The hybrid Transformer+LSTM+CLS model also performed competitively, suggesting that combining LSTM’s sequential encoding with self-attention can offer robust modeling for complex price dynamics. In contrast, the Bi-LSTM model performed the worst, not due to overfitting, but because its bidirectional structure can unintentionally use future information during training, making it misaligned with causal forecasting tasks.

The key takeaway is that attention-based and hybrid architectures are well-suited for capturing the nonlinear and temporal complexities inherent in export pricing. Their ability to process heterogeneous features in parallel makes them valuable tools for data-driven pricing strategy.

Future Research Recommendations:

I. Expand dataset size and scope: Incorporating more data across suppliers, regions, and time periods could improve generalizability and robustness.

II. Integrate competitive and environmental variables: Including additional factors such as export regulations, global stone market indices, and competitor pricing strategies may enhance model accuracy and practical applicability.

III. Develop adaptive forecasting models: Considering the volatile nature of export markets, implementing adaptive models that can dynamically adjust to environmental changes would significantly increase the models' real-world utility.

IV. Implement real-time decision support tools: Building intelligent user interfaces or dashboards powered by these models could assist market analysts and sales managers in making informed, data-driven decisions.

References

  1. [1] Cortez, C. A. T., Saydam, S., Coulton, J., & Sammut, C. (2018). Alternative techniques for forecasting mineral commodity prices. International journal of mining science and technology, 28(2), 309–322. https://doi.org/10.1016/j.ijmst.2017.09.001
  2. [2] Shi, H., Wei, A., Xu, X., Zhu, Y., Hu, H., & Tang, S. (2024). A CNN-LSTM based deep learning model with high accuracy and robustness for carbon price forecasting: A case of Shenzhen’s carbon market in China. Journal of environmental management, 352, 120131. https://doi.org/10.1016/j.jenvman.2024.120131
  3. [3] Sun, T., Li, H., Wu, K., Chen, F., Zhu, Z., & Hu, Z. (2020). Data-driven predictive modelling of mineral prospectivity using machine learning and deep learning methods: A case study from southern Jiangxi Province, China. Minerals, 10(2), 102. https://doi.org/10.3390/min10020102
  4. [4] James, A., Mahmud, R., Afrin, M., Mistry, S., & Krishna, A. (2023). Performance optimisation of regression-based machine learning models to predict mineral land prices for tokenization. 2023 IEEE engineering informatics (pp. 1-7). IEEE. https://doi.org/10.1109/IEEECONF58110.2023.10520573
  5. [5] Shi, T., Li, C., Zhang, W., & Zhang, Y. (2023). Forecasting on metal resource spot settlement price: New evidence from the machine learning model. Resources policy, 81, 103360. https://doi.org/10.1016/j.resourpol.2023.103360
  6. [6] Dumakor-Dupey, N. K., & Arya, S. (2021). Machine learning—a review of applications in mineral resource estimation. Energies, 14(14), 4079. https://doi.org/10.3390/en14144079
  7. [7] Yang, K., Zhang, X., Luo, H., Hou, X., Lin, Y., Wu, J., & Yu, L. (2024). Predicting energy prices based on a novel hybrid machine learning: Comprehensive study of multi-step price forecasting. Energy, 298, 131321. https://doi.org/10.1016/j.energy.2024.131321
  8. [8] Zhang, K., Cao, H., Thé, J., & Yu, H. (2022). A hybrid model for multi-step coal price forecasting using decomposition technique and deep learning algorithms. Applied energy, 306, 118011. https://doi.org/10.1016/j.apenergy.2021.118011
  9. [9] Yang, H., & Schell, K. R. (2022). GHTnet: Tri-Branch deep learning network for real-time electricity price forecasting. Energy, 238, 122052. https://doi.org/10.1016/j.energy.2021.122052
  10. [10] Lehna, M., Scheller, F., & Herwartz, H. (2022). Forecasting day-ahead electricity prices: A comparison of time series and neural network models taking external regressors into account. Energy economics, 106, 105742. https://doi.org/10.1016/j.eneco.2021.105742
  11. [11] Barzinpour, F., Moazeni, H., & Pishvai, M. (2016). A data envelopment analysis (DEA) model for supplier selection by considering sustainable criteria: A case study in a stone industry. Strategic management researches, 21(59), 89–115. (In Persian). https://smr.journals.iau.ir/article_525059_0.html
  12. [12] Khoshalan, H. A., Shakeri, J., Najmoddini, I., & Asadizadeh, M. (2021). Forecasting copper price by application of robust artificial intelligence techniques. Resources policy, 73, 102239. https://doi.org/10.1016/j.resourpol.2021.102239
  13. [13] He, K., Yang, Q., Ji, L., Pan, J., & Zou, Y. (2023). Financial time series forecasting with the deep learning ensemble model. Mathematics, 11(4), 1054. https://doi.org/10.3390/math11041054
  14. [14] SaeediAghdam, M., Sadeghi, A., Bahiraei, A., & HajiAsghari, S. Y. (2022). Presenting a stock price prediction model using deep learning algorithms and its application in the pricing of Islamic banks’ stocks. Quarterly journal of islamic economics and banking, 4, 117–134. (In Persian). http://mieaoi.ir/article-1-964-fa.html
  15. [15] Kiyani Mavi, R., & Sayadi Nik, K. (2015). Using different learning algorithms in the stock price prediction by using neural networks. Journal of development & evolution mnagement, 1393(Spec. Issue), 75–81. (In Persian). https://sanad.iau.ir/en/Article/949971?FullText=FullText
  16. [16] Gholami, N., & Shams Gharne, N. (2024). Presenting an optimized CNN-LSTM model for stock price forecasting in the tehran stock exchange. Financial management perspective, 14(45), 123–147. https://doi.org/10.48308/jfmp.2024.104892
  17. [17] Farmani, A., & Jahangiri, S. (2023). Comparing the performance of metaheuristic algorithms in predicting stock prices in the tehran stock exchange market. Majlis and economy, 1(2), 128-152. (In Persian). https://doi.org/10.22034/mec.2024.16767.1029
  18. [18] Nikparvar, S., Ahmadi, F., & Kalbkhani, H. (2024). Forecasting the price of cryptocurrencies using meta-learning algorithms. Iranian journal of wargaming, 7(14), e209992. https://doi.org/10.22034/ijwg.2024.472660.1090
  19. [19] Mathotaarachchi, K. V., Hasan, R., & Mahmood, S. (2024). Advanced machine learning techniques for predictive modeling of property prices. Information, 15(6), 295. https://doi.org/10.3390/info15060295