The memory concept behind deep neural network models: An application in time series forecasting in the e-Commerce sector

Filipe R. Ramos; Maria Teresa Pereira; Marisa Oliveira; Lihki Rubio

doi:10.31181/dmame622023695

Authors

Filipe R. Ramos CEAUL-Centro de Estatística e Aplicações, Faculdade de Ciências, Universidade de Lisboa, 1749-016 Lisboa, Portugal https://orcid.org/0000-0002-8326-3310
Maria Teresa Pereira ISEP-Instituto Superior de Engenharia do Porto, 4249-015 Porto, Portugal https://orcid.org/0000-0003-4556-9578
Marisa Oliveira ISEP-Instituto Superior de Engenharia do Porto, 4249-015 Porto, Portugal https://orcid.org/0000-0002-1918-0895
Lihki Rubio Universidad del Norte, Km. 5 vía Puerto Colombia, Barranquilla, 081007 Atlántico, Colombia https://orcid.org/0000-0001-9085-5683

DOI:

https://doi.org/10.31181/dmame622023695

Keywords:

E-Commerce, time series, deep neural network, forecasting, prediction error, computational cost

Abstract

A good command of computational and statistical tools has proven advantageous when modelling and forecasting time series. According to recent literature, neural networks with long memory (e.g., Short-Term Long Memory) are a promising option in deep learning methods. However, only some works also consider the computational cost of these architectures compared to simpler architectures (e.g., Multilayer Perceptron). This work aims to provide insight into the memory performance of some Deep Neural Network architectures and their computational complexity. Another goal is to evaluate whether choosing more complex architectures with higher computational costs is justified. Error metrics are then used to assess the forecasting models' performance and computational cost. Two-time series related to e-commerce retail sales in the US were selected: (i) sales volume; (ii) e-commerce sales as a percentage of total sales. Although there are changes in data dynamics in both series, other existing characteristics lead to different conclusions. "Long memory" allows for significantly better forecasts in one-time series. In the other time series, this is not the case.

Downloads

Download data is not yet available.

References

Atsalakis, G. (2016). New Technology in Shopping: Forecasting Electronic Shopping With the Use of a Neuro-Fuzzy System. Journal of Food Products Marketing, 23(5), 522–532. https://doi.org/10.1080/10454446.2014.1000445

Brownlee, J. (2018). Deep learning for time series forecasting: predict the future with MLPs, CNNs and LSTMs in Python. Machine Learning Mastery.

Cavalcante, R. C., Brasileiro, R. C., Souza, V. L. F., Nobrega, J. P., & Oliveira, A. L. I. (2016). Computational Intelligence and Financial Markets: A Survey and Future Directions. Expert Systems with Applications, 55, 194–211. https://doi.org/10.1016/j.eswa.2016.02.006

Chatfield, C. (2016). The Analysis of Time Series: an introduction (6th ed.). Chapman and Hall/CRC.

Chollet, F. (2021). Deep Learning with Python, Second Edition. Manning Publications.

Corsini, R. R., Costa, A., Fichera, S., & Framinan, J. M. (2022). A new data-driven framework to select the optimal replenishment strategy in complex supply chains. IFAC-PapersOnLine, 55(10), 1423–1428.

Costa, A., Ramos, F. R., Mendes, D., & Mendes, V. (2019). Forecasting financial time series using deep learning techniques. In IO 2019 - XX Congresso da APDIO 2019. Instituto Politécnico de Tomar - Tomar.

Data Science Academy. (2019). Deep Learning Book. Retrieved from http://deeplearningbook.com.br/

Diniz, A. P. M., Ciarelli, P. M., Salles, E. O. T., & Coco, K. F. (2022). Long Short-Term Memory Neural Networks for Clogging Detection in the Submerged Entry Nozzle. Decision Making: Applications in Management and Engineering, 5(1), 154–168. https://doi.org/10.31181/dmame0313052022d

Ghosal, S., Dey, S., Chattopadhyay, P. P., Datta, S., & Bhattacharyya, P. (2021). Designing optimized ternary catalytic alloy electrode for efficiency improvement of semiconductor gas sensors using a machine learning approach. Decision Making: Applications in Management and Engineering, 4(2), 126–139. https://doi.org/10.31181/dmame210402126g

Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In JMLR W&CP: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2010) (pp. 249–256). Sardinia: JMLR Workshop and Conference Proceedings.

Greff, K., Srivastava, R. K., Koutník, J., Steunebrink, B. R., & Schmidhuber, J. (2015). LSTM: A Search Space Odyssey. IEEE Transactions on Neural Networks and Learning Systems, 28(10), 2222–2232. https://doi.org/10.1109/TNNLS.2016.2582924

Hang, N. T. (2019). Research on a number of applicable forecasting techniques in economic analysis, supporting enterprises to decide management. World Scientific News, 119, 52–67.

Higueras-Castillo, E., Liébana-Cabanillas, F. J., & Villarejo-Ramos, Á. F. (2023). Intention to use e-commerce vs physical shopping. Difference between consumers in the post-COVID era. Journal of Business Research, 157, 113622. https://doi.org/10.1016/j.jbusres.2022.113622

Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

Jiang, Z., & Benbasat, I. (2014). Virtual Product Experience: Effects of Visual and Functional Control of Products on Perceived Diagnosticity and Flow in Electronic Shopping. Journal of Management Information Systems, 21(3), 111–147. https://doi.org/10.1080/07421222.2004.11045817

Jozefowicz, R., Zaremba, W., & Sutskever, I. (2015). An Empirical Exploration of Recurrent Network Architectures. In ICML - International Conference on Machine Learning.

Kingma, D. P., & Ba, J. L. (2015). Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings. International Conference on Learning Representations, ICLR.

https://doi.org/10.48550/arXiv.1412.6980

Koutník, J., Greff, K., Gomez, F., & Schmidhuber, J. (2014). A Clockwork RNN. 31st International Conference on Machine Learning, ICML 2014, 5, 3881–3889.

Lopes, D. R., & Ramos, F. R. (2020). Univariate Time Series Forecast. Retrieved from https://github.com/DidierRLopes/UnivariateTimeSeriesForecast

Lopes, D. R., Ramos, F. R., Costa, A., & Mendes, D. (2021). Forecasting models for time-series: a comparative study between classical methodologies and Deep Learning. In SPE 2021 – XXV Congresso da Sociedade Portuguesa de Estatística. Évora - Portugal.

Martínez-López, F. J., Feng, C., Li, Y., & Sansó Mata, M. (2022). Restoring the buyer–seller relationship through online return shipping: The role of return shipping method and return shipping fee. Electronic Commerce Research and Applications, 54, 101170. https://doi.org/10.1016/j.elerap.2022.101170

Modgil, S., Dwivedi, Y. K., Rana, N. P., Gupta, S., & Kamble, S. (2022). Has Covid-19 accelerated opportunities for digital entrepreneurship? An Indian perspective. Technological Forecasting and Social Change, 175, 121415. https://doi.org/10.1016/j.techfore.2021.121415

Pesaran, M. H., & Timmermann, A. (2004). How costly is it to ignore breaks when forecasting the direction of a time series? International Journal of Forecasting, 20(3), 411–425. https://doi.org/10.1016/S0169-2070(03)00068-2

Pineda, F. (1987). Generalization of Back propagation to Recurrent and Higher Order Neural Networks. Undefined.

Ramos, F. R. (2021). Data Science na Modelação e Previsão de Séries Económico-financeiras: das Metodologias Clássicas ao Deep Learning. (PhD Thesis, Instituto Universitário de Lisboa - ISCTE Business School, Lisboa, Portugal). https://doi.org/10.13140/RG.2.2.14510.02887

Ramos, F. R., Costa, A., Mendes, D., & Mendes, V. (2018). Forecasting financial time series: a comparative study. In JOCLAD 2018, XXIV Jornadas de Classificação e Análise de Dados. Escola Naval – Alfeite. https://doi.org/10.13140/RG.2.2.11548.41606

Ramos, F. R., Lopes, D. R., Costa, A., & Mendes, D. (2021). Explorando o poder da memória das redes neuronais LSTM na modelação e previsão do PSI 20. In SPE 2021 – XXV Congresso da Sociedade Portuguesa de Estatística. Évora - Portugal.

Ramos, F. R., Lopes, D. R., & Pratas, T. E. (2022). Deep Neural Networks: A Hybrid Approach Using Box&Jenkins Methodology. In Innovations in Mechatronics Engineering II. icieng 2022. Lecture Notes in Mechanical Engineering (pp. 51–62). Springer, Cham. https://doi.org/10.1007/978-3-031-09385-2_5

Ravichandiran, S. (2019). Hands-On Deep Learning Algorithms with Python: Master deep learning algorithms with extensive math by implementing them using TensorFlow. Packt Publishing Ltd.

Ren, X. X., Gong, Y., Rekik, Y., & Xu, X. (2022). Data-driven analysis on anticipatory shipping for pickup point inventory. IFAC-PapersOnLine, 55(10), 714–718. https://doi.org/10.1016/j.ifacol.2022.09.491

Rubio, L., & Alba, K. (2022). Forecasting Selected Colombian Shares Using a Hybrid ARIMA-SVR Model. Mathematics, Vol. 10, Page 2181, 10(13), 2181. https://doi.org/10.3390/math10132181

Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533–536. https://doi.org/10.1038/323533a0

Sezer, O. B., Gudelek, M. U., & Ozbayoglu, A. M. (2020). Financial time series forecasting with deep learning : A systematic literature review: 2005–2019. Applied Soft Computing, 90, 106–181. https://doi.org/10.1016/j.asoc.2020.106181

Tealab, A. (2020). Time series forecasting using artificial neural networks methodologies: A systematic review. Future Computing and Informatics Journal, 3(2). https://doi.org/10.1016/j.fcij.2018.10.003

Tkáč, M., & Verner, R. (2016). Artificial neural networks in business: Two decades of research. Applied Soft Computing, 38, 788–804. https://doi.org/10.1016/j.asoc.2015.09.040

Utku, A. (2023). Deep learning based an efficient hybrid prediction model for Covid-19 cross-country spread among E7 and G7 countries. Decision Making: Applications in Management and Engineering, 6(1), 502–534. https://doi.org/10.31181/dmame060129022023u

Utku, A., & Kaya, S. K. (2022). Multi-Layer Perceptron Based Transfer Passenger Flow Prediction in Istanbul Transportation System. Decision Making: Applications in Management and Engineering, 5(1), 208–224. https://doi.org/10.31181/dmame0315052022u

Wang, J., & Dai, C. H. (2004). A fuzzy constraint satisfaction approach for electronic shopping assistance. Expert Systems with Applications, 27(4), 593–607. https://doi.org/10.1016/j.eswa.2004.06.004

Willmott, C., & Matsuura, K. (2005). Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate Research, 30(1), 79–82. https://doi.org/10.3354/cr030079

Wilson, J. H., & Spralls III, S. A. (2018). What do business professionals say about forecasting in the marketing curriculum? International Journal of Business, Marketing, & Decision Science, 11(1), 1–20.