2072-64142411-140610.17059/ekon.reg.2024-3-21Regional Inflation Analysis Using Social Network DataАнализ региональной инфляции с использованием данных социальных сетейhttps://orcid.org/0000-0001-5132-7423Shcherbakov Vassiliy S. ЩербаковВасилий Сергеевич shcherbakovvs@mail.ruhttps://orcid.org/0000-0002-8106-9426Karpov Ilia A. Karpov Карпов Илья Андреевич karpovilia@gmail.comOmsk Regional Division of the Siberian Main Branch of the Central Bank of the Russian FederationОтделение по Омской области Сибирского главного управления Центрального банка Российской ФедерацииHSE UniversityНациональный исследовательский университет «Высшая школа экономики»203930946Copyright © 2024 Vassiliy S. Shcherbakov, Ilia A. KarpovCopyright © 2024 Василий Сергеевич Щербаков, Илья Андреевич Карпов2024Vassiliy S. Shcherbakov, Ilia A. KarpovВасилий Сергеевич Щербаков, Илья Андреевич КарповCC BY 4.0https://www.economyofregions.org/ojs/index.php/er/article/view/575https://www.economyofregions.org/ojs/index.php/er/article/download/575/344

Inflation is one of the most important macroeconomic indicators that have a great impact on the population of any country and region. Inflation is influenced by a range of factors, including inflation expectations. Many central banks take this factor into consideration while implementing monetary policy within the inflation targeting regime. Nowadays, a lot of people are active users of the Internet, especially social networks. It is hypothesised that people search, read, and discuss mainly only those issues that are of particular interest to them. It is logical to assume that the dynamics of prices may also be in the focus of users’ discussions. So, such discussions could be regarded as an alternative source of more rapid information about inflation expectations. This study is based on unstructured data from VKontakte social network used to analyse upward and downward inflationary trends (on the example of the Omsk region). The sample of more than 8.5 million posts was collected between January 2010 and May 2022. The authors used BERT neural networks to solve the problem. These models demonstrated better results than the benchmarks (e.g., logistic regression, decision tree classifier, etc.). It makes possible to define pro-inflationary and disinflationary types of keywords in different contexts and get their visualisation with SHAP method. This analysis provides additional operational information about inflationary processes at the regional level The proposed approach can be scaled for other regions. At the same time, the limitation of the work is the time and power costs for the initial training of similar models for all regions of Russia.

Инфляция как один из важнейших макроэкономических показателей оказывает большое влияние на население всех стран и регионов. На саму инфляцию влияет ряд факторов, в том числе инфляционные ожидания. Многие центральные банки учитывают этот фактор при реализации денежно-кредитной политики в режиме инфляционного таргетирования. В настоящее время многие люди являются активными пользователями интернета, особенно социальных сетей. Предполагается, что люди ищут, читают и обсуждают в основном только те темы, которые представляют для них особый интерес. Логично предположить, что динамика цен также может быть в фокусе обсуждений пользователей. Такие обсуждения можно рассматривать как альтернативный источник оперативной информации об инфляционных ожиданиях. В данной статье анализируются неструктурированные данные из социальной сети ВКонтакте для исследования восходящих и нисходящих трендов инфляции (на примере Омской области). Выборка из более чем 8,5 миллионов постов была собрана за период с января 2010 по май 2022 гг. Для решения задачи была использована нейронная сеть BERT, кото­-рая показала лучшие результаты по сравнению с бенчмарками (такими как логистическая регрессия, классификатор дерева решений и т. д.). Применение BERT-модели позволило определить проинфляционные и дезинфляционные типы ключевых слов в разных контекстах; метод SHAP позволил визуализировать полученные результаты. Подобный анализ дает дополнительную оперативную информацию об инфляционных процессах на региональном уровне. Предложенный подход может быть масштабирован для других регионов. При этом ограничением работы являются временные и энергетические затраты на первоначальное обучение аналогичных моделей для всех регионов России.

inflation, regional inflation expectations, machine learning, BERT, social networks, monetary policy, neural networkинфляция, региональные инфляционные ожидания, машинное обучение, BERT, социальные сети, денежно-кредитная политика, нейронная сеть

The article was prepared within the framework of the HSE University Basic Research Program.

Статья выполнена в рамках Программы фундаментальных исследований НИУ ВШЭ.

Ahmed, H., Traore, I., & Saad, S. (2017). Detecting opinion spams and fake news using text classification. Security and Privacy, 1 (1), 1-15. https://doi.org/10.1002/spy2.9Angelico, C., Marcucci, J., Miccoli, M., & Quarta, F. (2022). Can We Measure Inflation Expectations Using Twitter? Journal of Econometrics, 228 (2), 259-277.Armantier, O., De Bruine, W., Topa, G., van der Klaauw, W., & Zafar, B. (2015). Inflation expectations and behavior: do survey respondents act on their beliefs? International Economic Review, 56 (2), 505-536.Aromí, D., & Llada, M. (2020). Forecasting inflation with twitter. Asociación Argentina de Economía Política: Working Papers, No. 4308, 23. https://aaep.org.ar/works/works2020/AromiLlada.pdfBaybuza, I. (2018). Inflation forecasting using machine learning methods. Russian Journal of Money and Finance, 77 (4), 42-59.Bengio, Y., Ducharme, R., Vincent, P., & Jauvin, C. (2003). A neural probabilistic language model. Journal of machine learning research, (3), 1137–1155.Bradbury, J., Merity, S., Xiong, C., & Socher, R. (2016). Quasi-recurrent neural networks. https://arxiv.org/abs/1611.01576 Chakraborty, C., & Joseph, A. (2017). Machine Learning at Central Banks. Bank of England, Working Paper No. 674, 85. https://www.bankofengland.co.uk/-/media/boe/files/working-paper/2017/machine-learning-at-central-banks.pdf?la=en&hash=EF5C4AC6E7D7BDC1D68A4BD865EEF3D7EE5D7806Coibion, O., Gorodnichenko, Y., & Kamdar, R. (2018). The formation of expectations, inflation, and the Phillips curve. Journal of Economic Literature, 56 (4), 1447–1491.De Bruine, W., Manski, F. Ch., Topa, G., & van der Klaauw, W. (2011). Measuring consumer uncertainty about future inflation. Journal of Applied Econometrics, 26 (33), 454-478.Devlin, J., & Chang, M.-W. (2018). Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing. https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.htmlDevlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Vol. 1 (pp. 4171–4186). Minneapolis, Minnesota. https://doi.org/10.18653/v1/N19-1423Frisch, H. (1990). Theories of inflation. Cambridge University Press, 276.Giannone, D., Reichlin, L., & Small, D. (2008). Nowcasting: The real-time informational content of macroeconomic data. Journal of Monetary Economics, 55 (4), 665–676.Goloshchapova, I., & Andreev, M. (2017). Measuring inflation expectations of the Russian population with the help of machine learning. Voprosy ekonomiki, (6), 71-93. (In Russ.)Gurov, I. N. (2022). Inflation in Russia: differences between official data and population’s perception. Voprosy statistiki, 29 (3), 100-111. (In Russ.)Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Zhang, L., Han, W., Huang, M., Jin, Q., Lan, Y., Liu, Y., Liu, Z., Lu, Z., Qiu, X., Song, R., Tang, J., Wen, J.-R., Yuan, J., … Zhu, J. (2021). Pre-trained models: Past, present and future. AI Open, (2), 225-250. https://doi.org/10.1016/j.aiopen.2021.08.002Hinton, G., Vinyals, O., & Dean, J. (2019). Distilling the Knowledge in a Neural Network. NIPS 2014 Deep Learning Workshop, 9. https://doi.org/10.48550/arXiv.1503.02531Hu, Y. (2019). Text mining and data information analysis for network public opinion. Data Science Journal, 18 (7), 1-8. Jalal, N., Mehmood, A., Choi, G. S., & Ashraf, I. (2022). A novel improved random forest for text classification using feature ranking and optimal number of trees. Computer and Information Sciences, 34 (6), 2733–2742. https://doi.org/10.1016/j.jksuci.2022.03.012Jijo, B. T., & Abdulazeez, A. M. (2021). Classification Based on Decision Tree Algorithm for Machine Learning. Journal of Applied Science and Technology Trends, 2 (01), 20-28.Kokalj, E., Škrlj, B., Lavrač, N., Pollak, S., & Robnik-Šikonja, M. (2021). BERT meets Shapley: Extending SHAP Explanations to Transformer-based Classifiers. In: Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation (pp. 16-21). Association for Computational Linguistics. https://aclanthology.org/2021.hackashop-1.3/Kuratov, Y. M. (2020). Spetsializatsiya yazykovykh modeley dlya primeneniya k zadacham obrabotki estestvennogo yazyka [Specialization of language models for application to natural language processing tasks]. https://mipt.ru/upload/medialibrary/099/dissertatsiya-kuratov.pdf (In Russ.)Kuratov, Y., & Arkhipov, M. (2019). Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language. https://doi.org/10.48550/arXiv.1905.07213Lamla, M., & Lein, S. (2014). The role of media for consumers’ inflation expectation formation. Journal of Economic Behavior & Organization, 106, 62-77.Lan, Zh., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2020). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In: International Conference on Learning Representations (pp. 1-16). https://openreview.net/forum?id=H1eA7AEtvSLarsen, V. H., Thorsrud, L. A., & Zhulanova, J. (2021). News-driven inflation expectations and information rigidities. Journal of Monetary Economics, 117, 507-520.Lipton, Z. (2016). The Mythos of Model Interpretability. Communications of the ACM, 61 (10), 36-43. https://doi.org/10.1145/3233231Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. https://arxiv.org/pdf/1907.11692.pdf. Mamedli, M., & Shibitov, D. (2021). Forecasting Russian CPI with Data Vintages and Machine Learning Techniques. The Bank of Russia, Working Paper Series No. 70, 37. https://cbr.ru/StaticHtml/File/120016/wp-apr21_e.pdf. Mikolov, T., Cernocky, J., & Khudanpur, S. (2010). Recurrent neural network based language model. In: Interspeech 2010, Eleventh annual conference of the international speech communication association (pp. 1045–1048). Japan. https://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdfPeirano, R., Kristjanpoller, W., & Minutolo, M. C. (2021). Forecasting inflation in Latin American countries using a SARIMA–LSTM combination. Soft Computing, 25, 10851–10862. https://doi.org/10.1007/s00500-021-06016-5Petkevič, V. (2018). Media sentiment analysis for measuring perceived trust in government. Socialinis ugdymas, 50 (3), 23-45.Petrova, D. (2022). Assessment of inflation expectations based on internet data. Prikladnaya ekonometrika [Applied econometrics], 66, 25-38. (In Russ.)Ramadhani, P. P., & Hadi, S. (2021). Text classification on the Instagram caption using support vector machine. Journal of Physics: Conference Series, 1722, 012023. https://iopscience.iop.org/article/10.1088/1742–6596/1722/1/012023/pdfRanyard R., Del Missier F., Bonini N., Duxbury D., Summers B. (2008). Perceptions and expectations of price changes and inflation: A review and conceptual framework. Journal of Economic Psychology, 29 (4), 378–400.Santhosh Baboo, S., & Amirthapriya, M. (2022). Comparison of Machine Learning Techniques on Twitter Emotions Classification. SN Computer Science, 3, 35. https://doi.org/10.1007/s42979-021-00889-xSchembri, L. L. (2020). Perceived inflation and reality: understanding the difference. Bank of Canada. https://www.bankofcanada.ca/2020/08/perceived-inflation-reality-understanding-the-difference/Semiturkin, O., & Shevelev, A. (2022). Forecasting Regional Inflation Rates Using Machine Learning Methods: The Case of Siberia Macroregion. The Bank of Russia Working Paper Series No. 91, 16. https://cbr.ru/StaticHtml/File/134578/wp_91.pdf Shah, K., Patel, H., Sanghvi, D., & Shah, M. (2020). A comparative analysis of logistic regression, random forest and KNN models for the text classification. Augmented Human Research, 5, 12. https://doi.org/10.1007/s41133-020-00032-0Shcherbakov, V. S., Kharlamova, M. S., & Gartvich, R. E. (2022). Methods and models for nowcasting economic indicators with help of search queries. In: Materialy mezhregionalnoy nauchno-prakticheskoy onlayn-konferentsii “Razvitie ekonomiki regiona: prostranstvennaya transformatsiya, globalnye vyzovy i perspektivy ekonomicheskogo rosta” [Materials of interregional scientific and practical online conference “Development of the regional economy: sustainable transformation, global challenges and prospects of economic growth”] (pp. 117-127). (In Russ.)Shulyak, E. (2022). Macroeconomic Forecasting Using Data from Social Media. Dengi i kredit [Russian Journal of Money and Finance], 81 (4), 86–112. (In Russ.)Subies, G. G., Sánchez, D. B., & Vaca, A. (2021). BERT and SHAP for Humor Analysis based on Human Annotation. http://ceur-ws.org/Vol-2943/haha_paper2.pdfThrun, S., & Pratt, L. (1998). Learning to learn: Introduction and overview. In: Learning to Learn (pp. 3-17). Springer, Boston, MA. https://doi.org/10.1007/978-1-4615–5529-2_1Turc, I., Chang, M.-W., Lee, K., & Toutanova, K. (2019). Well-Read Students Learn Better: On the Importance of Pre-training Compact Models. https://doi.org/10.48550/arXiv.1908.08962Vaswani, A., Shazeer, N. M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is All You Need. https://doi.org/10.48550/arXiv.1706.03762Ahmed, H., Traore, I., & Saad, S. (2017). Detecting opinion spams and fake news using text classification. Security and Privacy, 1 (1), 1-15. https://doi.org/10.1002/spy2.9Angelico, C., Marcucci, J., Miccoli, M., & Quarta, F. (2022). Can We Measure Inflation Expectations Using Twitter? Journal of Econometrics, 228 (2), 259-277.Armantier, O., De Bruine, W., Topa, G., van der Klaauw, W., & Zafar, B. (2015). Inflation expectations and behavior: do survey respondents act on their beliefs? International Economic Review, 56 (2), 505-536.Aromí, D., & Llada, M. (2020). Forecasting inflation with twitter. Asociación Argentina de Economía Política: Working Papers, No. 4308, 23. https://aaep.org.ar/works/works2020/AromiLlada.pdfBaybuza, I. (2018). Inflation forecasting using machine learning methods. Russian Journal of Money and Finance, 77 (4), 42-59.Bengio, Y., Ducharme, R., Vincent, P., & Jauvin, C. (2003). A neural probabilistic language model. Journal of machine learning research, (3), 1137–1155.Bradbury, J., Merity, S., Xiong, C., & Socher, R. (2016). Quasi-recurrent neural networks. https://arxiv.org/abs/1611.01576 Chakraborty, C., & Joseph, A. (2017). Machine Learning at Central Banks. Bank of England, Working Paper No. 674, 85. https://www.bankofengland.co.uk/-/media/boe/files/working-paper/2017/machine-learning-at-central-banks.pdf?la=en&hash=EF5C4AC6E7D7BDC1D68A4BD865EEF3D7EE5D7806Coibion, O., Gorodnichenko, Y., & Kamdar, R. (2018). The formation of expectations, inflation, and the Phillips curve. Journal of Economic Literature, 56 (4), 1447–1491.De Bruine, W., Manski, F. Ch., Topa, G., & van der Klaauw, W. (2011). Measuring consumer uncertainty about future inflation. Journal of Applied Econometrics, 26 (33), 454-478.Devlin, J., & Chang, M.-W. (2018). Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing. https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.htmlDevlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Vol. 1 (pp. 4171–4186). Minneapolis, Minnesota. https://doi.org/10.18653/v1/N19-1423Frisch, H. (1990). Theories of inflation. Cambridge University Press, 276.Giannone, D., Reichlin, L., & Small, D. (2008). Nowcasting: The real-time informational content of macroeconomic data. Journal of Monetary Economics, 55 (4), 665–676.Goloshchapova, I., & Andreev, M. (2017). Measuring inflation expectations of the Russian population with the help of machine learning. Voprosy ekonomiki, (6), 71-93. (In Russ.)Gurov, I. N. (2022). Inflation in Russia: differences between official data and population’s perception. Voprosy statistiki, 29 (3), 100-111. (In Russ.)Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Zhang, L., Han, W., Huang, M., Jin, Q., Lan, Y., Liu, Y., Liu, Z., Lu, Z., Qiu, X., Song, R., Tang, J., Wen, J.-R., Yuan, J., … Zhu, J. (2021). Pre-trained models: Past, present and future. AI Open, (2), 225-250. https://doi.org/10.1016/j.aiopen.2021.08.002Hinton, G., Vinyals, O., & Dean, J. (2019). Distilling the Knowledge in a Neural Network. NIPS 2014 Deep Learning Workshop, 9. https://doi.org/10.48550/arXiv.1503.02531Hu, Y. (2019). Text mining and data information analysis for network public opinion. Data Science Journal, 18 (7), 1-8. Jalal, N., Mehmood, A., Choi, G. S., & Ashraf, I. (2022). A novel improved random forest for text classification using feature ranking and optimal number of trees. Computer and Information Sciences, 34 (6), 2733–2742. https://doi.org/10.1016/j.jksuci.2022.03.012Jijo, B. T., & Abdulazeez, A. M. (2021). Classification Based on Decision Tree Algorithm for Machine Learning. Journal of Applied Science and Technology Trends, 2 (01), 20-28.Kokalj, E., Škrlj, B., Lavrač, N., Pollak, S., & Robnik-Šikonja, M. (2021). BERT meets Shapley: Extending SHAP Explanations to Transformer-based Classifiers. In: Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation (pp. 16-21). Association for Computational Linguistics. https://aclanthology.org/2021.hackashop-1.3/Kuratov, Y. M. (2020). Spetsializatsiya yazykovykh modeley dlya primeneniya k zadacham obrabotki estestvennogo yazyka [Specialization of language models for application to natural language processing tasks]. https://mipt.ru/upload/medialibrary/099/dissertatsiya-kuratov.pdf (In Russ.)Kuratov, Y., & Arkhipov, M. (2019). Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language. https://doi.org/10.48550/arXiv.1905.07213Lamla, M., & Lein, S. (2014). The role of media for consumers’ inflation expectation formation. Journal of Economic Behavior & Organization, 106, 62-77.Lan, Zh., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2020). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In: International Conference on Learning Representations (pp. 1-16). https://openreview.net/forum?id=H1eA7AEtvSLarsen, V. H., Thorsrud, L. A., & Zhulanova, J. (2021). News-driven inflation expectations and information rigidities. Journal of Monetary Economics, 117, 507-520.Lipton, Z. (2016). The Mythos of Model Interpretability. Communications of the ACM, 61 (10), 36-43. https://doi.org/10.1145/3233231Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. https://arxiv.org/pdf/1907.11692.pdf. Mamedli, M., & Shibitov, D. (2021). Forecasting Russian CPI with Data Vintages and Machine Learning Techniques. The Bank of Russia, Working Paper Series No. 70, 37. https://cbr.ru/StaticHtml/File/120016/wp-apr21_e.pdf. Mikolov, T., Cernocky, J., & Khudanpur, S. (2010). Recurrent neural network based language model. In: Interspeech 2010, Eleventh annual conference of the international speech communication association (pp. 1045–1048). Japan. https://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdfPeirano, R., Kristjanpoller, W., & Minutolo, M. C. (2021). Forecasting inflation in Latin American countries using a SARIMA–LSTM combination. Soft Computing, 25, 10851–10862. https://doi.org/10.1007/s00500-021-06016-5Petkevič, V. (2018). Media sentiment analysis for measuring perceived trust in government. Socialinis ugdymas, 50 (3), 23-45.Petrova, D. (2022). Assessment of inflation expectations based on internet data. Prikladnaya ekonometrika [Applied econometrics], 66, 25-38. (In Russ.)Ramadhani, P. P., & Hadi, S. (2021). Text classification on the Instagram caption using support vector machine. Journal of Physics: Conference Series, 1722, 012023. https://iopscience.iop.org/article/10.1088/1742–6596/1722/1/012023/pdfRanyard R., Del Missier F., Bonini N., Duxbury D., Summers B. (2008). Perceptions and expectations of price changes and inflation: A review and conceptual framework. Journal of Economic Psychology, 29 (4), 378–400.Santhosh Baboo, S., & Amirthapriya, M. (2022). Comparison of Machine Learning Techniques on Twitter Emotions Classification. SN Computer Science, 3, 35. https://doi.org/10.1007/s42979-021-00889-xSchembri, L. L. (2020). Perceived inflation and reality: understanding the difference. Bank of Canada. https://www.bankofcanada.ca/2020/08/perceived-inflation-reality-understanding-the-difference/Semiturkin, O., & Shevelev, A. (2022). Forecasting Regional Inflation Rates Using Machine Learning Methods: The Case of Siberia Macroregion. The Bank of Russia Working Paper Series No. 91, 16. https://cbr.ru/StaticHtml/File/134578/wp_91.pdf Shah, K., Patel, H., Sanghvi, D., & Shah, M. (2020). A comparative analysis of logistic regression, random forest and KNN models for the text classification. Augmented Human Research, 5, 12. https://doi.org/10.1007/s41133-020-00032-0Shcherbakov, V. S., Kharlamova, M. S., & Gartvich, R. E. (2022). Methods and models for nowcasting economic indicators with help of search queries. In: Materialy mezhregionalnoy nauchno-prakticheskoy onlayn-konferentsii “Razvitie ekonomiki regiona: prostranstvennaya transformatsiya, globalnye vyzovy i perspektivy ekonomicheskogo rosta” [Materials of interregional scientific and practical online conference “Development of the regional economy: sustainable transformation, global challenges and prospects of economic growth”] (pp. 117-127). (In Russ.)Shulyak, E. (2022). Macroeconomic Forecasting Using Data from Social Media. Dengi i kredit [Russian Journal of Money and Finance], 81 (4), 86–112. (In Russ.)Subies, G. G., Sánchez, D. B., & Vaca, A. (2021). BERT and SHAP for Humor Analysis based on Human Annotation. http://ceur-ws.org/Vol-2943/haha_paper2.pdfThrun, S., & Pratt, L. (1998). Learning to learn: Introduction and overview. In: Learning to Learn (pp. 3-17). Springer, Boston, MA. https://doi.org/10.1007/978-1-4615–5529-2_1Turc, I., Chang, M.-W., Lee, K., & Toutanova, K. (2019). Well-Read Students Learn Better: On the Importance of Pre-training Compact Models. https://doi.org/10.48550/arXiv.1908.08962Vaswani, A., Shazeer, N. M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is All You Need. https://doi.org/10.48550/arXiv.1706.03762