Copyright Page
© 2022 International Monetary Fund
WP/22/259
IMF Working Paper
Strategy, Policy, and Review Department
Deep Reinforcement Learning: Emerging Trends in Macroeconomics and Future Prospects Prepared by Tohid Atashbar and Rui (Aruhan) Shi
Authorized for distribution by Stephan Danninger
December 2022
IMF Working Papers describe research in progress by the author(s) and are published to elicit comments and to encourage debate. The views expressed in IMF Working Papers are those of the author(s) and do not necessarily represent the views of the IMF, its Executive Board, or IMF management.
ABSTRACT: The application of Deep Reinforcement Learning (DRL) in economics has been an area of active research in recent years. A number of recent works have shown how deep reinforcement learning can be used to study a variety of economic problems, including optimal policy-making, game theory, and bounded rationality. In this paper, after a theoretical introduction to deep reinforcement learning and various DRL algorithms, we provide an overview of the literature on deep reinforcement learning in economics, with a focus on the main applications of deep reinforcement learning in macromodeling. Then, we analyze the potentials and limitations of deep reinforcement learning in macroeconomics and identify a number of issues that need to be addressed in order for deep reinforcement learning to be more widely used in macro modeling.
RECOMMENDED CITATION: T. Atashbar and R.A. Shi 2022. “ Deep Reinforcement Learning: Emerging Trends in Macroeconomics and Future Prospects”, IMF Working Papers, WP/22/259.
| JEL Classification Numbers: | C63, C89, D85, D87 |
| Keywords: | Reinforcement learning; Deep reinforcement learning; Artificial intelligence, RL; DRL; Learning algorithms; Macro modeling |
| Author’s E-Mail Address: | tatashbar@imf.org; ashi@imf.org |
| JEL Classification Numbers: | C63, C89, D85, D87 |
| Keywords: | Reinforcement learning; Deep reinforcement learning; Artificial intelligence, RL; DRL; Learning algorithms; Macro modeling |
| Author’s E-Mail Address: | tatashbar@imf.org; ashi@imf.org |
Title page
WORKING PAPERS
Deep Reinforcement Learning: Emerging Trends in Macroeconomics and Future Prospects
Prepared by Tohid Atashbar, and Rui (Aruhan) Shi
Contents
GLOSSARY
INTRODUCTION
I. WHAT IS REINFORCEMENT LEARNING AND DEEP REINFORCEMENT LEARNING?
A. Brief History
B. Theory
C. Recent Applications
II. ECONOMIC DEEP REINFORCEMENT LEARNING: APPLICATIONS AND EMERGING TRENDS IN MACROECONOMICS
A. Solution Methods
B. Bounded Rationality, Learning and Convergence
III. DEEP REINFORCEMENT IN MACROECONOMICS: PROSPECTS AND ISSUES
A. Prospects
B. Issues
IV. CONCLUSION
ANNEX
Full Algorithms
REFERENCES
FIGURES
1. A Markov Decision Process (A Reinforcement Learning Problem)
2. Comparison of RL and other methods
3. RL algorithm overview
4. A feedforward deep ANN
5. Workflow of deep RL algorithms
6. Multiagent RL
TABLE
1. Terminologies in reinforcement learning
Glossary
| AC | Actor-Critic |
| A2C | Advantage Actor-Critic |
| A3C | Asynchronous Advantage Actor Critic |
| ANNs | Artificial Neural Networks |
| CDRL | Causal Deep Reinforcement Learning |
| DL | Deep Learning |
| DRL | Deep Reinforcement Learning |
| DDPG | Deep Deterministic Policy Gradients |
| DQN | Deep Q Networks |
| MDP | Markov Decision Processes |
| ML | Machine Learning |
| RL | Reinforcement Learning |
| PPO | Proximal Policy Optimization |
| QRL | Quantum reinforcement learning |
| SARSA | State-Action-Reward-State-Action |
| TD | Temporal Difference |
| TRPO | Trust Region Policy Optimization |