Copyright Page
© 2023 International Monetary Fund
WP/23/40
IMF Working Paper*
Strategy, Policy and Review Department
AI and Macroeconomic Modeling: Deep Reinforcement Learning in an RBC Model
Prepared by Tohid Atashbar and Rui (Aruhan) Shi
Authorized for distribution by Stephan Danninger
February 2023
IMF Working Papers describe research in progress by the author(s) and are published to elicit comments and to encourage debate. The views expressed in IMF Working Papers are those of the author(s) and do not necessarily represent the views of the IMF, its Executive Board, or IMF management.
ABSTRACT: This study seeks to construct a basic reinforcement learning-based AI-macroeconomic simulator. We use a deep RL (DRL) approach (DDPG) in an RBC macroeconomic model. We set up two learning scenarios, one of which is deterministic without the technological shock and the other is stochastic. The objective of the deterministic environment is to compare the learning agent’s behavior to a deterministic steady-state scenario. We demonstrate that in both deterministic and stochastic scenarios, the agent’s choices are close to their optimal value. We also present cases of unstable learning behaviours. This AI-macro model may be enhanced in future research by adding additional variables or sectors to the model or by incorporating different DRL algorithms.
RECOMMENDED CITATION: Atashbar,T. and Shi, R.A. 2023. “AI and Macroeconomic Modeling: Deep Reinforcement Learning in an RBC model”, IMF Working Papers, WP/22/40.
| JEL Classification Numbers: | C63, C54; D83; D87; E37 |
| Keywords: | Reinforcement learning; Deep reinforcement learning; Artificial intelligence, RL; DRL; Learning algorithms; Macro modeling, RBC; Real business cycles; DDPG; Deep deterministic policy gradient; Actor-critic algorithms |
| Author’s E-Mail Address: | tatashbar@imf.org; ashi@imf.org |
| JEL Classification Numbers: | C63, C54; D83; D87; E37 |
| Keywords: | Reinforcement learning; Deep reinforcement learning; Artificial intelligence, RL; DRL; Learning algorithms; Macro modeling, RBC; Real business cycles; DDPG; Deep deterministic policy gradient; Actor-critic algorithms |
| Author’s E-Mail Address: | tatashbar@imf.org; ashi@imf.org |
Title Page
WORKING PAPERS
AI and Macroeconomic Modeling: Deep Reinforcement Learning in an RBC Model
Prepared by Tohid Atashbar and Rui (Aruhan) Shi
Contents
GLOSSARY
INTRODUCTION
I. AN OVERVIEW OF THE LITERATURE
II. A REAL BUSINESS CYCLE (RBC) MODEL
A. Households
B. Firms
C. Functional forms and parameters
D. A deterministic steady state
III. AI EXPERIMENTS
A. Experiment I: deterministic environment
B. Experiment II: stochastic environment
C. Issues during learning
IV. CONCLUSION
ANNEX I. DDPG ALGORITHM
REFERENCES
FIGURES
Figure 1. SL, UL and RL in ML
Figure 2 Labor hours during training (200 episodes)
Figure 3 Labor hour series during training and testing
Figure 4 Distance the steady state (SS) values for labor hour and consumption
Figure 5 Productivity shock series zt
Figure 6 Simulated series during 100 testing periods
Figure 7 Labor hour choice before and after learning (200 episode)
Figure 8 Distance to deterministic steady states (SS) for labor hour and consumption
Figure 9 Distance to deterministic steady states (SS) for output and investment
Figure 10 Output per unit of labor
Figure 11 Investment per unit of labor
TABLES
Table 1. Baseline parameters for RBC model
Table 2 Algorithm related parameters
Table 3 RL set up of the RBC model
Glossary
| AGI | Artificial General Intelligence |
| AI | Artificial Intelligence |
| ANN | Artificial Neural Networks |
| DDPG | Deep Deterministic Policy Gradient |
| DL | Deep learning |
| DNN | Deep Neural Network |
| DPG | Deterministic Policy Gradient |
| DQN | Deep Q-Network |
| DRL | Deep Reinforcement Learning |
| MADDPG | Multi-Agent Deep Deterministic Policy Gradient |
| RBC | Real Business Cycle |
| RL | Reinforcement Learning |
| SAC | Soft Actor-Critic |
| SL | Supervised Learning |
| TD3 | Twin Delayed DDPG |
| UL | Unsupervised Learning |
The authors would like to thank Stephan Danninger for his helpful comments and suggestions. We appreciate the views and suggestions provided by Mico Mrkaic, Dmitry Plotnikov, Sergio Rodriguez and attendees at the IMF SPR Macro Policy Division Brownbag Seminar. Comments by Allan Dizioli are also gratefully acknowledged. All errors remain our own.