The IMF plays a central role in the adjustment efforts of its member countries by assisting in the design of appropriate adjustment programs to achieve viability of the balance of payments, price stability, and sustained high growth, and by providing financing to support these programs.1 An IMF-supported adjustment program includes a mix of stabilization and structural reform measures aimed at restoring a sustainable balance between aggregate demand and supply, while simultaneously expanding the production of tradables. Such a program takes the form of a set of policy intentions outlined by the government in a “Letter of Intent” that the IMF judges to warrant financial support. This financial support is thus conditional on the policy measures being carried out (Williamson, 1983).2 The targets for the key macroeconomic variables (which in a typical program include the level of net international reserves, the current account balance, inflation, and the growth of real GDP),3 the choice of policies to achieve those targets, and the amount of financing provided by the IMF all result from extensive and detailed negotiations between the country authorities and the Fund. Thus, an IMF-supported adjustment program reflects both the individual economic situation of the country and the preferences of the government.
While the IMF’s role is generally regarded as both necessary and useful, an important question often raised in connection with IMF-supported adjustment programs is whether they work. In other words, are those programs effective in achieving their objectives of improving the current account balance, increasing international reserves, lowering inflation, and raising the growth rate? This is essentially an empirical question that requires evaluating the effects of past programs on the macroeconomic variables of interest. Such evaluations are conducted regularly by the IMF’s Policy Development and Review Department, and the results are reported to the IMF’s Executive Board.4 In addition, a number of studies have been undertaken over the past 20 years, both inside and outside the IMF, that examine this question using a variety of empirical methods.
Although the final verdict is not in, a growing consensus seems to have emerged that, on balance, IMF-supported programs have worked. But it turns out that evaluating the effects of programs is not easy, which is one reason for the controversy on the issue.
This chapter reviews evidence on the effects of IMF-supported programs, paying special attention to the methodologies employed in various studies to estimate these effects. The chapter is in the nature of a survey and relies exclusively on published studies. As such, it does not contain any new empirical evidence, but focuses instead on outlining the current state of thinking on the subject and assessing the existing evidence.
Conceptual and Methodological Issues
Defining Program Effectiveness
Defining the effectiveness of IMF-supported adjustment programs is not a straightforward task, for at least two reasons. First, although the success of a program is measured in terms of macroeconomic outcomes (for example, an improved balance of payments), the conditions agreed between the IMF and the country relate to policy variables (for example, the expansion of domestic credit or reduction in the fiscal deficit). While it is relatively easy to check whether the program country has implemented the agreed policy changes via the setting of performance criteria on key policy variables, it is more difficult to know whether these changes would lead to the desired macroeconomic outcomes. This is because IMF-supported programs are complex packages of policy measures that include, among other things, monetary and exchange rate policies, fiscal measures, policies to raise investment and improve its efficiency, trade liberalization, labor market reforms, and financial sector reforms. The theory underlying the dynamic links between such a policy package—combining aggregate demand policies with supply-enhancing and relative-price policies—and a set of multiple macroeconomic targets is not well established. Thus, from a theoretical view-point the adjustment package is not necessarily guaranteed to achieve the desired outcomes.
Second, the IMF-supported program is only one of many macroeconomic “shocks” to the country with a program. External shocks, such as changes in the terms of trade or in the cost of servicing foreign debt, will also affect the country’s ability to achieve the macroeconomic objectives of the program. Measures of program effectiveness have to filter out these unanticipated exogenous shocks.
In general, the links between the program and the eventual external and internal adjustment are not direct. The program’s policy recommendations must first be translated by the country into actual policy actions. These actions, and the financial resources provided to support the program, must interact with the structure of the economy to bring about the desired economic adjustment. While the government is implementing the policy reforms, exogenous factors (terms of trade movements, weather fluctuations, productivity shocks) can work through the economic structure to affect the macroeconomic outcome. These factors may help the cause of macroeconomic adjustment, but at times they may also work against the policy reforms.
Since an IMF-supported program seeks to achieve an adjustment in policies, and hence in macroeconomic outcomes, the proper measure of the effectiveness of the program has to isolate the impact of the program on the outcomes and compare that to the alternative of what would have happened in the absence of the program. In other words, a comparison has to be made between the actual outcome due to the program with the counterfactual—that is, the macroeconomic outcome that would have resulted had there been no program. The counter-factual is the right yardstick against which to assess program performance and is the standard most widely used in economics to measure the impact of government policy interventions.
Of course, the counterfactual is not observed. In reality, countries fall into only one of two states—program or nonprogram—but never both at the same time. An accurate measure of program effects requires comparison of experiences in these two states. Thus, the counterfactual has to be constructed, and the ideal counterfactual will be an equilibrium for the same economy facing the same exogenous factors, and differing only in its nonparticipation in the IMF-supported program.5
The informational requirements of the ideal measure of effectiveness are quite daunting. To compare the two states requires a macroeconomic model whose parameters remain invariant with respect to different policy settings. If the structural parameters themselves change, the comparison will require two models of the economy—one describing the structure when the country has a program, and the other describing the structure in the absence of a program. Information is also required on the values placed on the structural parameters and the policy reaction function parameters. Clearly, this amount of information is not available for the typical program country. Thus, various methods have been advanced to construct a counterfactual that can be a good approximation to the true counterfactual.
Alternative Approaches for Estimating the Effects of IMF-Supported Programs
In the literature four main approaches have been suggested to measure the effects of IMF-supported programs. Each approach constructs a different type of counterfactual, as follows:6
-
The before-after (BA) approach compares the macroeconomic performance under a program (or after it has been initiated) with performance before the program.
-
The with-without (WW) approach compares the macroeconomic performance in program countries with the performance in a control group of nonprogram countries.
-
The generalized evaluation estimator (GEE) approach compares performance in program and nonprogram countries, adjusting for differences in initial conditions among the countries and controlling for exogenous (principally external) influences.
-
The comparison of simulations (SIM) approach compares the simulated performance under IMF-supported program policies with simulated performance under some other set of policies.
The Before-After Approach
The basic advantage of the BA approach is ease of calculation. One merely has to compare the macroeconomic outcomes in the preprogram period with those in the program period to estimate the program effects.7 The problem, however—and it is an overriding one—is that the BA approach assumes that all other things remain equal. It thus will not yield an estimate of the independent effects of programs when the nonprogram determinants of the macroeconomic outcomes change between the preprogram and the program period. In reality, however, these nonprogram determinants do change markedly from year to year. Examples of such nonprogram determinants would be external factors like industrial country growth rates, terms of trade variations, and movements in international interest rates, as well as domestic factors such as productivity shocks and shifts in weather conditions. This means that the BA estimation of program effects will typically be biased and will vary over time. They will be biased because this approach incorrectly attributes all of the changes in outcomes between the preprogram and program periods to program factors.8 They will also vary over time, because the estimated program effects for a given year will often be dominated by specific nonprogram influences of that year. Thus, for example, if the terms of trade improve between year t and year t + 1, all t + 1 programs will look as if they performed very well, whereas if the terms of trade decline in some later year, all programs for that later year will look as if they performed poorly. In other words, the outcomes may have nothing to do with the program.
These shortcomings of the BA approach also make it a poor estimator of the ideal counterfactual, since it assumes a counterfactual in which policies and the external environment remain constant at their preprogram values.9 The basic reason why the BA approach is flawed as an estimator of the counterfactual is that the situation prevailing before the program is not likely to be a good predictor of what would have happened in the absence of the program, because governments turn to the IMF for help precisely when past policies have become unviable.
The With-Without Approach
The WW approach seeks to get closer to the ideal counterfactual. It is designed to overcome the inability of the BA approach to distinguish between program and nonprogram determinants of macroeconomic outcomes. The basic reasoning behind the WW approach is as follows. Assume that program countries and nonprogram countries are subject to the same nonprogram determinants—that is, they face the same external environment. Then by comparing before-after changes in outcomes in program countries with those in a control group of nonprogram countries, the effects of external factors will cancel out, leaving the difference in group performance to reflect only the effects of IMF-supported programs. Put in terms of the counterfactual, the idea is to use the observed performance in nonprogram countries as an estimate of what the performance in program countries would have been in the absence of an IMF-supported program.
However, the WW approach has problems as well. The main problem is that program countries can and do differ systematically from nonprogram countries before the start of a program, and this difference matters in performance evaluation. Basically, the point is that program countries are not randomly selected. Instead, they are “adversely” selected because of their relatively poor economic performance prior to the program period. This should not be surprising, because an essential requirement for IMF financial support is that the country have a balance of payments need. This alone suggests that program countries would be expected to have had weaker--than-average external positions—namely the current account balance and level of international reserves—when the program was implemented. The implication of nonrandom selection of program countries is that simple WW estimates of program effects will be biased. Intuitively, the bias occurs because, under nonrandom selection, the WW estimator attributes differences in outcomes exclusively to program status, when the difference in starting or initial positions is itself a cause of differences in the subsequent performance of the two groups. Furthermore, the direction of the bias can go either way. If past economic difficulties signal less serious current difficulties—even in the absence of a program—then the WW approach will overstate the positive effects of an IMF-supported program. Conversely, if past difficulties signal even more serious current difficulties, then the effects of an IMF-supported program will be understated.
The Generalized Evaluation Estimator Approach
The recognition of the biases inherent in both the BA and the WW approaches led to the development of the GEE approach.10 The GEE approach modifies the simple WW approach in two important ways. First, it accepts the nonrandom selection of program countries, identifies the specific differences between program and nonprogram countries in the preprogram period, and then controls for the differences in initial positions by comparing subsequent economic performance.11 Second, it attempts to capture the effects of policy and other variables on the macroeconomic outcomes, taking into account how policies would have evolved in the absence of a program.
To make the GEE approach operative requires identifying not only the initial conditions, but also the relevant reduced-form relationships linking policy instruments and other exogenous variables to policy targets, and the policy reaction functions that show how policy instruments evolve as the state of the economy changes. The reduced-form equations are needed to determine the effects of alternative policies on the target variables, controlling for exogenous variables. The policy reaction functions are needed to determine what policies would have been chosen, given preprogram conditions, in the absence of a program.
These empirical relationships require a substantial amount of data and are thus not easily implemented. There will still be some margin of error between the results of the GEE approach and the ideal counterfactual. However, this error is likely to be considerably smaller than the one for the BA and WW approaches. In summary, there are sound econometric reasons to prefer the GEE measure over the BA and WW approaches. First, the GEE approach uses more information about the program country and other nonprogram countries, and can thus define a more precise and accurate counterfactual. Second, statistical techniques are available to correct for selection bias with this methodology, whereas such corrections cannot be easily made with the BA and WW approaches.
The Comparison of Simulations Approach
Finally, there is the SIM approach, which unlike the other three approaches does not determine program effects from the actual macroeconomic outcomes in the program countries. Instead, it relies on simulations of econometric models to infer the hypothetical performance of policies included in an IMF-supported program and an alternative policy package. If the aim of the overall exercise is to evaluate the results of specific IMF-supported adjustment programs, then the use of actual program outcomes is indispensable. However, if the purpose is to evaluate the design and effectiveness of IMF-supported adjustment programs in general, then examining the likely effects of alternative policy packages can be quite useful and revealing.
In broad terms, the SIM approach carries three advantages. First, one can draw on a wider body of adjustment experience, since the database need not be restricted to countries with IMF-supported programs. Second, since the policy simulations are specified, one need not be concerned that incomplete implementation of policies, which is often a problem in IMF-supported programs, will blur the results. In contrast, the approaches that rely on actual outcomes require the untangling of the effects of the program from the degree of implementation, and this is usually not done. Third, the SIM approach, by its very nature, focuses on the relationship between policy instruments and policy targets. As such, it provides better information on how programs work than do approaches that look at the bottom line of policy targets.
There are, however, practical problems with the SIM approach. This approach requires an econometric model that incorporates the relations between various policies and certain important macroeconomic variables. Although several attempts have been made to build such models for developing countries, as yet no single model is available that covers the entire range of policy measures contained in a typical IMF-supported program.12 Existing econometric models are clearly unable to analyze all the questions relating to such programs. In particular, they do not capture the complex ways in which policy variables are related to the ultimate program objectives. Even if a suitable model were available, one would still have to face up to the critique of ex ante econometric policy evaluation (the so-called Lucas critique). Specifically, the parameters in econometric models may not remain invariant to changes in the policy regime, so it would be incorrect to treat such parameters as fixed across alternative policy simulations. In other words, the actual effects of hypothetical policy packages on macroeconomic variables may turn out to be quite different from the simulated results, and in ways that are difficult to know in advance.13 An additional concern is that, because of credibility factors, the effect of a given policy may be different when it is implemented within the context of an IMF arrangement than when it is implemented outside it. Agents may believe, for example, that policies agreed to with the IMF are more likely to be carried through and thus would be more inclined to change their behavior.
Empirical Applications
A summary of the studies evaluating the effects of IMF-supported programs is provided in Table 3.1. For convenience, the table focuses only on the principal macroeconomic variables—the overall balance of payments, the current account balance, inflation, and growth.14 Furthermore, comparisons of performance are made over a one-year time horizon, unless otherwise noted. Although comparing performance the year before the program with performance during the program year is essentially an arbitrary choice, most studies have restricted themselves to annual comparisons. In the discussion that follows, the studies are grouped according to the approach they used.
Summary of Empirical Evaluations of the Effects of IMF-Supported Programs
Direction of change: (+) indicates positive effect, (-) indicates negative effects, (0) indicates no effect An asterisk (*) indicates statistically significant at the 5 percent level, and (…) indicates not applicable.
IMF staff study.
Summary of Empirical Evaluations of the Effects of IMF-Supported Programs
| Effects1 | |||||||
|---|---|---|---|---|---|---|---|
| Study | Period | Number of Programs | Number of Countries | Balance of payments | Current account | Inflation | Growth |
| Before-after | |||||||
| Reichmann and Stillson (1978) 2 | 1963-72 | 79 | … | 0 | … | 0 | + |
| Connors (1979) | 1973-77 | 31 | 23 | 0 | 0 | 0 | 0 |
| Killick (1984) | 1974-79 | 38 | 24 | 0 | 0 | _* | 0 |
| Zulu and Nsouli (1985) 2 | 1980-81 | 35 | 22 | … | 0 | 0 | 0 |
| Pastor (1987) | 1965-81 | … | 18 | +* | 0 | 0 | 0 |
| Killick, Malik, and Manuel (1995) | 1979-85 | … | 16 | +* | +* | -* | +* |
| Schadler and others (1993) 2 | 1983-93 | 55 | 19 | + | - | - | + |
| With-without | |||||||
| Donovan (1981) 2 | 1970-76 | 12 | 12 | … | … | - | + |
| Donovan (1982) 2 | 1971-80 | 78 | 44 | + | + | - | - |
| Loxley (1984) | 1971-82 | 38 | 38 | 0 | 0 | -* | 0 |
| Gylfason (1987) | 1977-79 | 32 | 14 | +* | … | … | 0 |
| Generalized evaluation estimator | |||||||
| Goldstein and Montiel (1986) 2 | 1974-81 | 68 | 58 | - | - | + | - |
| Khan (1990) 2 | 1973-88 | 259 | 69 | +* | +* | - | -* |
| Conway (1994) | 1976-86 | 217 | 73 | … | +* | - | -,+* |
| Bagci and Perraudin (1997) | 1973-92 | … | 68 | +* | +* | - | +* |
| Dicks-Mireaux, Mecagni, and Schadler (1997) 2 | 1986-91 | 88 | 74 | … | … | - | +* |
| Comparison of simulations | |||||||
| Khan and Knight (1981) 2 | 1968-75 | … | 29 | + | + | - | - |
| Khan and Knight (1985) 2 | 1968-75 | … | 29 | + | + | - | -,+* |
Direction of change: (+) indicates positive effect, (-) indicates negative effects, (0) indicates no effect An asterisk (*) indicates statistically significant at the 5 percent level, and (…) indicates not applicable.
IMF staff study.
Summary of Empirical Evaluations of the Effects of IMF-Supported Programs
| Effects1 | |||||||
|---|---|---|---|---|---|---|---|
| Study | Period | Number of Programs | Number of Countries | Balance of payments | Current account | Inflation | Growth |
| Before-after | |||||||
| Reichmann and Stillson (1978) 2 | 1963-72 | 79 | … | 0 | … | 0 | + |
| Connors (1979) | 1973-77 | 31 | 23 | 0 | 0 | 0 | 0 |
| Killick (1984) | 1974-79 | 38 | 24 | 0 | 0 | _* | 0 |
| Zulu and Nsouli (1985) 2 | 1980-81 | 35 | 22 | … | 0 | 0 | 0 |
| Pastor (1987) | 1965-81 | … | 18 | +* | 0 | 0 | 0 |
| Killick, Malik, and Manuel (1995) | 1979-85 | … | 16 | +* | +* | -* | +* |
| Schadler and others (1993) 2 | 1983-93 | 55 | 19 | + | - | - | + |
| With-without | |||||||
| Donovan (1981) 2 | 1970-76 | 12 | 12 | … | … | - | + |
| Donovan (1982) 2 | 1971-80 | 78 | 44 | + | + | - | - |
| Loxley (1984) | 1971-82 | 38 | 38 | 0 | 0 | -* | 0 |
| Gylfason (1987) | 1977-79 | 32 | 14 | +* | … | … | 0 |
| Generalized evaluation estimator | |||||||
| Goldstein and Montiel (1986) 2 | 1974-81 | 68 | 58 | - | - | + | - |
| Khan (1990) 2 | 1973-88 | 259 | 69 | +* | +* | - | -* |
| Conway (1994) | 1976-86 | 217 | 73 | … | +* | - | -,+* |
| Bagci and Perraudin (1997) | 1973-92 | … | 68 | +* | +* | - | +* |
| Dicks-Mireaux, Mecagni, and Schadler (1997) 2 | 1986-91 | 88 | 74 | … | … | - | +* |
| Comparison of simulations | |||||||
| Khan and Knight (1981) 2 | 1968-75 | … | 29 | + | + | - | - |
| Khan and Knight (1985) 2 | 1968-75 | … | 29 | + | + | - | -,+* |
Direction of change: (+) indicates positive effect, (-) indicates negative effects, (0) indicates no effect An asterisk (*) indicates statistically significant at the 5 percent level, and (…) indicates not applicable.
IMF staff study.
Before-After Approach
In the early literature the most popular approach for evaluating the effects of programs was the BA approach. The first study to use this approach to analyze IMF-supported programs was by Reichmann and Stillson (1978). The authors examined 79 IMF-supported programs implemented during 1963–72 and compared the behavior of the balance of payments, inflation, and growth during the two years before and after the programs were initiated. Using nonparametric statistical tests, they found that programs generally succeeded in slowing the rate of expansion of total domestic credit and credit to the government—the two key monetary policy variables. However, the balance of payments significantly improved in only about one-fourth of the programs. Of the 29 programs in countries with high inflation during the program period, the inflation rate fell in 6 of the 11 programs for which there was a notable deceleration in the rate of domestic credit expansion. In the 9 programs that included a devaluation, inflation was higher in 5. Finally, the authors examined growth performance for 70 programs and concluded that, on balance, IMF-supported programs did not adversely affect growth. In certain cases, growth declined after the start of the program relative to the previous year’s rate of growth, but this result was matched by several cases in which the growth rate rose.
A similar procedure was followed by Connors (1979), who examined 31 programs adopted during 1973–77, comparing the behavior of the main macroeconomic variables one year before and after the inception of the programs. Connors concluded that IMF-supported programs apparently had no discernible effects on growth, inflation, the current account deficit, and the overall balance of payments.
The study by Killick (1984) attempted to capture the effects of lags by comparing the preprogram behavior of the balance of payments, the current account, growth, and inflation with the behavior of the same variables one and two years after the program. Killick used nonparametric statistics and regression analysis to gauge the effects of 38 programs during 1974–79. His results showed no statistically significant effects—positive or negative—of IMF-supported programs on the balance of payments or the current account deficit. There was a small positive effect on growth, but inflation was also higher because of the programs. Zulu and Nsouli (1985) also constructed before-after measures of program effects in their study of 35 programs for 22 African countries. They found lower growth for most countries, worsening inflation in just over half the countries, and no general pattern with respect to either the current account position or the overall balance of payments.
Also looking at regional patterns, Pastor (1987) estimated program effects for 18 Latin American countries during 1965–81. Using one-year comparisons, and on the basis of alternative statistical tests, Pastor concluded that IMF-supported programs led to significant improvement in the balance of payments, but they appeared to have had no effect on the current account, inflation, or the rate of growth of nominal GDP.15
Killick, Malik, and Manuel (1995) updated and extended the earlier Killick (1984) study to examine the effects of programs in 16 countries during 1979–85, They found that in these cases the results were quite different. IMF-supported programs led to improvements in the balance of payments, the current account balance, and the growth rate, while inflation was reduced. Also, in contrast to the previous study, the estimates of the program effects were statistically significant.
Finally, the study by Schadler and others (1993) examined the performance of 19 countries that entered into Structural Adjustment Facility and Enhanced Structural Adjustment Facility arrangements with the IMF during 1983–93. The results, none of which were tested for statistical significance, indicate that whereas the overall balance of payments improved, the current account balance worsened. At the same time, inflation declined, and the growth rate rose, in the context of programs.
With-Without Approach
The WW approach was first used in two studies by Donovan (1981, 1982), which analyzed a sample of programs implemented from 1970 to 1980. Donovan compared changes in target variables in program countries with contemporaneous changes in the same variables for a control group of nonprogram countries. In both studies the control group consisted of all non-oil developing countries, and the comparisons were carried out over one-year and three-year time horizons. The results indicated relative improvements for program countries in the ratios of the current account to GDP and of the overall balance of payments to exports, and in the rate of inflation for both the 1970–76 and 1971–80 periods. However, the growth performance of program countries differed little from that of the control group.
Loxley (1984) applied the same types of tests as Donovan (1982) to a group of 38 least-developed countries (defined as countries with per capita incomes of less than $690 in 1980) that had programs with the IMF during 1971-82. His results were less definitive, however, than those obtained by Donovan. The least-developed countries with programs did no better, on average, in terms of current account, balance of payments, and growth performance relative to the control groups (comprising other least-developed countries without programs, the 44 program countries considered by Donovan, 1982, and all non-oil developing countries). The relative improvement was statistically significant only in the case of inflation, and then only in the three-year comparisons.
Gylfason (1987) also used a version of the WW approach, taking as a control group a set of nonprogram countries that had experienced economic difficulties in the preprogram period. He studied changes in the growth rate of domestic credit, in the ratio of the balance of payments to GDP, and in the growth of output. Then he conducted tests to see whether these variables differed statistically between program countries and the control group. The results indicated that program countries experienced statistically significant reductions in domestic credit growth and improvements in the ratio of the balance of payments to GDP, but no significant difference was evident between program countries and the control group in real output growth.
Generalized Evaluation Estimator
To correct for the biases inherent in the BA and WW approaches, Goldstein and Montiel (1986) developed the GEE approach. This approach has now become the estimator of choice in evaluating the effects of IMF-supported adjustment programs.16
Goldstein and Montiel applied the GEE approach to a sample of 68 programs for 58 countries implemented during 1974–81. The authors found that program countries systematically demonstrated weaker performance—that is, higher inflation, slower growth, and larger current account and balance of payments deficits—than nonprogram countries in the preprogram period. Adjusting for these preprogram differences in performance and taking into account the effects of policy instruments on targets, Goldstein and Montiel used regression analysis to estimate the program effects. Two sets of interesting results emerged from this study. First, the programs had no statistically significant effects on the current account and balance of payments, the rate of inflation, or the growth of real output. Second, estimates of program effects using the GEE approach were quite different from those obtained with the BA or WW approaches in terms of the signs of the coefficients measuring program effects.
The GEE approach was refined and applied by Khan (1990) to a much larger sample of 259 programs in 69 countries during the period 1973-88. Both one-year and two-year comparisons were used to pick up the dynamics of the relationship between the program and the target variables. Khan found programs were associated with an improvement in the balance of payments, but the improvement was statistically significant only when the period of evaluation was extended to two years after the inception of the program. The current account deficit was immediately reduced, and this effect was strengthened over time. But the inflation rate, while lowered, did not show up as statistically significant in any of the tests performed. Finally, the growth rate declined in the program year, but when the time horizon of the performance evaluation was extended beyond the program year, the adverse growth effects were diminished.17
Khan’s (1990) results on inflation were supported by Dicks-Mireaux, Mecagni, and Schadler (1997) for the more recent 1986–91 period and for a larger group of countries. These authors found that IMF-supported programs did reduce inflation, but the coefficient measuring the program effects was not statistically significant. On the other hand, the results for growth were quite different from those found by the other studies. The study showed that the implementation of a program led to an immediate improvement in growth, and this effect was found to be statistically significant.
Another reason why standard approaches to measuring the effects of programs may be inadequate, and this also applies to the GEE approach, is the possibility that participation in IMF-supported programs is selective, not random. Indeed, countries are likely to be “self-selecting” themselves into IMF-supported programs. Program participation is a conclusion of successful negotiations between the IMF and the countries involved. The success of the negotiations depends on the existence of balance of payments need, as well as a policy package that is expected to alleviate this need and improve the viability of the long-term balance of payments. Consequently, the country must be prepared to take the policy measures that are agreed and to determine that the costs of alternative financing are less than making the necessary changes in policy. Thus, financing availability, implementation capacity, and the magnitude of required changes are all elements of the program-participation decision. Participation takes place according to countries self-selecting themselves on the basis of their respective economic circumstances. Looked at in this manner, the choice of participation in IMF-supported programs is itself an endogenous variable, and ignoring this characteristic could lead to biased results. Thus, the participation decision has to be estimated simultaneously with the performance and policy-choice parameters that are part of the GEE approach to allow for the endogenous determination of the program-participation variable.
Two recent studies take selectivity and the decision to participate into account in estimating program effects. Conway (1994) applied the modified GEE approach to a sample of 217 programs implemented during 1976-86. The results were the following: the current account improved, inflation was lower, and the rate of growth first fell and then rose. All these effects were statistically significant. The Conway results were confirmed by a similar analysis done by Bagci and Perraudin (1997) for 68 program countries over the 1973–92 period. The authors used a maximum-likelihood estimation method—in contrast to Conway (1994), who employed two-stage least-squares—to estimate the two-equation model. Bagci and Perraudin found that IMF-supported programs improved the current account balance and the overall balance of payments, reduced inflation, and raised growth in the short run, and these changes were also all statistically significant.
Comparison of Simulations
An alternative to examining actual program experiences, as has been done in the cross-country studies described, is to perform policy simulations with a macroeconomic model to estimate program effects (the SIM approach). For example, Khan and Knight (1981) constructed a small dynamic econometric model and estimated its parameters on a pooled cross-section time-series sample of 29 developing countries, most of which had programs with the IMF. They then investigated the hypothetical effects of a stabilization program whose objective was to achieve an external balance target using policies that figure prominently in IMF-supported programs. The simulation experiments showed that such a program produced a sharp price deflation in the first year, followed by a temporary burst of inflation as prices returned to their equilibrium. Output, on the other hand, contracted sharply in the first year, then rose temporarily above its full-employment level, approaching equilibrium gradually over several years.
In a further study, Khan and Knight (1985) extended the simulation analysis to compare alternative policy packages. Specifically, they compared the results for the balance of payments, inflation, and real output growth of a package of demand-management policies (that is, a once-for-all reduction in the rates of growth of domestic credit and government expenditures, plus a devaluation) with a combined package of demand-management and structural policies (that is, the demand-management policies plus a set of structural reforms that would gradually raise the growth rate of capacity output). The demand management package improved the balance of payments almost immediately, but at the cost of a temporary higher rate of inflation and a short-run reduction in growth. The simulations of the combined package showed that structural policies could help to offset, at least partially, any short-run effects that result from demand restraint and the inflationary consequences of devaluation. Furthermore, the long-run effects on the balance of payments, inflation, and growth turned out to be much more favorable than the short-run effects.
Conclusions and Implications for Future Work
Over the past two decades, a number of empirical studies have examined the effects of IMF-supported adjustment programs on key macroeconomic variables, such as the current account and the overall balance of payments, inflation, and economic growth. Such evaluations play an important role in the design of programs because the lessons they yield, both positive and negative, can be incorporated into IMF operations. This chapter has provided an overview of the methodologies adopted and the results obtained by the studies, with a view to assessing where the IMF stands with respect to its knowledge of the effectiveness of past programs and where it needs to go to improve future evaluations of IMF-supported adjustment programs. The survey of the literature pointed to two broad conclusions.
First, the empirical analysis has been conducted using different methodologies, the relative merits of which deserve careful examination. Many of the earlier studies attempted to gauge program effectiveness by comparing macroeconomic outcomes in program countries with performance before the implementation of the program or with the observed performance of countries without programs. If the proper standard for measuring program effects is to compare the macroeconomic outcomes under a program with the outcomes that would have emerged in the absence of a program, or under a different set of policies, then none of the earlier approaches—labeled as the before-after (BA) and with-without (WW) approaches—is fully satisfactory. More recent studies attempt to apply the counterfactual criteria to evaluating program performance through estimation of policy-reaction functions for program and nonprogram countries and through simulation experiments with macroeconomic models. One can place more confidence in the results yielded by these later studies.
Second, it now appears that IMF-supported programs do improve the current account balance and the overall balance of payments. The results for inflation are less clear, however. Most of the recent studies indicate that the rate of inflation falls, but the change is generally found not to be statistically significant. In the case of growth, the results seem to indicate that output will be depressed in the short run as the demand-reducing elements of the policy package dominate. Over time, as macroeconomic stability is established and the structural reform elements of the program start to take hold, growth revives. These newer empirical results indicate that, on average, IMF-supported adjustment programs have been more effective in achieving their objectives than earlier analyses suggested.
What needs to be done for future evaluation work? The principal message that emerges from this survey is that comparing a program’s macroeconomic outcomes with the corresponding outcomes obtained under an alternative set of feasible policies is the most appropriate way of judging program effects. However, the difficulties in using this criteria should not be underestimated. By definition, determining the counterfactual involves a degree of subjectivity and is thus hard to employ in practice. Until a suitable econometric technique is developed to estimate the counterfactual, any study of the quantitative effects of IMF-supported programs will not yield complete answers. In this context, the generalized evaluation estimator (GEE) approach, modified to allow for the endogeneity of the decision to enter into a program, appears most promising and has to be considered seriously in future evaluations. This approach takes evaluation a long way toward estimating the “ideal” counterfactual.
The estimates from the GEE approach can also be improved by taking explicit account of the degree of implementation of the policies agreed between the IMF and the program country. In the studies to date, all programs have been treated alike, whether or not the policy intentions specified in the program were carried out. A sample of countries with varying implementation records can bias the judgment about effectiveness. Had the tests been restricted only to countries that had successfully implemented the recommended policies, an even more positive picture might have emerged. Future studies should attempt to separate successful implementers from other program countries, although this is not always easy to do when there are waivers and program modifications that alter program targets and policies.
Case studies, as opposed to large multicountry studies, permit a deeper analysis of program implementation.18 However, case studies are time-consuming and expensive, and it is difficult to generalize from the findings of a few case studies.19 Large cross-country samples, in contrast, are more amenable to the application of standard statistical techniques, since there are sufficient numbers of data points. But in the process, the evaluation loses some of the country-specific aspects of programs, including the degree to which the policies in the program were implemented. Based on this reasoning, supplementing the results from cross-country studies with information from case studies highlighting specific issues may well be a good compromise.
In conclusion, as long as IMF-supported programs are to be an integral part of the adjustment strategies of developing countries, the search for the most appropriate way to evaluate the effects of past programs must continue. Much progress has been made, but more refinements of the empirical methodology are still needed to increase confidence in the cross-country empirical analyses that attempt to answer the question: Do IMF-supported programs work?
Bibliography
Bagci, P., and William Perraudin, 1997, “Do IMF Programs Work?” Global Economic Institutions Working Paper (New York: Global Economic Institutions).
Connors, T., 1979, “The Apparent Effects of Recent IMF Stabilization Programs,” International Finance Discussion Paper 135 (Washington: U.S. Board of Governors of Federal Reserve System).
Conway, Patrick, 1994, “IMF Lending Programs: Participation and Impact,” Journal of Development Economics, Vol. 45 (December), pp. 365–91.
Conway, Patrick, 1998, “Evaluating IMF Programs: Methodology and Empirical Estimates,” (unpublished; Chapel Hill, North Carolina, May).
Corbo, Vittorio, and P. Rojas, 1992, “World Bank-Supported Adjustment Programs: Country Performance and Effectiveness,” in Adjustment Lending Revisited: Policies to Restore Growth, ed. by Vittorio Corbo, Stanley Fischer, and S. Webb (Washington: World Bank).
Corbo, Vittorio, P. Rojas, and Stanley Fischer, 1994, “Structural Adjustment, Stabilization, and Policy Reforms: Domestic and International Finance,” in Handbook of Development Economics Vol. 3, ed. by J. Behrman and T. N. Srinivasan (Amsterdam and New York: North-Holland).
Dicks-Mireaux, Louis, Mauro Mecagni, and Susan Schadler, 1997, “Evaluating the Effects of IMF Lending to Low-Income Countries” (unpublished; Washington: International Monetary Fund).
Donovan, Donal J., 1981, “Real Responses Associated with Exchange-Rate Action in Selected Upper Credit Tranche Stabilization Programs,” Staff Papers, International Monetary Fund, Vol. 28 (December), pp. 698–727.
Donovan, Donal J., 1982, “Macroeconomic Performance and Adjustment Under IMF-Supported Programs: The Experience of the Seventies,” Staff Papers, International Monetary Fund, Vol. 29 (June), pp. 171–203.
Faini, R., Jaime De Melo, A. Senhadji, and J. Stanton, 1991, “Growth-Oriented Adjustment Programs,” World Development, Vol. 19 (August), pp. 957–67.
Goldsbrough, David, Sharmini Coorey, Louis Dicks-Mireaux, Balazs Horvath, Kalpana Kochhar, Mauro Mecagni, Erik Offerdal, and Jianping Zhou, 1996, Reinvigorating Growth in Developing Countries: Lessons from Adjustment Policies in Eight Economies, IMF Occasional Paper 139 (Washington: International Monetary Fund).
Goldstein, Morris, and Peter J. Montiel, 1986, “Evaluating IMF Stabilization Programs with Multicountry Data: Some Methodological Pitfalls,” Staff Papers, International Monetary Fund, Vol. 33 (June), pp. 304–44.
Gomulka, S., 1995, “The IMF-Supported Programs of Poland and Russia, 1990-94: Principles, Errors and Results,” Journal of Comparative Economics, Vol. 20 (June), pp. 316–46.
Gylfason, T., 1987, “Credit Policy and Economic Activity in Developing Countries with IMF Stabilization Programs,” Princeton Studies in International Finance, Vol. 60 (August).
Johnson, Omotunde, and Joanne Salop, 1980, “Distributional Aspects of Stabilization Programs in Developing Countries,” Staff Papers, International Monetary Fund, Vol. 27 (March), pp. 1–23.
Khan, Mohsin S., 1990, “The Macroeconomic Effects of IMF-Supported Adjustment Programs,” Staff Papers, International Monetary Fund, Vol. 37 (June), pp. 195–231.
Khan, Mohsin S., and Malcolm D. Knight, 1981, “Stabilization Programs in Developing Countries: A Formal Framework,” Staff Papers, International Monetary Fund, Vol. 28 (March), pp. 1–53.
Khan, Mohsin S., and Malcolm D. Knight, 1985, IMF-Supported Programs and Economic Growth, IMF Occasional Paper 41 (Washington: International Monetary Fund).
Khan, Mohsin S., Peter J. Montiel, and Nadeem Ul Haque, eds., 1991, Macroeconomic Models far Adjustment in Developing Countries (Washington: International Monetary Fund).
Killick, Tony, 1984, The Quest for Economic Stabilization: The IMF and the Third World (New York: St. Martin’s).
Killick, Tony, 1995, IMF Programmes in Developing Countries (London: Routledge).
Killick, Tony, and M. Malik, 1995, “Country Experiences with IMF Programmes in the 1980s,” World Economy, Vol. 15 (September), pp. 599–632.
Killick, Tony, M. Malik, and M. Manuel, 1995, “What Can We Know About the Effects of IMF Programmes?,” World Economy, Vol. 15 (September), pp. 575–97.
Loxley, J., 1984, The IMF and the Poorest Countries (Ottawa: North-South Institute).
Pastor, M., 1987, “The Effects of IMF Programs in the Third World: Debate and Evidence from Latin America,” World Development, Vol. 15 (February), pp. 249–62.
Polak, Jacques J., 1991, “The Changing Nature of IMF Conditionality,” Princeton Essays in International Finance, Vol. 184.
Reichmann, Thomas M., and Richard T. Stillson, 1978, “Experience with Programs of Balance of Payments Adjustment: Stand-By Arrangements in the Higher Credit Tranches, 1963-72,” Staff Papers, International Monetary Fund, Vol. 25 (June), pp. 119–139.
Santaella, Julio, 1995, “Four Decades of IMF Arrangements: Macroeconomic Stylized Facts Before the Adjustment Programs,” IMF Working Paper 95/74 (Washington: International Monetary Fund).
Schadler, Susan, Franck Rozwadowski, Siddarth Tiwari, and David Robinson, 1993, Economic Adjustment in Low-Income Countries—Experience Under the Enhanced Structural Adjustment Facility, IMF Occasional Paper 106 (Washington: International Monetary Fund).
Sisson, Charles, 1986, “IMF-Supported Programs and Income Distribution in LDCs,” Finance and Development, Vol. 23 (March), pp. 33–36.
Williamson, John, ed., 1983, IMF Conditionality (Washington: Institute for International Economics).
Zulu, Justin B., and Saleh M. Nsouli, 1985, Adjustment Programs in Africa: The Recent Experience, IMF Occasional Paper 34 (Washington: International Monetary Fund).
The authors are grateful to Tam Bayoumi, Patrick Conway, Stanley Fischer, Morris Cold-stein, and a number of IMF Institute colleagues for very helpful comments on the draft of this chapter.
Santaella (1995) has detailed the macroeconomic characteristics of countries prior to adopting an IMF-supported program. Typically, the countries are suffering from a worsening current account balance, a loss of international reserves, and an increase in inflation.
See Polak (1991) for a recent description of the conditionality associated with IMF-supported programs.
Other macroeconomic objectives could include the resumption or expansion of private capital inflows, exports, and investment.
Henceforth evaluations of programs will also be conducted by the IMF’s newly established Independent Evaluation Office.
For a simple theoretical derivation of the relevant counterfactual for assessing the effects of programs, see Conway (1998).
Another approach, referred to as the actual versus targets approach compares actual performance under the program with the objectives specified in the program. This approach, which does not involve the counterfactual, is not reviewed here. For examples of this type of analysis, see Reichmann and Stillson (1978) and Zulu and Nsouli (1985).
The pre- and postprogram periods can be of any length. As shown later, most studies using this approach look at the year before and the year after the program is initiated, but this is arbitrary.
For a demonstration of the bias, see Goldstein and Montiel (1986) and Conway (1998).
By making a judgmental correction for the influence of nonprogram factors it would be possible to improve the estimates of the counterfactual that would emerge from a mechanistic application of the BA approach. However, such judgmental corrections are difficult to make, especially when the number of nonprogram factors is large.
This procedure was developed by Goldstein and Montiel (1986), and later refined by Khan (1990) and Conway (1994).
By selecting the control group of nonprogram countries to include only those that had balance of payments problems in the preprogram period, Gylfason (1987) also attempted to adjust for the bias while applying the WW approach.
See, for example, the various models contained in Khan, Montiel, and Haque (1991).
Some of the analyses in Khan, Montiel, and Haque (1991) attempt to handle this problem by adjusting certain relevant parameters of the model.
Some studies have considered other variables, such as the degree of income inequality. See Johnson and Salop (1980), Sisson (1986), and Pastor (1987).
In contrast to other studies. Pastor (1987) focuses on nominal rather than real GDP.
The method has also been employed in evaluating World Bank structural adjustment programs. See Corbo and Fischer (1994) for a discussion of the merits of this approach, and Faini and others (1991) and Corbo and Rojas (1992) for empirical applications.
Khan (1990) also showed that the achievement of macroeconomic stability, defined in various ways, had a positive impact on medium-term growth.
For an example of the case study approach to program evaluation, see Gomulka (1995) and Goldsbrough and others (1996).
An interesting attempt to combine the merits of single- and multicountry studies has been undertaken by Killick (1995).