The Macroeconomic Effects of ESAF-Supported Programs
Revisiting Some Methodological Issues

This paper examines whether ESAF-supported programs during 1986-91 had significant independent effects on growth, inflation and the external debt service ratio. Econometric estimates of the Generalized Evaluation Estimator (GEE) identify statistically significant beneficial effects on output growth and the debt service ratio but no effects on inflation. The robustness of these estimates is also examined. Diagnostic tests cast doubt on the applicability of the GEE framework to the ESAF-eligible countries, and the results obtained using it.

Abstract

This paper examines whether ESAF-supported programs during 1986-91 had significant independent effects on growth, inflation and the external debt service ratio. Econometric estimates of the Generalized Evaluation Estimator (GEE) identify statistically significant beneficial effects on output growth and the debt service ratio but no effects on inflation. The robustness of these estimates is also examined. Diagnostic tests cast doubt on the applicability of the GEE framework to the ESAF-eligible countries, and the results obtained using it.

I. Introduction

The means of evaluating Fund-supported programs are the subject of perennial controversy, at the heart of which are two broad issues. First, what sorts of questions should be asked in such evaluations? In general, reviews have focussed on one or more of three broad, but distinct, questions: are Fund-supported programs designed effectively to address countries’ macroeconomic problems; are such programs adequately implemented; and do such programs have significant independent effects—that is, do they bring about developments significantly different from those that would have occurred in the absence of Fund support? A second critical issue for program evaluation is the methodology used to address the chosen questions. Clearly, the methodology needs to be tailored to the question being addressed, but the range of possibilities is large: at one end of the spectrum are individual case studies employing anything from simulation models to descriptive analyses; at the other end are aggregate exercises based on comparisons of data from before and after programs, of data from program countries and another control group, of actual data and a hypothetical counterfactual, and of target and actual policies and outcomes.

A recent staff study of programs supported through the Enhanced Structural Adjustment Facility (ESAF) has renewed interest in these issues. 2/ The study was focussed mainly on answering the first of the three questions identified above for motivating evaluations of Fund-supported programs: specifically, it examined whether the policies and outcomes during SAF/ESAF arrangements conformed to the objectives of the ESAF and why some countries were more successful than others in progressing toward the ESAF objective of external viability, satisfactory growth, and low inflation. It relied heavily on before-after comparisons of economic developments for the average of all countries, subgroups of the countries and individual countries. The study generated a lively debate (see Killick (forthcoming)) about other questions that might have been addressed. One such question, on which attention has been focussed in some other reviews of IMF- or World Bank-supported programs, is how much the presence of Fund or Bank support improved the outcomes of important macroeconomic target variables from what they would have been in the absence of such support.

The purpose of this paper is to re-examine the data from ESAF-supported programs approved during the first six years of the facility’s operation with the aim of identifying the independent effects of the programs on important macroeconomic variables—growth, inflation, and a key indicator of progress toward external viability (the external debt service ratio). 3/ In order to do this, it is necessary to construct a policy counterfactual (that is, policies that would have been implemented in the absence of Fund support) and develop a framework for quantitatively differentiating the effects of the counterfactual policies, exogenous developments (for example, terms of trade changes or weather), initial conditions, and the involvement of the Fund. A methodology incorporating these elements was developed by Goldstein and Montiel (1986) by adapting techniques from the literature on labor training evaluation. Applications of the technique, referred to as the General Evaluation Estimator (GEE), can be found in Greene (1989), Khan (1990), Corbo and Rojas (1992), and Faini, De Melo, Senhadjo and Stanton (1991); for the most part, these studies have adhered closely to the GEE as proposed in Goldstein and Montiel. In this study, the application of the GEE also broadly follows Goldstein and Montiel to facilitate comparisons of results with those of other studies and because data constraints thwarted most efforts to elaborate on the basic framework.

A second objective is to investigate the robustness of the estimates of the effects of Fund-supported programs using the GEE for the sample of ESAF-eligible countries and to experiment with some techniques for easing the restrictive assumptions in the GEE. The GEE, as it has been applied in virtually all earlier studies, requires that two basic conditions be met: first, that a single, relatively simple macroeconomic model reasonably captures the interaction between macroeconomic policies and outcomes for a large number of countries over time; and second, that the counterfactual to policies during periods when Fund-supported programs are in place can be characterized by a policy reaction function estimated for periods when and in countries where Fund support is not in place. The paper devotes considerable emphasis to testing these basic propositions. This effort is important because conceptually the GEE is superior to other methodologies that have been used for identifying the independent effects of Fund-supported programs. The scope for reliably quantifying the effects of Fund support, therefore, is contingent on the validity, for any sample, of the assumptions underlying the GEE.

II. Specification of the Model

1. The basic model

The GEE is geared toward answering the question “Did the involvement of the IMF through an ESAF arrangement significantly improve the outcomes for important macroeconomic variables relative to what they would have been in the absence of ESAF-support?”. To answer this question the macroeconomic outcomes or target variables in countries are described as a function of: (i) policies that would have been observed in the absence of a Fund-supported program; (ii) exogenous external factors; (iii) the existence or otherwise of a Fund-supported program; and (iv) unobservable random shocks:

yij=βoj+βjkxik+αjhwih+βjIMFdi+εij(1)

where yij is the jth target variable in country i, xik is a k-element vector of policy variables that would be observed in country i in the absence of Fund support, wih is an h-element vector of exogenous external variables for each country i, di is a dummy variable equal to one if a Fund program is in effect and zero otherwise, and εij is a zero mean, fixed variance, serially uncorrelated error. For the jth target variable, βjk and αih are kxl and hxl vectors, respectively, of fixed parameters. After postulating a rule for policies in the absence of a Fund-supported program (xik), the model is estimated using pooled cross-section and time-series data drawn from countries and periods in which Fund support was in place and those in which Fund support was absent. The aim is to get consistent estimates for βjIMF, the “independent effect” of Fund-supported programs on each target variable. If these are statistically significant at a reasonable confidence level, Fund-supported programs are found to have significant effects.

Policies adopted in the absence of a Fund-supported program (xik) are directly observable only for nonprogram periods, and thus a key element of the GEE is the construction of a counterfactual for policies during programs. In Goldstein and Montiel (1986) and subsequent empirical applications, this counterfactual is based upon a policy reaction function that links changes in policy instruments to the deviation of the observed lagged value for each target from its desired value, yijd. Specifically, the policy reaction function is described by:

Δxik=γkj[yijdyij(1)]+ηik(2)

where yij is a j-element vector of target variables, ηik is a zero mean, fixed variance, serially uncorrelated error term assumed to be uncorrelated with εij, and Δ is the first difference operator. 4/ The kxj parameter matrix γkj indicates the extent to which policy instruments are adjusted in response to disequilibria in the target variables.

Substituting equation (2) into equation (1) and subsuming yijd in the constant (that is, assuming yijd is invariant across countries and over time) this substitution gives: 5/

Δyij=βj0(βjkγkj+1)yij(1)+βjkxik(1)(3)+αjhwih+βjIMFdi+(εij+βjkηik)

Equation (3) constitutes the basic GEE reduced form model as applied in most earlier studies. While conceptually rigorous, its operational usefulness depends on the validity of several rather restrictive assumptions. The remainder of this section considers some of these issues and suggests possible ways of addressing them. 6/

2. The counterfactual—which policy reaction function?

The choice of a policy reaction function is a critical step in applying the GEE, and the theoretical and empirical literature offers a profusion of possibilities. The policy reaction function can be set explicitly in an optimizing framework, but this does little, if anything, to narrow the choice; different specifications of policymakers’ objective functions and of the constraints they face generate a wide range of alternative reaction functions (see Turnovsky (1977)). This paper starts with the specification of the policy reaction function used by Goldstein and Montiel (1986), Greene (1989) and Khan (1990) (equation (2)) and experiments with two alternative less restrictive specifications, which are described in this sub-section.

According to equation (2), policies are adjusted only in response to target disequilibria in the previous period. However, it is plausible that policy adjustments in the current period also take account of the impact of external exogenous factors not captured in the lagged value of the target variable. Changes in the terms of trade or in primary commodity prices may have lagged effects; for example, fiscal revenues this year may be weakened by a drop in export prices last year. Assuming policymakers adjust policies with information only on past exogenous shocks (that is, policies are set at the beginning of the year), an enhanced policy reaction function with exogenous external factors (wih) lagged one period gives:

Δxik=γkj[yijdyij(1)]+μihwih(1)+ηik(4)

When equation (4) is substituted into equation (1) to eliminate the unobservable values of xik (and subsuming yijd in the constant) the specification of the GEE becomes:

Δyij=βj0(βjkγkj+1)yij(1)+βjkxik(1)+αjhwih+βjkμihwih(1)+βjIMFdi+(εij+βjkηik)(5)

A second variation of equation (2) results from setting the formulation of policies explicitly in an optimizing framework. One simple approach is to assume that policymakers in each country i choose policies (xik) to minimize a quadratic loss function (L) of the form

L=(yijyijd)2(6)

Subjecting this to the constraint of the economic model postulated in the GEE framework,

yij=β0j+βjkxik+αjhwih+εij(7)

the policy reaction function takes the form in levels

xik=γ^0+γ^khwih1/(8)

with yijd (assumed to be fixed) and other constant terms subsumed in γ^o. After substitution for xik in equation (1), the reduced form GEE becomes

Δyij=β^j0yij(1)+(βjkγ^kh+αjh)wih+βjIMFdi+εij(9)

In this equation, terms in xik drop out and the interpretation of the estimated coefficients on the lagged target variable and the exogenous external factors differs from that in equation (3). 7/

In practice, in an exercise where the goal is to evaluate the effectiveness of Fund-supported programs, an appeal to optimizing principles to motivate the policy reaction function may be thought to create a potential conundrum for the counterfactual approach. If countries are assumed to adopt optimal policies in the absence of a Fund-supported program, why would a country ever turn to the Fund for support? Following this line of argument, the influence of a program would depend on the Fund’s seal of approval strengthening confidence in the economy (and catalyzing external assistance) and thereby enhancing the effectiveness of any given stance of policy. Yet, if policies are the same with or without a program, this presumes that domestic and foreign economic agents fail to recognize the equivalence of policies under the two regimes. In reality the Fund’s seal of approval expands the opportunity set of policies beyond that available to a country not receiving Fund support. In fact, almost all of the countries that had ESAF arrangements were also seeking agreements with creditors to reschedule or restructure external debt, for which entering an arrangement from the Fund was a prerequisite.

Even with careful specification of the policy reaction function, there remains a question of whether individual country behavior can be sensibly aggregated in a uniform model that is stable across countries and over time. Specifically, differing institutional characteristics (for example, the degree of policy discipline inherent in specific exchange rate arrangements or the relationship with a major donor), changing political conditions, or varying severities of economic distress are likely to result in countries formulating policies with respect to different or changing objective functions or subject to different or changing constraints. Another question is whether it is appropriate to assume that the policy reaction function of a program country had it not received Fund support is identical to that of a nonprogram country that did not seek Fund support. For example, the counterfactual for a country receiving Fund support may involve the imposition of exchange restrictions, while countries that do not seek Fund support may constrain themselves to “Fund-type” policies—that is, avoiding the use of exchange restrictions.

3. Characterizing the independent effect of Fund-Supported programs

The simple GEE framework captures the independent effect of Fund support in an additive term (βjIMFdi) constant across countries and over time. This term is meant to capture four channels through which a Fund-supported program could affect the macroeconomic targets: (i) changes in the state of confidence in the economy; (ii) changes in the desired value of targets, for example through structural reforms aimed at raising the rate of potential growth; (iii) policies different from what they would have been in the absence of a program; and (iv) changes in the effectiveness of any given stance of policies.

Ideally, the specification of the effect of Fund support would not be restricted to an additive term constant across countries and time but would decompose program effects in a way allowing their magnitude and type to vary across programs. In doing so, it is reasonable to assume that the first two channels described above—changes in confidence and in desired values of targeted variables—could be captured in additive terms. Even here, however, it would be optimal to allow the coefficient βjIMF to vary across program years (that is by allowing program-specific coefficients).

To capture the third channel of program influences—policies different from what they would have been in the absence of Fund-support—modifications to the model are needed. 8/ The policies supported by the Fund differ for each country, reflecting countries’ preferences on how to address their problems, the nature of the macroeconomic disequilibria, and the severity of the external financing constraint. Thus, for given relationships between policies and targets, the effect of Fund-supported programs on macroeconomic targets should differ across countries and over time. This variation could be captured with multiplicative dummies to relate the estimate of program effects directly to the difference between policies under the program and the counterfactual. The effect of a Fund-supported program would be

βijkIMF=βjk(xikPxik)+AjIMF(11)

where xikP is the vector of (observed) policies adopted under a program, and AjIMF is a constant. The first term in (11) captures the effect of changes in policy instruments and AjIMF all other program effects.

The fourth type of program effect—changes in policy effectiveness—operates by altering the βjk parameters, for given changes in policy variables. For example, changes in fiscal and credit policy under a Fund-supported program may have a greater impact than they would otherwise have had owing to credibility effects when policies and targets are set in a preannounced medium-term framework. In these circumstances, estimates of equation (3) that assume constancy of the βjk coefficients are open to the “Lucas critique” (Lucas (1976)), and may not measure the true value of βjIMF. Ideally, the variation of the βjk parameters would be endogenized by modelling the link between public policies and private expectations. Alternatively, the βjk coefficients could be allowed to vary across countries and over time, or the variation in the βjk coefficients could be restricted to systematic differences between program and nonprogram periods. Under the latter specification, the effect of Fund support is:

βijIMF=βjkPxikPβjkxik+AjIMF(12)

or

βijIMF=βjk(xikPxik)+(βjkPβjk)xikP+AjIMF(12a)

of which expression (11) is a special case and βikP is a vector of parameters linking policies and targets during programs. The second term in equation (12a) captures the effect of changes in policy effectiveness.

Two steps are needed to get estimates of βjIMF as defined in equation (12) or (12a). First, the policy reaction function must be estimated to get values of xik for program periods (the unobservable counterfactual policies). Second, using the estimates of xik and substituting for βjIMF in equation (1), the following equation must be estimated.

yij=βoj+βjkxik(1di)+βjkPxikPdi(13)+αjhwih+AjIMFdi+εij

In sum, plausible characterizations of the influence of programs on targets suggest that a simple invariant additive term in the GEE may not do full justice to the range of potential program effects. In practice, the possibility of a more informative decomposition of program effects that allows variation across countries requires a large sample and the ability to identify empirically a stable policy reaction function as examined in Section IV.

4. Dynamics and the importance of initial conditions

Dynamic characteristics of the economy are likely to influence the effects of Fund-supported programs in two main ways that are captured to varying degrees in the simple GEE model. First, the full effects of changes in policies—those supported by the Fund or the counterfactual—and exogenous influences on target variables, particularly output growth, may occur with long lags. In the simplest form of the GEE presented in this paper, such dynamic effects are not allowed for: effects from changes in policies and exogenous factors are assumed to occur within one year. Some applications (Khan (1990)) have eased this restriction by estimating the model with two year average (rather than single year) data for the target variables. This procedure goes some way toward accounting for short lags, but at the cost of not accounting for contemporaneous effects of policies in the second year, which may or may not be a program year.

A second role for dynamic influences works through initial conditions in two ways. First, departures from the desired values of the macroeconomic targets in the preprogram period may generate a policy response even in the absence of a program. In the GEE model, this influence is captured by the term βjkγkj in the coefficients on the lagged target variables in equation (3). If the likelihood of adopting a program is positively correlated with negative prior shocks, then a failure to account for corrective policy, responses that would have taken place in the absence of Fund support would lead to an overstatement of the true effect of a Fund-supported program.

Second, current values of the target variables may be influenced by initial conditions through inertial effects. 9/ These are not captured in the static model postulated in equation (1), where the stochastic terms εij and ηik are assumed to be serially uncorrelated: all shocks are assumed to be transitory and to cause one-period changes in target variables that are fully reversed in the following period. This is a quite restrictive assumption. Not only does it require that the time series for each target variable be stationary, but it also rules out a wide range of stationary stochastic processes for which the impact of temporary shocks persists over time. 10/ If, in fact, significant inertia exists, then imposing full one-period reversion to mean will result in an understatement of the positive effects of a Fund-supported program after a negative shock.

In principle, the following more general dynamic form of equation (1) would allow a wider range of potential dynamic effects:

yij=ρs(L)1βoj+ρs(L)1βjk(L)xik+ρs(L)1αjh(L)wih+ρs(L)1βjIMFdi+ρs(L)1εij(14)

where L is the lag operator and ρs(L) is a lag polynomial in L of order s which captures inertia in the behavior of the target variables. The order of the lag operators may differ for each variable. Substituting equation (2) (the policy reaction function) into equation (14) gives:

Δyij=β^j0(ρs(L)1βjk(L)γkj+1)yij(1)+ρs(L)1βjk(L)xik(1)+ρs(L)1αjh(L)wih+ρs(L)1βjIMFdi+(ρs(L)1εij+ρs(L)1βjk(L)ηik)(15)

which is a general dynamic form of the reduced form GEE (equation (3)).

III. Estimation Procedures

1. The sample

The model, as specified in equation (3) with some of the modifications suggested in Section II, is estimated for the period 1986-91 with data for 61 of the 66 countries (Appendix Table 1) eligible to use ESAF resources as of 1992 (exclusions are noted below). Nineteen of these countries had ESAF arrangements at some time during the sample period. 11/ The sample was restricted to ESAF-eligible countries, rather than a larger set of developing countries, for two reasons: first, the many years when stand-by and extended arrangements were in effect in non-ESAF-eligible countries could not be characterized as nonprogram years, yet these arrangements were in many respects not comparable to ESAF arrangements; second, including only low-income countries reduced the scope for parameter instability owing to differences in structural and institutional conditions between low-income countries and other developing countries.

Even the sample restricted to ESAF-eligible countries is quite diverse. The program countries are dominated by heavily indebted African countries with a narrow range of exports and relatively simple market structures. The nonprogram countries are more diverse including, in about equal proportions, indebted African countries and small Caribbean or Pacific island economies with close institutional and financial links to particular industrial countries. There are also a few South and Southeast Asian countries.

For estimation, a number of data points are dropped from the full sample of ESAF-eligible countries. Some observations are excluded because of inadequate data owing to civil strife (Afghanistan (1990-91), Angola, Liberia, and Nicaragua (all years)); major discontinuities which could not be corrected (Djibouti (all years)); extreme isolation (Albania (all years)); or political discontinuities (Democratic Republic of Yemen (1990-1991)) and Yemen Arab Republic (1989-1991)). Years in which countries had a SAF arrangement (except when followed immediately by an ESAF arrangement), Stand-by, or Extended arrangement are omitted because they were considered invalid as nonprogram counterfactuals; this, together with lack of data in 1991, exclude Somalia entirely from the sample.

The exclusion of data points renders the structure of the panel data of program and nonprogram years incomplete. Techniques for analyzing panel data in which missing observations are random or follow a regular (or “rotating”) pattern are not applicable because the gaps in the data do not conform to either of these patterns. Instead, for estimation purposes, the panel data are handled as a pooling of annual observations, with the number of program and nonprogram years varying from country to country.

For the purpose of investigating the stability and the dynamic specification of the GEE a larger set of observations for each country would have been preferable. Extending the sample, however, would have required dropping several countries owing to the lack of consistent data for earlier periods. Also, the number of additional useable observations (that is, years in which countries did not have a Stand-by or Extended arrangement in place) would have been a rather small proportion of the total.

Even for the limited sample covered, the quality of data is poor. In many instances, the accuracy of the measures of macroeconomic variables, such as GDP, is likely to very weak. Also, ad hoc correction for breaks in the series were frequently needed. These fundamental weaknesses in the data qualify the inferences or judgements that can be drawn from them.

2. Definitions: targets, policies, exogenous influences and period of Fund support

This study considers three target variables (yji) that reflect the objectives of the ESAF: (i) the growth rate of real GDP; (ii) the average rate of consumer price inflation; and (iii) a measure of external viability—the ratio of external debt service to exports. The last target is preferred to other external indicators, such as the current account, the overall balance of payments, or the level of international reserves, as a measure of external viability for several reasons. 12/ Most countries entered ESAF arrangements with large debt overhangs, and reducing the debt service ratio to manageable levels was the primary external objective. Within this goal, targeted and actual outcomes for other external variables varied widely depending on initial conditions and prospects for attracting concessional inflows. Moreover, for non-program periods, developments in these other variables sometimes reflected the imposition of formal or informal trade and exchange restrictions rather than changes in the viability of the external position. Increases in reserves were at times associated with the accumulation of arrears. Also, a reserves target did not exist for a sizable proportion of the countries (CFA and ECCB country members) that did not directly own international reserves

Three policy instruments (xik) are considered: (i) the deficit of the central government in relation to GDP; (ii) the growth of net domestic assets of the banking system (NDA) 13/; and (iii) the change in the nominal effective exchange rate (NEER). Ideally, the vector of policy instruments would also include indicators of structural reforms/conditions and institutional arrangements (such as flexible or fixed exchange rate regimes). However, these variables cannot be easily quantified or reduced to an index. The external environment indicators (wih) comprise changes in the terms of trade and the growth of export markets. 14/ Whenever possible, the data were taken from Executive Board documents.

Several difficulties arise in defining the variable denoting the presence of a Fund-supported program (di). First, the distinction between program and non-program years is blurred when Fund support starts in the middle of the year. For this study, any year in which a SAF/ESAF program was in effect for six months or more was considered a program year. In applying this rule, the program periods (periods over which a program was framed) rather than annual arrangement periods (periods over which programs were approved) were considered. Even this rule, however, does not clearly delineate the period during which Fund support influenced policies and outcomes. Usually, substantive negotiations and policy actions occurred in anticipation of Fund support in the year preceding the formal program. Also, in some cases, the Fund had a considerable influence on policies after an ESAF arrangement. For example, The Gambia’s SAF/ESAF arrangements stretched from FY1986 (July-June) to FY1990, but even in FY1991, the Fund monitored macroeconomic developments vis-a-vis quantified targets agreed with the authorities.

How to handle variations in program implementation across countries presents a second difficulty. Some would argue that implementation should be reflected in the estimated model, for example, by representing the influence of the Fund as the proportion of purchases made relative to total access under the arrangement. Purchases are likely to be an imperfect indicator of implementation, however, because purchases in SAF/ESAF programs are scheduled at low (six-month) frequencies and waivers may be granted to permit purchases even in the event of policy slippages. Alternatively, the effectiveness of Fund support could be judged in terms of outcomes, whether or not programmed policies are implemented: a period when a country has an agreed program with the Fund but fails to implement it or meet the targets is treated as a program year when the effect of the Fund is zero. This study takes the latter approach and uses a binary, one-zero index of Fund involvement for the dummy variable (di).

A third point to recognize is that estimates of Fund-supported programs will likely include the effects of parallel World Bank programs. As distinct from the Fund’s Stand-by and Extended Facilities, SAF/ESAF arrangements require explicit collaboration among country authorities, the World Bank and the Fund.

IV. Results

1. The generalized evaluation estimator

Estimation results were obtained for the basic GEE equation (3) and several of the modified versions suggested in Section II to ease the restrictiveness of the simple framework: (i) the expanded equation (5), which includes lagged exogenous external influences in the policy reaction function was estimated; 15/ (ii) country and time dummies were introduced to help account for some of the cross-country differences in economic structures and time-specific exogenous developments not captured in the terms of trade and market growth variables; and (iii) a correction procedure developed by Heckman (1979) (explained in detail in the Annex) was tried to correct for possible sample selection bias. Other methods discussed in Section II to ease the restrictive assumptions of the GEE were not feasible. Specifically, consideration of a richer characterization of the effects of Fund support as described in equations (11-13) was not possible because the estimates of the policy reaction function, which are needed to estimate counterfactual policies during program periods, proved to be very poor. Also, because of the short time series available, a more complex dynamic structure for the GEE could not be explored.

Regression estimates based on pooled time-series, cross-country data are prone to heteroschedasticity, and even after the inclusion of country-specific and time dummies in the estimated equations, regression residuals displayed heteroschedasticity. 16/ A weighted least squares correction for heteroschedasticity was not attempted because, without information on the form of heteroschedasticity, the primacy of one weighing scheme over another is unknown. 17/ Instead, the ordinary least squares coefficients were retained, and the reported t-statistics were computed from heteroschedasticconsistent estimates of the standard errors based on White’s variance-covariance estimator that provides consistent estimates even when the exact form of heteroschedasticity is not known. 18/

Table 1 presents the preferred estimates from these exercises. These estimates exclude the lagged values of the exogenous variables (the modification suggested in equation (5)) because their coefficients were not significantly different from zero and the terms had little effect on the fit of the regression or the estimates of the other coefficients. The results also exclude the time dummies mentioned above because their coefficients were generally not significant and had little effect on (or worsened) the fit of the equations. The impact of Fund-supported programs is found to be sizeable and statistically significant with respect to growth (at the 5 percent level) and the external debt service ratio (at the 10 percent level), but not inflation. On average, growth rates are found to be more than 1 percentage point per annum higher during program periods than they would have been in the absence of a Fund-supported program. Debt-service ratios are found on average to be more than 5 percentage points lower during program periods than they would have been in the absence of a Fund-supported program. This effect on the debt-service ratio is not attributable to the link between Paris and London Club agreements and Fund support, because the debt service ratio is measured before debt relief. 19/

Table 1.

Estimates of the GEE 1/

article image

The regression estimates were obtained using a ordinary least squares procedure, with country-specific dummies included in the specification. Standard errors and t-statistics of coefficients are computed using White’s heteroschedasticity-consistent variance-covariance estimator. The figures in parentheses are t-statistics; R¯2 is the adjusted coefficient of determination; S.E.E. is the standard error of the regression. A single asterisk indicates statistical significance at the 5 percent level; two asterisks indicate statistical significance at the 1 percent level.

On the other right hand-side variables very few coefficients are significant at the 5 or 1 percent confidence level. The exceptions are own lagged levels of the target variables, which enter with coefficients significantly different from zero at the 1 percent level; the lagged level of the real GDP growth rate and the change in the nominal effective exchange rate (which has an implausible positive sign) in the inflation equation; and the terms of trade in the debt service ratio equation. The lagged fiscal balance and lagged net domestic asset growth are not found to have a significant impact on the outcome of any target variable. 20/

In broad terms, these results are similar to those of Khan (1990)—the most recent and largest study using the GEE methodology to estimate the effects of programs supported by upper credit tranche Stand-by and Extended arrangements. That study uses a sample of 1,104 observations spanning 69 countries over the period 1973-88, including 315 program years. 21/ It is noteworthy that Khan’s sample is quite different from that used in this study: it excludes SAF/ESAF arrangements from the program sample, ends several years prior to the sample used in this paper and includes low- and middle-income developing countries. Like this study, Khan finds that lagged values of target variables significantly influence current changes of these variables. He also finds few significant effects of lagged policies on current target variables; the main difference in this respect is that Khan finds significant and plausible effects of the lagged fiscal balance on the target variables, while in this study the fiscal effect is found to be insignificant. Both studies find programs lead to reductions (though not statistically significant) in inflation. The most important difference between the two studies, however, is in the estimate of the effect of Fund support on growth: Khan finds a significantly negative effect on growth, in contrast to the significantly positive effect in the current study.

2. The policy reaction function

Identification of the coefficients and measurement of their standard errors in the policy reaction function from the reduced form GEE is neither possible nor necessary for obtaining estimates of the βjIMF terms. 22/ Nevertheless, obtaining estimates of them, by directly estimating the counterfactual policy reaction functions with data from the (observable) nonprogram periods only, is useful for two reasons. First, it provides a means of evaluating the validity of the reaction functions for the sample over which they can be estimated. Second, direct estimates of the policy reaction function are part of the procedure to correct for sample selection bias arising when unobservable factors that influence countries’ decisions to receive Fund support also influence their policy reactions. To identify whether this potential source of bias exists, and to correct for it, a two-step procedure first proposed by Heckman (1979) is used. 23/ First, a probit model of the probability of not having a program is estimated. Second, the probit estimates are used to calculate the inverse Mills ratio 24/ (for each nonprogram observation) which, when included in the policy reaction function estimated for the nonprogram observations, accounts for the bias due to nonrandom sampling. Annex I describes the procedure in detail and reports estimates of the probit model.

As with the GEE, country and time dummies were introduced in the policy reaction functions. Most of the time and country dummies had coefficients insignificantly different from zero and had little effect on or worsened the overall fit of the equation (reduced the R¯2) and therefore were not retained in the reported results.

The regression estimates of the policy reaction function are poor in several respects (Table 2): the R-squared statistics are negative or very close to zero; t-statistics for individual coefficients are insignificant (except on the debt service ratio in the equation for the nominal effective exchange rate); F-tests can not reject the null joint hypothesis of zero slopes; and the regression residuals exhibit statistically significant heteroschedasticity and nonnormality. The insignificant coefficients of the inverse Mills ratio suggest that sample selection bias is not present in the sample. However, in view of the poor performance of the estimated policy reaction function this result is not particularly revealing. In short, these estimates provide a weak basis for deriving estimates of the unobservable counterfactual policies for program periods. Thus, no attempt was made to estimate equation (13) to identify more complex characterizations of the independent effects of Fund-supported programs.

Table 2.

Estimates of the Policy Reaction Function 1/

article image

The regression estimates were obtained from the sample of nonprogram observations using an ordinary least squares procedure. Standard errors and t-statistics of coefficients are computed using White’s heteroschedasticity-consistent variance-covariance estimator. The figures in parentheses are t-statistics; is the adjusted coefficient of determination; S.E.E. is the standard error of the regression. A single asterisk indicates statistical significance of the 5 percent level; two asterisks indicate statistical significance at the 1 percent level.

Values of the inverse Mills ratio were computed using the estimated probit equation reported in Annex Table 1.

3. Significance and stability of the estimates

How much confidence can be placed on the estimates of the effects of programs? The applications of the GEE to date have tended to focus on the size and significance of the estimates of the βjIMF parameters, without testing the validity of the basic assumptions in the GEE or, in some cases, even without reporting indicators of the overall goodness of fit. In this section, some simple experiments to determine the robustness of the parameters estimates are reported.

There are many ways to evaluate the regression estimates, and the approach taken here is not intended to be exhaustive. As measured by the R-squared statistic, the overall fit of the estimated equations is modest (almost nil for the external debt service ratio). This, together with the evidence of heteroschedastic residuals (even after the inclusion of country and time dummies) and the large number of coefficients insignificantly different from zero or with counterintuitive signs, suggests the possibility of misspecification and, therefore, of biased coefficient estimates. One potentially important omission is structural conditions/reforms that figured prominently in SAF/ESAF-supported programs. A second concern is the possibility of heterogeneity bias (Hsiao (1986)). 25/ This arises when parameter estimates are obtained by imposing identical coefficients on pooled data but the true values of the parameters differ significantly across countries and/or over time, owing for example to differences in policy regimes. In these circumstances, pooled estimates that ignore parameter heterogeneities are likely to be biassed. A third concern is that the regression residuals fail to pass the Jarque-Bera test for normality, signalling the risk of invalid inferential statements, even in large samples (Jarque and Bera (1987)).

The reliability of the parameter estimates is also revealed by the stability properties of the model. Earlier applications of the GEE have not reported tests of stability. 26/ Most investigate changes in program effectiveness as measured by the βjIMF coefficients (or the equivalent for World Bank programs) between two sub-periods of the sample used; Greene (1989) and Corbo and Rojas (1992) also report changes in estimates of the other coefficients of the reduced form GEE. Yet, in each study, the evidence of changes in point estimates of the βjIMF and other coefficients (at times statistically significant) is not seen as a possible indication of instability in the underlying model.

The most appealing approach to exploring instability would be to estimate a varying parameters model in which estimated parameters are allowed to vary across countries 27/ and over time 28/ and therefore also between program and nonprogram periods. This general approach has the merit of permitting tests of stability and uniformity restrictions nested within a less restrictive framework. 29/ However, such an approach requires a data set considerably larger than that available in this study: to explore inter-country instability, the number of data points for each country must exceed the number of regressors; and to explore instability between program and nonprogram periods, the number of data points for each country in each regime (program and nonprogram) must exceed the number of regressors.

In light of the constraints imposed by the structure of our sample (that is, the small and unequal number of annual observations for each country), the stability of the individual parameters βjk and βjIMF is examined by recursive regression methods. 30/ Two types of recursive exercises were done: in the first, the reduced form GEE was estimated recursively, starting with a baseline sample (comprising all nonprogram observations plus one program observation) and then adding in program observations one at a time (reported as “program recursions”); 31/ in the second, the recursive procedure was carried out by starting with the full sample and then subtracting nonprogram observations one-by-one (reported as “nonprogram recursions”). The share of estimates in the recursions that differs significantly from the pooled sample estimates provides an indication of the sensitivity of the estimated parameters to variations in the sample. Also of interest, particularly for parameters where whole sample estimates are significantly different from zero, is the share of recursion estimates that differ significantly from zero. 32/

The recursion estimates of the effectiveness of IMF support—the βjIMF coefficients—show that the point estimates and statistical significance are sensitive to changes in the sample (Charts 1-3). Of the six recursive exercises (two for each of the three equations), four produce estimates of βjIMF that are significantly different from the whole sample estimates in at least 15 percent of the recursions. Significant deviations are more frequent in the program than in the non-program recursions. In all the recursive exercises, a sizeable share (64-100 percent) of recursion estimates are not significantly different from zero. Thus, the finding of significant effects of Fund support on growth and the debt service ratio cannot be considered robust to variations in the composition of the sample.

Chart 1:
Chart 1:

Recursive Point Estimates and Confidence Intervals for IMF Program Dummy Variable

(Real GDP growth rates)

Citation: IMF Working Papers 1995, 092; 10.5089/9781451951486.001.A001

Chart 2:
Chart 2:

Recursive Point Estimates and Confidence Intervals for IMF Program Dummy Variable

(Rate of inflation)

Citation: IMF Working Papers 1995, 092; 10.5089/9781451951486.001.A001

Chart 3:
Chart 3:

Recursive Point Estimates and Confidence Intervals for IMF Program Dummy Variable

(External debt service ratio)

Citation: IMF Working Papers 1995, 092; 10.5089/9781451951486.001.A001

For the coefficients on the lagged policy variables, the recursive exercises suggest that the full sample estimates of policy effects are relatively robust (Tables 3 and 4). Typically, small proportions (in 13 of the 18 exercises, none) of the recursions produce estimates significantly different from the full sample estimates. As almost all of the full sample estimates are insignificantly different from zero, most (11) of the 18 recursions produce estimates that are also not significantly different from zero. There are a few exceptions to these generalizations, although none suggest the basis for questioning the general pattern of lagged policy variables having little effect on target variables.

Table 3.

Share of Statistically Significant t-statistics at the 5 Percent Level in Recursive Estimates of the GEE 1/

article image

Excluding full sample estimates, the total number of recursive estimates was equal to 87 for program year observations, and to 202 for nonprogram year observations.

Recursive procedure starts with the baseline sample (all nonprogram observations plus one program observation) and adds program observations one-by-one.

Recursive procedure starts with the entire sample and subtracts out one-by-one non-program observations.

Table 4:

Generalized Evaluation Estimates of Policy Parameters (β) 1/

article image

Standard errors and t-statistics computing White’s heteroschedasticity-consistent variance-covariance estimator. A single asterisk indicates statistical significance at the 5 percent level; two asterisks indicate statistical significance at the 1 percent level.

In order to investigate the stability of the parameters in the policy reaction functions (γkj), a simplified recursive exercise over the sample of nonprogram observations was performed. The recursions began with an initial sample of 25 observations, and observations were added one-by-one. 33/ The results, reported in Tables 5 and 6, indicate that the coefficient estimates from the full sample of non-program observations are generally robust across recursions, and (except for the lagged external debt service ratio in the equation the nominal effective exchange rate policy) are insignificantly different from zero. Although the estimates of the coefficient on inflation in the exchange rate equation reveal a significant reversal of sign across recursions, sign reversals are not widespread; most of the estimates are neither significantly different from zero nor from the full sample estimate.

Table 5:

Sensitivity of Estimates of Policy Reaction Function Parameters (γ) 1/

article image

Standard errors and t-statistics computing White’s heteroschedasticity-consistent variance-covariance estimator. A single asterisk indicates statistical significance at the 5 percent level; two asterisks indicate statistical significance at the 1 percent level.

Recursive least squares estimates obtained by adding observations one-by-one to an initial sample of 25 nonprogram years (corresponding to a minimum of 20 degrees freedom).

Table 6:

Share of Statistically Significant t-statistics at the 5 Percent Level in Recursive Estimates of the Policy Reaction Function 1/

(In Percent)

article image

Excluding full nonprogram sample estimates, the total number of recursive estimates was equal to 178 for non-program years observations. Shares of statistically significant t-statistics at the 10 percent level are reported in parentheses.

4. Dynamics and initial conditions

The estimation results can shed some light on the questions raised in Section II.4 about on the adequacy of the dynamic specification of the GEE. Specifically, independent estimates of γkj from the policy reaction function for nonprogram periods together with the estimates of the βjk from the reduced form GEE can be used to provide an indication of the appropriateness of the static specification of the GEE—that is, that all shocks to target variables are fully reversed in one period. The product of the estimates of these two parameters for each equation is close to zero. In this case, the coefficient on the own lagged target variable in the reduced form equation (-(ßjkγkj+1) from equation 3) would equal minus one if the assumption of complete reversion to mean were valid. Except in the growth equation, this hypothesis is rejected by the data. 34/ Nevertheless, the reduced form estimates of the coefficients on the lagged values of the target variables are significantly different from zero, suggesting that there is partial reversion to mean in the target variables: initial conditions do influence subsequent macroeconomic performances, but a richer specification of the dynamics is warranted for the sample under study.

V. Conclusion

With respect to the central objectives of this paper—to use the GEE framework to identify the independent effects of ESAF support during 1986-91 on key macroeconomic variables and to assess whether the assumptions underlying the GEE are applicable to the ESAF-eligible countries-conclusions can be summarized as follows. For output growth and the debt-service ratio, sizable beneficial effects that are statistically significantly different from zero are identified. 35/ The effects on inflation are not significantly different from zero. Diagnostic tests, however, cast doubt on the applicability of the GEE framework to the ESAF-eligible countries: the overall fit of the model is poor; estimates of the coefficients on many variables are insignificantly different from zero; regression residuals are heteroschedastic and nonnormally distributed; and the estimates of the coefficient on the dummy for ESAF support are quite sensitive to variations in the sample. A striking finding is that the counterfactual policy reaction function does not have any significant explanatory power for the sample of nonprogram observations. These results suggest that, for the sample reviewed, the GEE model is not correctly specified, parameter estimates may be biased, and conventional tests of significance may be misleading.

The results also have implications for any effort to identify the independent effects of Fund financial support. The GEE is a conceptually rigorous framework, superior to before-after and simple control group comparisons for identifying the independent effects of Fund financial support. It is, however, based on many restrictive assumptions that are necessary to define the counterfactual and to specify in a simple framework the main determinants of important endogenous macroeconomic variables. An important shortcoming of most recent applications of the GEE is their focus on the bottom line—the estimates of the effects of Fund support—with little or no evaluation of the validity of the underlying model; indeed, some studies have reported only estimated coefficients on the dummy variables and their standard errors, without diagnostic statistics or the estimates for other coefficients. One important lesson to be drawn from this study is that the validity for any given sample of the premises of the GEE methodology must be investigated before reliable conclusions about the independent effects of Fund-supported programs can be drawn from it.

This paper also points to some elaborations of the basic GEE framework that would likely help reduce its inherent restrictiveness and perhaps make it applicable in a broader range of circumstances. While this paper experimented with a few of these, such experimentation was limited by basic constraints. It is worth pointing out some of these here and indicating areas where substantially more experimentation will be needed to produce dependable estimates of the independent effects of Fund support.

  • Identifying policy reaction functions and a structural economic model that are both simply specified and common to what invariably is a wide range of developing countries is difficult. Future applications of the GEE may need to move toward applications to single countries or a small group of relatively homogeneous countries.

  • The stark distinction between periods when Fund-supported programs are in place and when they are not may not be conducive to capturing the independent effect of such programs. The Fund’s influence is exerted throughout the course of negotiations, prior actions, the program itself, program extensions, and post-program monitoring. From a broader perspective, there is a question whether and how the Fund’s involvement through lending arrangements should be differentiated from its influence through annual Article IV consultations (when the Fund staff provide advice on the spectrum of policies covered by ESAF arrangements).

  • Many key influences on macroeconomic performance, particularly structural and institutional changes, are inherently difficult to quantify and include in the GEE framework. For ESAF-supported programs, and increasingly even for programs supported through other windows, such changes—in quantitative exchange and trade restrictions, agricultural marketing arrangements, and financial institutions, for example—are a central focus and need to be accounted for explicitly.

  • The results reported in this paper suggest the presence of inertial effects in the target variables examined. In its commonly-applied form, however, the GEE assumes full and immediate reversion to mean of target variables. Also, this simple form investigates only the within year effects of Fund support on targeted macroeconomic variables. In general, however, particularly when structural and institutional changes are central to programs, effects are expected to be spread out over a far longer period. In order to enrich the dynamic specification of the GEE, longer time series than have been used to date are needed. This is a particularly onerous requirement for the ESAF-eligible countries, where the quality of the data is poor and consistent series without important breaks in definitions are not available even over the short period examined in this paper (1986-1991).

ANNEX: Sample Selection Bias and a Model of Program Participation

Sample selection bias in the parameter estimates of the policy reaction function (equation (2)) fitted over nonprogram observations can arise when unobservable factors that make a country more likely to seek Fund assistance also make a country more likely to have adopted a different policy package in the absence of a program than another country facing similar circumstances. To correct for this sample selection bias, following Heckman (1979) the estimated policy reaction function is augmented by an additional regressor which accounts for the bias due to nonrandom sampling.

The correction requires the estimation of a model of the probability of program status. Program status is characterized by a dummy variable di that equals one if the country has a Fund-supported program, a zero otherwise,

Ii = δ/D + πi

di = 0 if Ii>I*

di = 0 if Ii≤I*

where I is a random variable, which is an index of country-specific characteristics that determines the probability of country i not having a program; and D is an n-element vector of variables that determines program status; δ is a nx1 vector of parameters; πi is a zero mean fixed variance error term. The bias in the policy reaction function stems from a correlation between πi and the error term (ηik) in the policy equation (2). 36/ Estimates of the probability of nonprogram status were obtained using the probit model (fitted over the full sample):

Prob (di = 1) = φ (-δ/D)

Prob (di = 0) = 1 - Φ (-δ/D)

where φ and Φ denote the destiny and distribution function for a standard normal variable, respectively.

The estimated parameters of the probit equation are used to compute the inverse Mills ratio, i.e., the ratio φ(-δ/D)/[1-Φ(-δ/D)] 37/, which is included in the policy reaction function fitted over nonprogram countries to obtain unbiased estimates of the policy response coefficients. Assuming that ηik and πi have a bivariate normal density, the inverse Mills ratio takes into account the separate effect of not having a program on policy responses. Fitted values of the estimated policy reaction function provide the counterfactual set of policies (xik) for program countries, which can be used in the estimation of the GEE (equation (1)) to obtain an unbiased estimate of βjIMF. 38/

Several explanatory variables were considered for the probit model of program status: the ratio of the overall balance of payments to exports 39/; the external debt service ratio; the ratio of the flow of external payments arrears to exports 40/; the real GDP growth rate; the inflation rate; the percentage change in the terms of trade; the growth rate in export markets; a dummy variable with a value of 1 for persistent arrears with the Fund (which would preclude a program), 0 otherwise; and a dummy variable with a value of 1 if a country had previously had a Fund-supported program, 0 otherwise—countries familiar with Fund program operations may be more likely to adopt a program. The explanatory power of these regressors was weak, and at times was associated with unexpected or counterintuitive signs. In part, the difficulty of explaining the nonprogram/program status of a country reflects the fact that the program periods under review are several years in duration: there can be substantial changes (improvements) in the macroeconomic variables that prompted countries to implement a Fund-supported program during the program period. Also, while “economic need” may turn a country towards adopting a Fund-supported program, variables such as external arrears may not explain the precise timing of the decision to start a program.

The best fit probit regression that was used to correct for sample selection bias in the policy reaction function is reported in Annex Table 1. It should be noted that the residuals of the probit model exhibited evidence of nonnormality, in which case the equation estimates and the calculated inverse Mills ratio are likely to be inconsistent (see Greene (1993)). 41/ Therefore, the correction for sample selection bias, which relies on the consistency of the inverse Mills ratio, should be interpreted with caution.

Table 1:

Probit Model of Nonprogram Status 1/

article image

The figures in parentheses are t-statistics.

The pseudo-R2 measure is equal to 1-(logLUR/LogLR), where LogLUR is the maximum of the likelihood function when maximized with respect to all parameters and LogLR is the maximum when maximized with respect to the constant term only.

A predicted probability greater than or equal to 0.50 is associated with nonprogram status.

APPENDIX

Table 1.

Sample of Countries and Program/Nonprogram Years 1/

article image

Fiscal years t/t + 1 were considered to correspond to calendar year t if the fiscal year started before or on July 1st.

Includes observations in which a SAF, ESAF, SBA or EFF arrangement was not in place.

References

  • Breusch, T. and A. Pagan,A Simple Test for Heteroschedasticity and Random Coefficient Variation,Econometrica, Vol. 47 (1979) pp. 128794.

    • Search Google Scholar
    • Export Citation
  • Corbo, Vittorio and Patricia Rojas,World Bank Supported Adjustment Programs: Country Performance and Effectiveness”, in Adjustment Lending Revisited. Policies to Restore Growth, eds. Corbo, Vittorio, Stanley Fischer, and Steven B. Webb (Washington: The World Bank, 1992).

    • Search Google Scholar
    • Export Citation
  • Faini, Riccardo, Jaime de Melo, Abdelhak Senhadji and Julie Stanton,Growth-Oriented Adjustment Programs: A Statistical Analysis”, World Development, Vol. 19 (1991), No. 8, pp. 95767.

    • Search Google Scholar
    • Export Citation
  • Goldstein, Morris and Peter J. Montiel,Evaluating Fund Stabilization Programs with Multicountry Data: Some Methodological Pitfalls,Staff Papers, International Monetary Fund (Washington), Vol. 33 (June 1986) pp. 30444.

    • Search Google Scholar
    • Export Citation
  • Greene, Joshua,The Effects of Fund-Supported Adjustment Programs in African Countries, 1973-86,IMF Working Paper No. WP/89/38 (Washington: International Monetary Fund, 1989).

    • Search Google Scholar
    • Export Citation
  • Greene, William H. Econometric Analysis (2nd Edition), New York: Macmillan, 1993

  • Harvey, Andrew C., Time Series Models, New York: John Wiley, 1981.

  • Heckman, James,Sample Selection Bias as a Specification Error,Econometrica, Vol. 47, No. 1 (January 1979) pp. 15361.

  • Hsiao, Cheng, Analysis of Panel Data, New York: Cambridge University Press, 1986.

  • Jarque, Carlos, M. and Anil K. Bera,A Test for Normality of Observations and Regression Residuals,International Statistical Review, Vol. 55, No. 2 (1987), pp. 16372.

    • Search Google Scholar
    • Export Citation
  • Khan, Mohsin S.,The Macroeconomic Effects of Fund-Supported Adjustment Programs,”, Staff Papers, International Monetary Fund (Washington) Vol. 37 (June 1990) pp. 195231.

    • Search Google Scholar
    • Export Citation
  • Killick, Tony,Can the IMF Help Low-Income Countries?”, The World Economy, forthcoming.

  • Lucas, Robert E.,Econometric Policy Evaluation: A Critique,” pp. 1946 in K. Brunner and A. H. Metzer (eds.), The Phillips Curve and Labor Markets, Amsterdam: North Holland, 1976.

    • Search Google Scholar
    • Export Citation
  • Maddala, G., Introduction to Econometrics, 2nd Edition, New York: MacMillan, 1992.

  • Priestley, M.B., Spectral Analysis and Time Series, London: Academic Press, 1981.

  • Schadler, Susan, Franek Rozwadowski, Siddharth Tiwari and David O. Robinson, Economic Adjustment in Low-Income Countries - Experience Under the Enhanced Structural Adjustment Facility, IMF Occasional Paper No. 108 (Washington: International Monetary Fund, 1993).

    • Search Google Scholar
    • Export Citation
  • Swamy, P.A.V.B.,Efficient Inference in a Random Coefficient Regression Model”, Econometrica, Vol. 38, No. 2 (March 1970), pp. 31123.

    • Search Google Scholar
    • Export Citation
  • Turnovsky, Stephen J., Macroeconomic Analysis and Stabilization Policy, London: Cambridge University Press, 1977.

  • White, HalbertA Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity,Econometrica, Vol. 48, (1980), pp. 81738.

    • Search Google Scholar
    • Export Citation
  • World Bank, Adjustment Lending Policies for Sustainable Growth, Policy and Research Series, Paper 14, (Washington, 1990).

1/

The authors are indebted to Sharmini Coorey for her considerable input during the early stages of the project and to Kirsten Fitchett for her invaluable research assistance. We also thank Adam Bennett, David Burton, Benedicte Vibe Christensen, David Goldsbrough, Morris Goldstein, Mohsin Khan, Peter Montiel, Roger Nord, and Alessandro Zanello for their comments on an earlier draft of this paper. The views expressed in this paper are those of the authors and do not necessarily represent those of the International Monetary Fund.

2/

The study (Schadler et al. (1993)) reviews the experience of the 19 countries that had ESAF arrangements approved before mid-1992. The ESAF, described fully in the paper, is the Fund’s lending facility for supporting, at concessional interest rates, low-income countries undertaking three-year programs of macroeconomic adjustment and structural reform.

3/

As such, the exercise reported in this paper, while complementary to that in Schadler et al., is completely separate from it. The two exercises address quite different questions and their results do not provide comparable assessments of Fund-supported programs.

4/

Implicitly, the lack of mutual correlation between ηik and εij implies that changes in policy instruments (Δxik) are not influenced by contemporaneous exogenous shocks to the target variables.

5/

By assumption the stochastic structure is one in which shocks are transitory, thereby ruling out a source of bias in equation (3) because of nonzero correlation between the error terms and explanatory variables.

6/

Some of these restrictive assumptions characterize other, especially cross-section, estimators.

1/

This form of the policy reaction function is derived by assuming that E(εij) = 0, and that the covariances between the parameters βjk and εij, and between αjh and εij, are equal to zero in the analytical derivation.

7/

To derive equation (2) from an optimizing framework, equation (6) must be maximized subject not to the economic model postulated in the GEE (i.e., equation (7)), but rather to the constraint

Δyij=bjkΔxik+uij

where the parameters bjk are assumed to be stochastic and uij is a random disturbance. Also necessary are the simplifying assumptions that E(uij) = 0 and that the covariance between bjk and uij is set equal to zero in the analytical derivation.

8/

Much of this discussion draws upon Goldstein and Montiel (1986).

9/

Inertial effects may arise for a variety of reasons, such as backward-looking indexation, slowly-adjusting expectations, staggered contracts, and transaction costs.

10/

In the absence of stationarity the concept of reversion to mean is not well defined because the mean of the stochastic process is not time-invariant, and the series will tend to move continuously away from a given level as a result of the endless impact of past and current shocks. See Harvey (1981) and Priestley (1981).

11/

For these countries, program years are those when either a SAF or ESAF arrangement was in place. SAF arrangements typically have less stringent conditionality than ESAF arrangements. For most countries that had ESAF arrangements, however, prior SAF arrangements were used to establish commitment to adjustment and were close in nature to ESAF programs.

12/

The debt-service ratio is not, however, an infallible indicator of progress toward external viability. For example, changes in this ratio may at times only reflect a temporary variation in export prices.

13/

In principle, the controllable monetary policy instrument of the authorities is net domestic assets (or credit) of the central bank. However, data on a comparable basis across countries were not available.

14/

In principle, only one of market price and volume indicators should be used. In practice, the suitability of each varies among the countries: the terms of trade are relevant for small primary producers, but world market growth is relevant for large primary producers and countries with differentiated exports.

15/

The policy reaction function resulting from the optimization subject to the constraint implied by equation (1) was not tried because it represented a substantial simplification of even equation (3).

16/

Statistically significant values of the Breusch-Pagan test for heteroschedasticity (Breusch and Pagan (1980)) were observed in the estimated GEE equation for inflation (at the 5 percent significance level) and the external debt service ratio (at the 1 percent level).

17/

With only one nonprogram observation for several countries, there are insufficient degrees of freedom to use the common weighted least squares procedure in which observations for each country are weighted by the inverse of the standard deviation of the corresponding estimated residuals.

18/

See White (1990). The properties of White’s heteroschedasticconsistent variance-covariance estimator are conditional upon a correct specification of the estimated equation up to an additive error term.

19/

The effect of stock of debt reduction operations associated with Fund-supported programs would be reflected in measures of the debt service ratio before debt relief in years subsequent to the debt reduction. However, in the sample of program countries considered in this study no stock of debt operations linked to Fund-supported programs were undertaken.

20/

The residuals of most of the estimated equations fail to pass the Jarque-Bera test for normality. T-tests should, therefore, be interpreted cautiously as they may be sensitive to nonnormality in a fashion that is determined by the numerical value of the regressors. This cautionary note applies also to other regression diagnostic tests (see Jarque-Bera (1987)).

21/

Khan estimates the GEE for a slightly different array of economic variables. Targets comprise the real GDP growth rate, the inflation rate, the ratio of the current account to GDP, and the ratio of the balance of payments to GDP. Policy variables are the growth of domestic credit, the ratio of the fiscal balance to GDP, and the real effective exchange rate. A time trend and the change in the terms of trade were also included.

22/

The γkj parameters cannot be identified from the single equation estimates because the number of structural parameters exceeds the reduced form coefficients. However, by pooling the parameter estimates from the three equations the γkj parameters can be identified if policy instruments and macroeconomic targets are equal in number. When the number of instruments is less than that of targets the γkj parameters cannot be identified, and when they exceed the number of targets, multiple solutions exist. It is not possible to measure the standard errors of the estimates. For the alternative specifications of the GEE (equations (5) and (15)), which introduce lagged exogenous variables and an unrestricted serial correlation coefficient on lagged target variables, the γkj parameters are not identifiable regardless of the number of targets and instruments.

23/

Goldstein and Montiel (1986), Greene (1989), and Khan (1990) do not attempt to correct for this potential source of bias, but several studies of World Bank adjustment lending do (see World Bank (1990), Faini et al. (1991), and Corbo and Rojas (1992)).

24/

The inverse Mills ratio is φ(-δ′Di)/[1-Φ(-δ′Di)] where φ and Φ are, respectively, the density and distribution function for a standard normal variable; D and δ are a vector of explanatory variables and parameters, respectively, in the probit model. The ratio is a monotone decreasing function (ranging from 0 to ∞) of the probability that an observation is selected into the sample of nonprogram countries, [1-Φ(-δ′Di)].

25/

The key problem in heterogeneity bias is that the imposition of identical parameters leads to an averaging of coefficients that differ greatly across countries (or time) and therefore produces nonsensical results. In effect, the assumption of a “representative agent” that can be described by an average is not valid.

26/

To the extent that the high estimated standard errors found in this study are a general feature of GEE estimates, measuring parameter instability may not be very productive because the data fit the model too imprecisely for stability tests to have much power.

29/

For example, the presence of heteroschedastic errors can be taken into account in estimation and testing of coefficient variation; this property is particularly appealing in pooled cross-section, time-series data.

30/

Stability is assessed in terms of the point estimates and standard errors of individual parameters. An alternative approach would be to conduct Chow or Wald tests for the joint stability of the coefficients of interest on the recursive estimates of the equations. However, this was not possible because the fitted equation does not meet the requirement of both tests that the set of regressors remains constant over recursive estimates: the country dummies, which entered with statistically significant coefficients, change across the recursions.

31/

The additional program country observation was required to avoid singularity in the presence of the IMF dummy variable (di).

32/

These recursive procedures consider only a small subset of all subsamples that could be drawn from the data set. Thus, the range of coefficient estimates reported in Tables 3-6 does not necessarily encompass global maximum and minimum values for the sample as a whole.

33/

The size of the initial block of observations was chosen so as to start with a reasonable number of degrees of freedom (20).

34/

The null hypothesis of an estimated coefficient equal to minus one on the lagged dependent variable is rejected at the 10 percent level for real GDP growth, 5 percent level for the inflation rate, and 1 percent level for the debt service ratio.

35/

For the external debt service ratio, statistical significance was at the 10 percent level.

36/

In terms of the reduced form GEE, a correlation between the πi and the error term in (εij + βjkηik) in equation (3) will lead to biassed coefficient estimates.

37/

This ratio is a monotone decreasing function (bounded by 0 and ∞) of the probability that an observation is selected into the sample of nonprogram countries, φ(δ/D) or identically [1-Φ(-δ/D)].

38/

An alternative procedure to obtain a consistent estimate of βIMF would be to use the predicted probability of undertaking a program as an instrument for di in equation (3); in this case the policy reaction function need not be estimated. This procedure also corrects for the possibility that the choice of a country to have a program may depend on expectations of better performance in the target macroeconomic variables, i.e., the dummy is endogenous with respect to the dependent variable in equation (3). See World Bank (1990) and Corbo and Rojas (1992) for applications of this procedure.

39/

The overall balance of payments was measured including scheduled debt service payments and excluding exceptional financing, in order to provide a measure of underlying pressures on the external position. An alternative indicator of an external need to seek a Fund-supported program would be the level of international reserves. However, for several countries in the sample (CFA and ECCB country members) data on reserves are not available.

40/

Two versions of this variable were tried: one in which both positive and negative changes in arrears were recorded, and one in which only positive changes were included. The second version isolates the buildup of arrears, (typically prior to programs) and excludes the repayment of arrears (typically during programs).

41/

The probit residuals (πi) were regressed upon a constant, and the residuals of this auxiliary regression revealed evidence of nonnormality; the Jarque-Bera test for nonnormality had a test statistic of 47.29, statistically significant at the 1 percent level.

The Macroeconomic Effects of ESAF-Supported Programs: Revisiting Some Methodological Issues
Author: Ms. Susan M Schadler, Mr. Louis Dicks-Mireaux, and Mr. Mauro Mecagni
  • View in gallery

    Recursive Point Estimates and Confidence Intervals for IMF Program Dummy Variable

    (Real GDP growth rates)

  • View in gallery

    Recursive Point Estimates and Confidence Intervals for IMF Program Dummy Variable

    (Rate of inflation)

  • View in gallery

    Recursive Point Estimates and Confidence Intervals for IMF Program Dummy Variable

    (External debt service ratio)