A Simple Model of Program Effects

For the purposes of this paper, it was desirable for such a model to have four broad features. First, the model should be general enough that the two dominant existing statistical approaches to ex post program evaluation—the before-after approach and the control-group approach—could be treated as special cases of the more general model. In this way the assumptions implicit in the existing methodologies can be identified and evaluated. Second, the model should incorporate nonprogram determinants of macroeconomic outcomes of both an international and a country-specific nature. Third, given that Fund stabilization programs operate primarily by altering the design or stance (or both) of macroeconomic policies, the model should indicate the determinants of indigenous changes in macroeconomic policy so that the macroeconomic outcomes expected in the absence of a program can be explicitly defined. In other words, we want a model in which the policy instruments and not just macroeconomic out-comes are endogenous. Finally, the model should indicate what objective factors, if any, determine the probability that a country will have a Fund program during a given period. The reason for treating Fund program status or program-country selection as an endogenous variable is that this is the only way in which to investigate the consequences of systematic differences between program countries and nonprogram countries. Obviously, if such differences exist before a program period, they need to be taken into account in any subsequent comparison of program and non-program countries to the extent that they affect macroeconomic performance. Failure to include such differences would mean that variations in macroeconomic performance between the two country groups could be attributed to the presence or absence of a program, when in reality the differences might in large part reflect other factors.

In equations (1) through (4) below, we set out a model of Fund program effects that contains these basic features:

In these equations, y_ij is the; th macroeconomic outcome or target variable in country i; x_i is a K-element vector of macroeconomic policy variables that would be observed in country i in the absence of a Fund program; W is an M-element random vector of world nonprogram variables; z_i is a random variable that serves as the index of country-specific characteristics that determines the probability of country i having a Fund program during a given period; d_i is a dummy variable that takes the value of unity if a country has a Fund program and the value of zero otherwise; y^d_i is the desired value of the vector y_i; z* is the threshold value of z_i that divides program from nonprogram countries; ε_ij, η_i, and π_i, are unobservable error terms (with zero means and fixed variances) that are serially and (for simplicity) mutually uncorrected; β_ij, α_ij, β_ij^IMF, γ, and δ are constants with the appropriate dimensions; Δ is the first-difference operator; the subscript -1 indicates the previous period; and a prime (′) denotes the transpose of a matrix.

The variable y_ij in equation (1) should be considered as one of the primary targets of a stabilization program, such as the current account, the overall balance of payments, the inflation rate, the real growth rate, and the like.⁶ Equation (1) can then be interpreted as positing that the change in this macroeconomic outcome or target variable will be a function of four factors: (1) changes in macroeconomic policy instruments (for example, the rate of domestic credit expansion; government tax revenues, expenditure, or both; the exchange rate; and the like) that would have occurred in the absence of a program; (2) changes in world economic conditions (such as changes in world oil prices or changes in real economic activity in industrial countries); (3) the total effect of a Fund program if the country has a program in place during that period; and (4) a host of unobservable shocks that are specific to country i.

A special word of comment is appropriate for β_ij^IMF, which is the coefficient that indicates the effect of a Fund program on macroeconomic outcomes. In our view, this coefficient should incorporate at least three channels or avenues by which Fund programs can affect y_ij. First, Fund programs can alter the value of macroeconomic policy instruments from what that value would be in the absence of such programs. Note, however, that since Δx_i is defined as the change in policy instruments that would occur in the absence of a program, Δx_i is directly observable only for non-program countries; for program countries, Δx_i must be estimated (through equation (2)). In any case, the important implication is that a program can affect y_ij by making the actual change in policy instruments different from Δx_i. The second potential channel of program effect is by altering what might be called the general state of confidence about the economy of country i. Here the successful negotiation of a credible program with the Fund may, for example, have a positive effect on private and official capital inflows into country i that may indeed be quantitatively more significant than the financial resources supplied by the Fund itself in support of the stabilization package. This, like the first channel of effect, is of course an empirical question; suffice it to note here that the measurement of such confidence effects of programs is an extremely difficult task in practice. The third and final channel of potential program effect is by changing the parameters β_ij for any given size change in the policy instruments. In other words, programs can work not only by, say, making monetary and fiscal policies more restrictive than they would otherwise be, but also by improving or reducing the effectiveness of any given stance of policy. The ways in which behavioral parameters can shift in response to policy changes have been outlined by Lucas (1976), but here it is enough merely to note that programs can change the expectations of agents in the economy about the future course of x_i and y_ij, and these altered expectations can in turn affect Δy_ij.⁷ The assumptions in equation (1) that unobservable country-specific shocks have zero expected means and are serially un-correlated imply that, other things being equal, a negative shock to, say, country i’s balance of payments in period f is not expected to be repeated in the next period. In other words, the model contains a regression-to-the-mean characteristic for macroeconomic outcomes that provides for some automatic stabilization. Of course, only the data can decide whether such an assumption is consistent with the recent experience of program and nonprogram countries.

The basic notion represented in equation (2) is that the authorities display a systematic policy reaction to perceived disequilibria in their macroeconomic target variables. More specifically, equation (2) says that the change in country i’s macroeconomic policy instruments between the current and previous period will be a function of the difference between the desired value of the macroeconomic target variables in this period, y_i^d, and their actual value in the preceding period, (y_i)-1, with γ serving as the coefficient that indicates the responsiveness of the policy instruments to such target disequilibria. For example, in the case of stabilizing policy behavior, equation (2) would suggest that a current account deficit in the preceding period that was large relative to the authorities’ target deficit would call for a downward adjustment in, say, the rate of domestic credit expansion in this period. Because Δx_i is defined as the change in policy instruments in country i that would occur in the absence of a program, equation (2) spells out “normal” policy behavior by the authorities and thus provides one approach to estimating the counterfactual for program countries.

In addition, so long as γ carries the correct sign, equation (2) also implies that there may well be stabilizing policy action in non-program countries. Idiosyncratic, country-specific policy behavior is intended to be captured by the error term η_i, in equation (2).

Equations (3) and (4) constitute perhaps the greatest departure in this model from the earlier literature on program evaluation by suggesting that the presence or absence of a Fund program should itself be treated endogenously and, in particular, as a function of observable country-specific characteristics. There are strong a priori reasons for believing that Fund program status is not random. A necessary (but not sufficient) condition for the use of Fund resources is that the country display a balance of payments need.⁸ This implies that, among the population of potential claimants for Fund resources, the sample of countries with Fund programs in place at any given time is likely to have displayed less favorable external balance performance before the program period itself than the population at large.

As written, equation (3) uses the difference between the desired values of macroeconomic target variables in this period and their actual values in the preceding period to explain the probability that country i will have a Fund program this period. Under the assumption that the desired target values y_i^d are constant over time but not necessarily across countries, this difference reduces to a formulation in which the actual values of macroeconomic outcomes in the preprogram period, (y_i)_-1, influence program-country selection. Equation (3) should therefore be capable of capturing systematic selection of program countries by the Fund on the basis of balance of payments need because the vector (y_i)_-1 will include preprogram values of country i’s external accounts.

Note further that such specification of equation (3) deliberately makes for a potentially serious problem. As written, the preprogram outcomes (y_i)_-1 help to explain program selection in equation (3) and also policy reaction Δx_i in the absence of a program in equation (2). Because both z_i and Δx_i can influence the change in macroeconomic outcomes between the program and preprogram year, it can be seen that our model sets up the troublesome possibility that it will be difficult to separate program from non-program determinants of Δy_i. As we shall demonstrate later, this problem will be present so long as the determinants of Fund program status d_i are correlated with the determinants of Δy_i, whether through Δx_i or through any other variable explaining Δy_i. As also discussed later, this problem disappears if program selection is random, since δ in equation (3) will then be zero; that is, it will not be possible to relate program-country selection z_i—hence, ultimately, d_i—to any observable objective factors.

With the general outlines of this model of Fund program effects in mind, we can next proceed to analyze how estimated program effects will differ from true program effects under a variety of shorthand estimation techniques.

The Before-After Approach

This approach to ex post program evaluation has been used both by Fund staff (for example, Reichmann and Stillson (1978)) and by outside observers (for example, Connors (1979), Killick and Chapman (1982)). Although these studies utilized multicountry samples, this approach is not necessarily a cross-sectional technique because the (implicit) parameters estimated are allowed to differ across countries.

Recalling that β_ij^IMF is the “true” effect of a Fund program on the j th target variable in country i, the before-after approach estimates β_ij^IMF—call it β_ij^{IMF, A}— as:

where P denotes the set of program countries. Thus, any change in a target variable in a program country (or in a group of program countries) is attributed exclusively to program effects. The estimate, β_ij^{IMF, A}, is sometimes subjected to (nonparametric) statistical tests of significance and sometimes not.

The fatal flaw of the before-after approach is that it relies on assumptions of other things being equal that are highly implausible. To see this, let us introduce only equation (1) from the general model of program effects:

Now suppose that the preceding period was one during which there was no Fund program in effect (so that d_i = 0 for t - 1). Then,

From equation (5), the before-after approach then gives

Taking expectations of equation (7), conditional on the presence of a Fund program in country i and on observed changes in the world economic environment, we have

Thus, the before-after approach would produce an unbiased estimate of program effects—that is, $E\left({\beta }_{ij}^{IMF,A}|iєP,\Delta W^{\prime }\right)={\text{β}}_{ij}^{IMF}$ and only if

In other words, an unbiased estimate of program effects would require that the nonprogram determinants of y_ij would have behaved in such a way as to leave y_ij unchanged, on average, between the preprogram and current program periods. Reference to the 1973–81 period—when large changes in world oil prices, large year-to-year changes in industrial country real gross national product (GNP), and significant shifts in real interest rates created serious difficulties for the external positions of developing countries (see International Monetary Fund (1983), Goldstein and Khan (1982), and Khan and Knight (1983))—gives sufficient reason to doubt that ΔW′α_ij, would be zero for most program countries, even over short periods. By the same token, the ex post record of money supply growth and fiscal deficits by developing countries during the same period (see International Monetary Fund (1983) and Gylfason (1983)) generates skepticism that changes in domestic policy instruments would have been such as to offset exactly, on the average, the effects of external and internal shocks for most program countries. In short, we should expect the other-things-equal assumption of the before-after approach to be violated in practice. As such, estimates of program effects under this approach are likely to be contaminated by nonprogram factors.

The Traditional Control-Group Approach

A second approach, hereafter called the traditional control-group approach, has a long history in empirical labor economics but appears to have been first applied to analysis of experience with Fund programs by Donovan (1982).⁹ More recently. Gylfason (1983) has adopted a more sophisticated version of it.

This technique in effect uses the behavior of a control group (a group of nonprogram countries) to estimate what would have happened in the program group in the absence of programs. Thus it implicitly assumes that only the program itself distinguishes the group of program countries from the control group. It is therefore natural to interpret this as a cross-sectional approach. Specifically, in terms of the model, we can drop all the country i subscripts from the coefficients because these are now assumed to be identical across countries. In addition, because we are now dealing with country groups, β_ij^IMF represents the mean effect of Fund programs on the j th macroeconomic target variable. The equation for Δy_ij can now be written as

Under the control-group approach, β_ij^IMF is estimated by

where a bar over a variable represents its mean, and N denotes the set of nonprogram countries.

To investigate the properties of this estimator, we again take expectations. Applying this procedure to equation (11) yields¹⁰

From equation (12), it can be seen that the condition for ${\widehat{\beta }}_{ij}^{IMF,B}$ to represent an unbiased estimate of the true program effects, ${\beta }_{ij}^{IMF}$, is that

In other words, the groups of program and nonprogram countries have to be drawn from the same population in the sense that the expected value of the change in nonprogram determinants of y_ij must be the same for members of both groups. Comparing equation (13) with equation (9) shows that the control-group approach is not necessarily less restrictive than the before-after approach. Although the control-group approach controls for the effect of changes in the global economic environment (that is, the term ΔW′α_ij that appears in equation (9) drops out as a source of bias in equation (13) because such global factors are assumed to affect program and nonprogram countries equally), it does so at the expense of introducing a new source of bias—the characteristics of nonprogram countries (that is, the term $-E[(\Delta {x^{\prime }}_i{\beta }_j+\Delta {є}_{ij})|iєN]$ appears in equation (13) but not in equation (9)).¹¹

The foregoing suggests that the choice between the before-after approach and the control-group approach to estimating program effects ought to depend on one’s a priori beliefs about similarities between program and nonprogram countries and about the relationship between domestic and global determinants of Δy_ij. Specifically, if program and nonprogram countries are believed to be quite similar on average, and if the domestic determinants of Δy_ij are not believed to offset international influences on Δy_ij, then equation (13) is more likely to be satisfied than equation (9); hence the control-group approach will provide a better (less biased) estimate of program effects than will the before-after approach. We next proceed to investigate, first, the nature of the bias that is generated by this methodology when the determinants of program-country selection are correlated with the determinants of macroeconomic performance and, second, the nature of the biases in the specific (but probably most relevant) case in which both program-country selection and macroeconomic performance depend on macroeconomic performance before the program period.

Nonrandom Selection of Program Countries

To examine these issues it is helpful to introduce the index of unobservable country-specific characteristics, z_i, that regulates the probability that country i will have a program during any given period. Specifically, we now introduce equation (4) from the general model of program effects:

where z* is an arbitrary threshold value for z_i. Instead of also introducing equation (3), assume for the moment that E(z_i) = 0 and $E(z_i^2)={\sigma }_z^2$ Equation (4) says that a country will have a program if its index of country-specific characteristics is greater than z*; if not, it will not have a Fund program, at least in that period. The probability that a country will have a program is therefore equal to the probability that z_i >z*.

We can now use equation (4) to rewrite the necessary condition, as previously expressed in equation (13), for an unbiased estimator of the program effects under the control-group approach; that is,

Recall that both Δx′ _iβ_j+Δєc_ij and z_i are random variables. Suppose that the correlation between these two variables is given by ρ_xz and that the expected value of $\Delta {x^{\prime }}_i{\beta }_j+\Delta {є}_{ij}$ We show in Appendix I that if Δx′ _iβ_j+Δєc_ij and z_i have a joint normal distribution, then

In other words, if Δx′_i β_j + Δє_ij and z_i are correlated (ρ_xz ≠ 0), then our expectation of Δx′_i β_j+Δє_ij will depend on the value taken by z_i. This result is intuitive. Suppose, for example, that ρ_i is positive. Then, relatively large values of Δx′_i β_j+Δє_ij are associated with relatively large values of z_i. Thus, if we know that z_i is relatively large for some country i but do not observe Δx′_i β_j+Δє_ij our expectation is that Δx′_i β_j+Δєc_ij will also be relatively large for this country. Likewise, if ρ_xz is negative, then relatively large values of z_i will be associated with relatively small values of Δx′_i β_j+Δє_ij and observing a large z_i will lead us to expect a small Δx′_i β_j+Δєc_ij. In contrast, if Δx′_i β_j+Δє_ij and z_i are known to be uncorrelated, then observing a large z_i gives no basis on which to expect Δx′_i β_j+Δє_ij to be either particularly large or particularly small for the ith country.

Because program countries are those for which z_i>z*, program countries as a group will exhibit a relatively large z_i. Likewise, the representative z for the nonprogram group will be relatively small. It follows that if ρ_xz > 0 the difference between the program and nonprogram groups with respect to the expected change in target variable y_j consists of both the effect of the program on the change in y_j and the difference between the relatively large Δx′_i β_j+Δє_ij expected for the program group and the relatively small Δx′_i β_j+Δє_ij expected for the nonprogram group. Because this second component of the expected difference must be positive, the expected difference will exceed the true effect of the program on the change in target y_j. A similar analysis establishes that the expected difference will be less than the true program effect when ρ_xz<0. When ρ_xz = 0, it remains the case that z_i is relatively large for program countries and relatively small for non-program ones, but this gives no reason to expect that Δx′_i β_j+Δє_ij will be systematically different between the two groups; therefore, the only difference we are justified in expecting is that attributable to the effects of the program. These considerations imply that

The relations in expression (15) can be derived formally by substituting expressions (14a) and (Hb) in equation (12).

Thus, ρ_xz is the crucial parameter in determining the direction of the bias in the control-group methodology. Specifically, if the determinants of program selection (z_i) are positively correlated with the determinants of macroeconomic performance that would have occurred in the absence of a program (Δx′_i β_j+Δє_ij), then the control-group estimate of program effects (β_ij^{IMF, B}) will overstate true program effects (β_ij^IMF). Conversely, only if the determinants of program selection are uncorrelated with the determinants of macroeconomic performance (ρ_xz = 0) is the control-group estimator an unbiased indicator of true program effects.

The significance of the preceding analysis is that it permits us to move from the vague statement that “if the program and non-program groups are different, then the control-group approach will be biased”—a statement that is not correct—to the precise identification of ρ_xz as the critical parameter determining both the presence and the direction of bias.¹² In assessing the adequacy of the control-group methodology, the relevant question then is whether there are any reasons inherent in the nature of the problem that would lead us to believe that this correlation (ρ_xz) will be nonzero.

The model embodies precisely such a nonzero correlation because both the determinants of program status in equation (4) and of normal policy changes in equation (2) are linear functions of macroeconomic outcomes before the program period. To show this formally, first rewrite the model by taking the transpose of equation (2), substituting for Δx′_i in equation (1), and making some small changes in notation:¹³

where

To determine whether the control-group estimator of program effects will be biased, we again need to examine the correlations between −(y_i)′_-1γ′β_j and z_i and between Δ̃ε_ij and z_i. These will be determined by the signs of

where Σ_y-1 is the covariance matrix of ${(y_i)}_{-1},{\sigma }_{є}^2$ is the variance of ε_ij, and δ_j is the j th component of δ. These results assume that ε_ij, η_i, and π_i are mutually uncorrelated.

The crucial thing to notice about the covariances portrayed in equations (18a) and (18b) is that they can in general be expected to be nonzero—a finding that implies that the control-group estimator of program effects will be biased. The more interesting issue, however, is why this estimator turns out to be biased. Our analysis suggests that the determinants of program status will be correlated with the nonprogram determinants of Δy_ij for two reasons.

First, preprogram values of key macroeconomic target variables, (y_i)_-1, are likely to trigger policy responses, Δx_i even in the absence of programs, as originally suggested in our policy reaction function (equation (2)). In terms of equation (18a), this assumption shows up as a nonzero value for the covariance ${{\beta }^{\prime }}_i\gamma {\displaystyle \sum {}_{y-1}\delta }$

Second, negative transitory shocks in the preprogram period are by their very nature unlikely to recur during the program period (recall that ε_ij has an expected value of zero and is assumed to be serially uncorrelated), with the result that changes in macroeconomic target variables between the preprogram and program periods, Δy_i, will display regression to the mean with respect to past shocks; in terms of equation (18b), this result shows up as a nonzero value for σ² δ_j.

We next inquire about the direction of this bias. For that source of bias that arises from regression to the mean, we can provide an unambiguous answer under reasonable assumptions. This is not possible for the bias arising from the existence of policy reaction functions; in this case we can, however, spell out the conditions necessary for that source of bias to disappear.

Consider the bias arising from regression to the mean. Because Fund programs are designed to move target variables in the desired direction, we expect the product of β_ij^IMF and δ_j to be greater than zero (that is, β_ij^IMF δ_j). The logic is that, if a below-target value of y_ij (for example, the current account surplus) causes a country to come to the Fund for assistance (δ_j >0), the Fund program will seek to increase actual y_ij (that is, β_ij^IMF > 0); hence, β_ij^IMF δ_j > 0. Likewise, for those target variables (for example, the rate of inflation) for which the likelihood of program participation is increased when (y_i)-t >y_i^d, then δ_j<0 and we can expect β_ij^IMF < 0; here, too, the product β_ij^IMF δ_j will still be greater than zero. The relevance of β_ij^IMF δ_j); hence, β_ij^IMF is that, because we know from equation (18b) that the correlation between Δ̃δ_ij and z_i carries the same sign as δ_j, we can conclude that regression to the mean contributes to a correlation between the determinants of program Status (z_i) and the nonprogram determinants of Δy_ij (that is, ρ_xz), a correlation that has the same sign as β_ij^IMF. From our earlier analysis, especially equation (15), we then know that in these circumstances the control-group approach will overstate the true effect of a Fund program. In short, if program countries are more likely to have experienced negative temporary shocks in the preprogram period, a comparison of changes in mean macroeconomic outcomes between program and nonprogram countries will, under plausible assumptions, overstate the beneficial effect of a program. A negative shock in the preprogram period simultaneously increases the probability of program participation and increases the probability of a positive change in y_ij in the program period. Thus, attributing all of this improvement in y_ij to a Fund program overstates the true independent effect of the program.

The direction of bias arising from the existence of policy reaction functions depends on the characteristics of such functions, which is of course an empirical question. Nevertheless, we can show that the bias will disappear under two conditions.

The first condition is that δ = 0; that is, when Fund program status can no longer be related to observable country characteristics and when all countries therefore have an equal probability of becoming program countries. In this case, the covariance represented by equation (18a) is zero as long as π_i is uncorrected with ε_ij and η_i. In this case of random selection, both sources of bias disappear because the original premise of the control-group approach—that program and nonprogram countries are similar— is satisfied.

The second condition for the policy-reaction bias to disappear is γ = 0. Again, this would make the covariance represented in equation (18a) equal to zero under our assumption. In other words, if γ = 0, the policy reactions of the authorities cannot be systematically related to observable characteristics; that is, we would not observe the systematic policy reaction functions represented by equation (2). Note, however, that even when γ = 0, the bias in the control-group estimator attributable to regression to the mean would still remain. This is so because δ = 0 eliminates the improvement in y_ij that is attributable to nonprogram policy actions but not that improvement attributable to automatic stabilization from reversible country-specific shocks.

To sum up, we have argued in this section that there are strong ex ante reasons for believing that the past procedures used to estimate the effects of Fund programs in multicountry samples are subject to significant sources of statistical bias. Because the non-program determinants of macroeconomic outcomes cannot in general be expected to behave in such a way as to leave these outcomes unchanged from year to year, the potential problems with the before-after approach can be readily acknowledged. The problems with the control-group approach are also important, but they are perhaps more subtle. As shown above, comparing mean macroeconomic outcomes between groups of program and non-program countries will lead to biased estimates of program effects whenever the determinants of program selection are correlated with the determinants of macroeconomic outcomes that would have occurred in the absence of a program.

A Modified Control-Group Estimator

Consider the following modified estimator, β^_j ^{IMF, M} for Fund program effects:

Reference to equation (11) reveals that this modified estimator differs from the traditional control-group estimator in two respects: the modified estimator contains the additional term $-(\overline{x_P}-\overline{x_N}{)}^{\prime }{\beta }_j$, and it is specified in level rather than in first-difference form.¹⁴

To investigate the properties of this estimator, write the basic equation for the jth macroeconomic target variable in country i (equation (1)) in level form:

Taking expectations of equation (19), after substituting from equation (20), we then obtain

Thus, the modified control-group estimator will be unbiased so long as the unobservable country-specific determinants of y_ij (that is, ɛ_ij), are uncorrelated with the determinants of program status (Z_i). In such a case, one can set $E({\epsilon }_{ij}|z_i>0)=E({\epsilon }_{ij}|z_i\le 0)=0$ and thereby justify the last equality above.

The reason that the modified estimator is unbiased can be explained intuitively by using the conclusions from Section I. Recall that we established there that the traditional control-group estimator would be biased if the nonprogram determinants of Δy_ij (that is, changes in domestic macroeconomic policy and changes in unobservable shocks) differed systematically between program and nonprogram countries (that is, if Δx_i and Δɛ_ti were correlated with Fund program status, z_i). The modified control-group estimator removes both sources of bias present in the traditional version. By subtracting the term $({\overline{x}}_P-{\overline{x}}_N{)}^{\prime }{\beta }_j$ an adjustment is made for any differences in indigenous macroeconomic policy between program and nonprogram countries. As regards the second potential source of bias (regression to the mean), note that systematic differences between program and nonprogram countries with respect to changes in unobservable shocks are to be expected only because the program-selection rule makes it more likely that countries with negative shocks in the preprogram period will subsequently adopt programs. But also note that the expected level of such shocks—that is, E(ɛ_ij)—is zero for all countries. Thus, under our assumptions about the distribution of ɛ_ij, this source of bias is present in estimators expressed in first-difference form that fail to control for prior shocks but not in those (such as the modified estimator) expressed in level form.

Operational Aspects of the Modified Control-Group Estimator

The traditional control-group estimator has an obvious attraction: estimated program effects require only the calculation of mean changes in macroeconomic outcomes for program and for nonprogram countries; that is, only of Δ¯ y_jp and (Δ¯ y _j)_N The estimation requirements for the modified control-group estimator, however, are substantially more demanding. Not only do we need values for three additional variables or parameters (¯x_N, ¯x_p, and β_j), but we also face the problem that two of these (¯x_p and β_j,) are not observed directly. Recall that ¯x_p is not observed because x_i refers to policies that would have been undertaken in the absence of programs; thus, x_i is equal to observed policies in nonprogram countries but not in program countries. Hence, implementation of the modified control-group approach requires estimating x_i for program countries (as well as estimating the parameter β_j linking Δx_i to Δy_ij).

The policy vector x_i is generated by the reaction function (2). In practice, an important limitation of the modified control-group estimator is that such reaction functions may be highly unstable, both across countries and in a given country over time. On the one hand, in extreme cases of instability the problem of estimating the counterfactual becomes insoluble. On the other hand, if crosscountry instability is dominant, the solution is to abandon multi-country samples in favor of country studies. In any case, the issue is an empirical one. In what follows, we describe the calculation of a modified control-group estimator that is conditional on the existence of stable policy reaction functions.

The first step in estimating xi for program countries is to fit the reaction function (equation (2)) to observable data for non-program countries. The only unobserved variable in equation (2) is the country-specific vector of desired macroeconomic out-comes, y_i^d. If this variable can be assumed to be constant over time, it can be captured by a set of country-specific constants, giving the policy-reaction equation the following final form:

The fitted values of this equation for program countries constitute the counterfactual Δx_i. In effect, this procedure uses data on observed policy behavior in nonprogram countries to identify normal policy reaction in given policy-target circumstances.¹⁵ This normal policy reaction is then used to estimate what “would have been” in program countries if there had not been a Fund program.

An important caveat is in order about another potential source of systematic differences between program and nonprogram countries. Because both the setting of policy instruments in equation (22) and the acceptance by a country of a Fund program as specified in equation (3) reflect policy decisions of the authorities, any unobservable factors, π_i, that make a given country more likely to go to the Fund for assistance—such as a general commitment to adjustment—may also make that country more likely to have adopted a different policy package in the absence of a program, Δx_i, than another country facing similar observable (policy-target) circumstances. In this case, the behavior of nonprogram countries would not be a good guide to the counterfactual in program countries —even after observable preprogram characteristics of the two groups are controlled for. Formally, this possibility would manifest itself in the model as correlation between the error terms π_i, in equation (3) and η_i in equation (2). If such a correlation is present, then equation (22) will provide a biased estimate of Δx_i for program countries —in essence because it fails to remove this additional source of sample-selectivity bias.¹⁶

But this additional source of bias can be eliminated, even though both η_i and π_i, are unobservable. The reader is referred to Heckman (1979) for a description of the appropriate procedure. For our purposes, we note that the procedure requires the specification and estimation of a model of program participation —that is, of equation (3). Thus, removal of the two sources of sample-selectivity bias we have identified requires the specification and estimation of models of endogenous policy formation (equation (2)) and program participation (equation (3)).

With (¯y_j)_P, (¯y_j)_N and ¯x_N observed directly, and with ¯x_P estimated as outlined above, the remaining element necessary for application of the modified control-group estimator is the parameter vector β_j, which links normal policy changes in the absence of programs to changes in the macroeconomic target variables. Until now, this vector has been assumed to be known. For our purposes, any unbiased estimator of _j will suffice. Perhaps the simplest way to produce such an estimate is to fit the macro-economic outcome equation (20) in level form to a pooled cross-sectional time-series data sample by using observed values for the policy vector x_i.¹⁷

If the objective is solely to obtain an unbiased estimate of total program effects, we can substitute the policy-reaction equation (2) for Δx_i into the level-form equation (20) and derive

Fitting equation (23) to observable data will then yield an estimate of total program effects through the estimated coefficient β_j^IMF on the dummy variable d_i. This procedure does not, however, take into account any sample-selectivity bias arising from systematic differences in reaction functions between program and non-program countries. If the error terms in equations (2) and (3) are correlated, then the reduced-form approach has to be augmented by the Heckman (1979) correction in order to obtain unbiased estimates of program effects. This shortcut works because it essentially controls for observable differences between program and nonprogram countries. But it cannot yield information on how total program effects are apportioned between changes in policy instruments and other factors.

Analyzing How Programs Work

To analyze the three different channels by which programs can affect macroeconomic outcomes, it is helpful to introduce some additional notation. Let x_{i, IMF} be the vector of policy instruments adopted under a program, βx_{j, IMF}-the vector of coefficients linking these policy instruments to the target variable y_ijy, and CON_{i, IMF} any unmeasurable confidence effects on y_ij attributable to a program. As before, x_i and β_j will be the values of policy instruments and their coefficients in the absence of a program. We can then express the total effect of a Fund program, β_j^IMF, as:

Rewriting the level-form equation (20) for y_ij with the substitution for β_j yields

It is clear that if separate estimates of β_{j, IMF} and CON_{i, IMF} could be obtained, it would be possible to identify the separate channels through which a program affects y_ij. Given the estimate of x_i, for program countries and the estimate of β_j as outlined above, we next need to estimate the following equation: