The World Bank has long been well known for the care and intensity of its project appraisal before supporting an investment. But how much is actually realized of the forecasts and projections made? Was that degree of thoroughness and detail worthwhile? How sound have the Bank’s decisions proved in hard reality? What do we learn from experience?
The 1960s saw the start of systematic attempts, inside and outside the Bank, to answer questions of this type. The main stimulus was probably the enormous growth, especially in the United States, of public expenditures on social programs that were not readily subject to market measures of performance. The Bank itself set up a unit specifically devoted to what we call evaluation in 1970.
Six years later one can see some signs that evaluation will be applied much more widely, beyond the public sector: heightened recognition of corporations’ broader social responsibilities, growing impatience with traditional global financial indicators, and increased responsibilities, especially in the United States, of the “audit committees” of corporations’ boards of directors. Ten years from now, systematic review and analysis of investments after they have been made may become the general norm.
This article is based on a talk given by the author in April 1976 at the faculté des Science Economiques of the University of Clermont-Ferrand, France.
While “evaluation” has increasingly tended to take on the sense, in English, of a retrospective review, or a review of something at least already ongoing, the more elegant Latin languages still use the same word for such analysis whether ex ante (a priori) or ex post (a posteriori). From the point of view of economic method, it is true that the difference is only one of perspective. The central focus of any investment analysis is the comparison between the situation with the investment and that without it. The net effect of the investment is found by deducting the flow of costs and benefits without investment from the flow representing the situation with investment. Ex ante, both flows must be projected on the basis of certain hypotheses. Ex post, one flow—with investment—is at least partially supplied by history (with more or less precision, depending on the data available), and it is necessary to extend this to cover the outstanding 10, 20, or 30 years of the project’s expected life and to invent a representation of the situation without investment. Considerable judgment is involved in either case.
The only feature that might be considered peculiar to ex post analysis, still not different in principle from ex ante analysis, is the correction which may have to be made to recorded prices and costs for purposes of economic analysis. Financial figures may need adjustment by a price index to eliminate the effects of inflation. Equally, shadow prices, if used at the time of appraisal, will need to be reassessed if the long-term outlook for the national economy has changed considerably in the meantime. For the purposes of economic analysis, similar amounts of money in different periods should represent approximately the same amount of real resources for the owner of the investment.
Effectiveness, efficiency, economy
The main questions to be asked about an investment, whether in prospect or in retrospect, are always the same. First, effectiveness: are the means provided and the measures envisaged or undertaken in connection with the project adequate to achieve the physical objectives sought? Second, efficiency: is the economic rate of return at least equal to what one would expect to attain in other lines of investment (generally assumed to be about 10 per cent)? Third, economy: are the investment and operating costs per unit—for example, per beneficiary family, per grad uating student, or per kilometer constructed—reasonable in comparison with these costs for other similar projects in similar economic conditions?
Sometimes, of course, it is useful to evaluate a project by criteria additional to those that may have been used in the original appraisal—for instance, the effect of the project on income distribution or employment in a region. Road improvements have occasionally encouraged the migration of skilled people away from the region, and the penetration of manufactured consumer goods into it, without stimulating and increasing local production. The discovery and study of such unexpected effects are an important contribution of evaluation studies to our understanding of development phenomena. In addition, the application of a new criterion to a project previously undertaken can also shed important light on the usefulness of the new criterion for analyzing future projects.
Studies which deal impartially with all these questions fulfill very adequately the “accountability” function—showing how far the funds entrusted to the Bank by its shareholders and bondholders were effectively allocated to development. Much of the Bank’s evaluation work has in fact been devoted to meeting this responsibility.
Useful conclusions emerge too. Examples are the significance of “start-up” problems in translating an appraisal report expeditiously into action; the need for broader project definition (explicit inclusion of more of the essential complementary elements) in weaker economic-administrative environments that cannot respond automatically to the opportunities opened up; and the importance of operational efficiency to the final success of a project. Minor lessons are also useful operationally, such as the frequent need for large reserves for covering contingencies in the construction of tunnels, the inadequacy of studies on soil and construction materials in many past civil works projects, or the tendency sometimes to overlook the effects of variations in the weather from year to year in estimating agricultural yields.
Other conclusions concern economic results. For example, by the end of 1975 reports had been issued in the Bank on about 170 completed development projects—more than 20 per cent of the total number of projects completed with the financial assistance of the Bank and the International Development Association (IDA) up to that time. Of these well under 10 per cent seemed likely to yield less than the minimum economic return of 10 per cent sought by the Bank (or an approximate qualitative equivalent in sectors where returns cannot be soundly quantified). Some overall estimates of the average return on the more than US$10 billion investment involved in all 170 projects indicate a figure of around 15 per cent.
The Bank’s participation
Findings as described above represent only a part of the potential of ex post evaluation. Here the analogy between ex ante and ex post work may in fact be more constraining than stimulating. For insofar as a major objective of evaluation is to learn from accumulated experience, it is not so much projects completed that should be the focus of analysis as decisions taken by the Bank in their regard.
Any combination of success and failure between a project and the decision to finance it is possible. Despite a correct decision to undertake it, a project may largely fail due to unforeseeable developments during implementation—for example, a decline in the market for its output or a large increase in the price of inputs. If the original analysis had been done in terms of probability, it could be that all possibilities were well considered and very accurately weighed, but that the combination of events which had only a 5 per cent probability of occurring did actually take place. The exact opposite happened with six of the projects (in agriculture, energy, and roads) on which evaluation reports were circulated in 1975. They seemed to show satisfactory returns, but this was entirely due to the fact that deficiencies in the forecasts for costs, crop yields, and so on were offset by completely unexpected increases in the value of project output, brought about by increases in the world prices of energy and certain crops during recent years. From the point of view of return, these projects were successful. But, from a scientific standpoint, the fact that the original decisions were not correct in their own terms is more meaningful—especially when one considers that additional increases in prices of raw materials in general are not very likely to occur in coming years.
The importance of decisions as the focal point for evaluation, however, goes well beyond this type of situation, for the Bank’s decision as to whether or not to finance a project is only the final result of many decisions on a smaller scale regarding the composition of the project, the kind of prior studies requested, institutional arrangements required of the borrower, conditions of lending, and so on. These decisions are, in turn, all linked to decisions as to the quantity and quality of effort to be devoted to a project by the Bank itself, ranging nowadays from a brief appraisal of a ready-made project with a familiar borrower to virtual promotion from scratch, in a manner analogous to the activities of some nineteenth century European development banks. The Bank is, of course, responsible for allocating wisely not only the funds it lends, but also the considerable funds it spends to cover its own operation.
The key tests of effectiveness, efficiency, and economy thus apply equally to the Bank’s participation in a project. The analysis has to compare the outcome of the project with and without the Bank’s participation. In doing so, it is assumed, which is usually the case, that the project itself would have been carried out sooner or later, in one way or another, even without the Bank.
Questions as to the role of just one of the participants in a project—in this case the Bank—are naturally difficult to answer. The overall balance sheet to date is positive, with major contributions by the Bank itself to many projects and a distribution of effort which seems to have improved over time. Among the strengths of the Bank as reflected in the projects studied, one could point to useful reductions it has suggested in the scale of many of them, and the stress it has placed on financial prudence and good maintenance. Among the Bank’s relative weaknesses, one might mention its frequent difficulties in participating effectively at the level of borrowers’ institutional arrangements, training, and staffing issues. The studies have also indicated a need to somewhat increase the effort devoted to assistance in preparing projects at their earliest stages, and to divert a little of the staff effort involved at the margin in perfecting operations with strong borrowers to further work on more basic issues in the economically weaker countries.
Evaluation and action
A good study, properly focused on an agency’s field of action and recognizing the limits of its power, may still fall flat in the real world of heavy day-to-day operational pressures. Perhaps a characteristic weakness of evaluation studies is that they only add to the abundance of perceptive and intelligent studies that have had slight influence on action and which already fill libraries on social and economic development. It is also one of the reasons that the Bank has placed increasing emphasis on the preparation of evaluation studies of their own activities and operations by the departments and staff directly responsible and also by the Bank’s borrowers. A lesson discovered by oneself is more effective operationally than a lesson—even a more profound or a more specific one—imposed, so to speak, by a service specializing in evaluation. The latter can, instead, help to maintain a critical spirit by means of guidelines, questions and discussions, review of draft reports, and more thorough random checks.
Whether or not self-evaluation produces open and constructive self-criticism and results in objective and provocative evaluations, depends above all on the open-mindedness, courage, and self-confidence of the institution, its management, and staff. The World Bank has these qualities. But realization of the full potential of this way of organizing project evaluation work, started three years ago, remains probably the principal challenge to the Bank in this area.
The ex post project evaluation system described, and now applied to all projects completed with Bank-IDA assistance shortly after the end of their construction/investment phase, serves many users in many ways. But its principal use to the evaluation service itself is to assist in its main task of pursuing, by means of more thorough comparative studies, selected topics of broader importance. The purpose of these studies is to reach conclusions and make suggestions that are as specific and concrete as possible, for improving the Bank’s policy—either in very general areas like rural development or the role of national investment banks in development, or on procedural questions such as the Bank’s participation in project preparation, the selection and use of consultant engineers, or regulations governing the procurement of goods and services with loan funds.
The only difficulty with relying entirely on the ex post project evaluation system for drawing attention to cases and issues that need this more thorough treatment is that the projects covered are approved for Bank support under policies that are not necessarily still current. It is an intrinsic problem of evaluation to avoid an arid historicism while retaining the objectivity and independent viewpoint which are its primary strengths. This is why, without trying to turn itself into an Ombudsman (in the Scandinavian style) who receives and settles appeals from people who feel they have been poorly treated by government, the evaluation service at the Bank has always been interested in receiving any complaints that clients may have, so that they may be studied in detail and on a comparative basis, as an additional way of identifying possible improvements in the Bank’s policies. Thus, a good policy review typically covers certain recent cases, sometimes suggested by borrowers who have ideas about how to simplify or improve the Bank’s policy, and older cases, where one can see more clearly the final outcome of the policy adopted.
There are certain mechanisms for concentrating attention within the organization on the suggestions that emerge from these reviews. While the Bank is too large and multifaceted an organization for all the sources of most new policy initiatives to be pinpointed, it is clear that these reviews have produced the main identifiable impact of the evaluation effort on changes in Bank policy. Among the rather wide range of matters to which they have contributed one could cite, for example, the emphasis on strategies for development finance companies, the concept of sector lending, development of a more open policy toward land reform, greater attention to utility tariff structures and to the development impact of highways, new policies for promoting the development of domestic financial markets, or the use of project start-up plans. Some time after the completion of any such policy review, the evaluation service has undertaken independent surveys of the effectiveness of action taken by the Bank’s management on suggestions made. The first two reports of this kind were circulated to the Executive Directors in 1975, covering evaluations concluded in 1972 and discussing in detail the actions taken on each of the 50 suggestions which had been made.
“… a key element in a monitoring system is a clear initial statement of the project targets and of their rationale …”
These surveys found the Bank’s action up to that time to be less than satisfactory on only one suggestion, which concerned the general introduction of project monitoring. This involves establishing at the time of project appraisal very precise key objectives, year by year, in the form of indicators, and assuring a satisfactory information system to furnish on a regular basis the current values of these indicators for comparison with the projected values. Such an approach is, in fact, now followed in more and more projects.
Later evaluation studies have underlined that a key element in a monitoring system is a clear initial statement of the project targets and of their rationale, so that the actual results can be readily compared with them and the implications of this comparison can be immediately perceived. The hope now is that the development and spread of the philosophy of evaluation throughout the Bank will lead to recognition of the need for greater clarity at the appraisal stage concerning the Bank’s decisions and participation. This means a clearer record of the assumptions and hypotheses underlying the forecasts and decisions of the Bank; more information on changes in a project introduced by the Bank and their acceptability to the borrower; and more details on the work put into the project by the Bank staff. Clearer treatment at this stage of the risks foreseen would also make feasible an excellent but hitherto unrealistic suggestion, namely, to carry out special evaluations of projects which turned out to be much more successful than had been expected. Thus, appraisal now does have something of principle to learn from evaluation.