Chapter 8. Organizational Learning: The IMF’s Use of New Knowledge Generated by the IEO
- Moisés Schwartz, and Ray C. Rist
- Published Date:
- January 2017
How has the IEO contributed to learning at the IMF? What are the IEO’s channels to promote learning and the use of new knowledge spawned by evaluation? How has the IMF responded to this new knowledge?
Chapter 7 defined the learning organization as an ideal towards which organizations should strive to evolve, and defined organizational learning as the processes or activities of learning that take place in the organization. This chapter focuses on organizational learning within the IMF. The vehicles for potential organizational learning at the Fund are formal and informal self-evaluation as well as independent evaluation reports.1 Here we focus on the IMF’s use of new knowledge generated by IEO and the extent to which the Fund’s established methods to collect, assimilate and disseminate this new knowledge are effective and promote learning. We conclude that the IEO’s contribution to the Fund in terms of learning has been confined to organizational learning in as much as we believe the IMF has thus far not yet achieved the ideal of a learning organization. Notwithstanding, we see the organizational learning that takes place as a result of an IEO evaluation report as a vital step in bringing the Fund closer to this ideal. In an effort to assist the Fund to move further towards this ideal, we analyze whether the new knowledge afforded by independent evaluation can become a more integral part of the processes and procedures within the Fund.
IEO’s TOR give clear prominence to the role that IEO should play in the learning process of the Fund. As mentioned in Chapter 7, the TOR note that the IEO “. . . is intended to serve as a means to enhance the learning culture within the Fund [and] improve the institution’s ability to draw lessons from its experience and more quickly integrate improvements into its future work” (IMF, 2000b). Thus the TOR leave no doubt that the knowledge provided by the IEO should be integrated into the Fund’s activities.
In theory, individuals learn as agents of an organization. It stands to reason, therefore, that staff can acquire new knowledge from IEO evaluation reports. However, for this learning to become organizational learning, this individual knowledge must be transferred and stored in the memory of the organization. One way in which this occurs is by incorporating this knowledge in routines, rules, standard operating procedures, staff guidelines, manuals, and other organizational documents. The organization al memory then regulates the behavior of the organization and that of its members. What is the process by which the Fund absorbs the IEO’s contribution to knowledge? Is the new knowledge provided by independent evaluation finding its way into organizational learning? As the basis for examining these questions, let us briefly recall the three levels of organizational learning.
Three Levels of Organizational Learning
In Argyris and Schon’s terminology, as discussed in Chapter 7, IEO reports provide findings that one would expect to allow the Fund to engage in single-loop, double-loop, and deuterolearning. The reports (i) provide the analysis that addresses the gaps between desired and existing conditions (single-loop learning); (ii) set up the conditions that would allow the organization to learn how to change the existing assumptions and conditions within which single-loop learning operates (double-loop learning); and (iii) provide the analysis to determine whether the learning process itself is optimal (deuterolearning). Is the Fund actually absorbing and using the new knowledge generated by IEO in the most effective way?
Organizational Learning Through IEO Reports
IEO reports have covered a range of topics on the IMF’s key activities and, as such, are a potent mechanism to promote learning within the Fund. Learning in the Fund and the use of new knowledge generated by the IEO is an intricate process involving different players and circumstances. Through its reports, the IEO promotes dialogue, discussion, and learning both within and outside the Fund. In some instances, the IEO leads the way and promotes dialogue on a specific topic. In other instances, the IEO’s contribution adds to an existing discussion or provides impetus to a previously discussed topic. As a result, measuring the IEO’s specific contribution to learning and change can at times be difficult. Nonetheless, the fact that the IEO is a credible and independent entity has allowed its findings and recommendations to be seriously considered as part of such discussions, and some of its lessons have been internalized by the Fund.
Within this jigsaw puzzle of interactions, the IEO promotes learning and the use of new knowledge through three channels: stimulating debate within the Fund; stimulating debate outside the Fund, and thus serving as a facilitator for external feedback; and the established procedures to follow up on IEO reports.
(i) Stimulating debate within the IMF. The delivery of an IEO report creates, in its own right, some discussion. This takes place through formal and informal meetings among Executive Directors, management, and staff. The formal meetings require management, staff, and Board to take an official view about the report, and thus IEO reports motivate dialogue. In addition, several informal meetings take place. For instance, some IMF departments have found it useful to hold informal meetings and discussions on some IEO evaluation reports, and the IEO has on occasion been invited to make presentations or take part in such sessions. Moreover, individual reading by staff members produces introspective learning and self-examination, which is fundamental for organizational learning.
But even before the IEO engages formally in an evaluation report, independent evaluation stimulates debate and learning within the organization. This is because the IEO undertakes a thorough consultation with the Board, staff, and management on possible topics for evaluation. Furthermore, before an evaluation report is finalized, the IMF staff has an opportunity to comment on a draft. All of these are opportunities to engage in dialogue and learning.
(ii) Stimulating debate outside the IMF. Externally, IEO reports offer a vehicle to stimulate dialogue among country authorities, civil society, and the public at large. The IEO takes part in seminars and presents its findings to country authorities and other audiences. This external channel, while less structured than the internal formal conduit to discuss IEO reports at the IMF, has succeeded in promoting change, as some IEO findings have fed back into internal institutional responses and learning.
(iii) Following established procedures to follow up on IEO reports. The established follow-up process for IEO reports obliges management and staff to take action on Board-endorsed IEO recommendations and to periodically monitor the implementation of those actions. This implementation stage would be expected to be the main tool through which the IMF learns and uses the new knowledge generated by independent evaluation. This expectation comes from the fact that presence of well-established procedures would ensure that some action and response takes place.
Thus internal, external, formal, and informal channels play a role in fostering the IMF’s use of the knowledge produced by the IEO, and the interaction among these channels—along with other elements and conditions outside the control of the IEO, such as other Fund initiatives or external circumstances—affects the Fund’s ability to learn from its experience.
Nevertheless, the impact of independent evaluation on institutional learning, and on the use of new knowledge through the formal follow-up channel, depends heavily on the receptivity of management and staff. If these stakeholders are willing to accept and incorporate the new knowledge into the IMF’s policies and operations, the formal channel has a high probability of delivering significant change. However, if the evaluation findings are met with defensive attitudes and no receptivity, the follow-up process is likely to fail.
Stedman, in reviewing the implementation of IEO recommendations, states:
Our main conclusion is that the IMF has taken some action on the majority of the recommendations examined. . . . At the same time, issues remain with respect to the implementation of many recommendations. For instance, actions may have been taken to implement a recommendation but also failed over time to satisfy the objective set out; or an issue may have persisted despite the targeted steps taken to address it. And in a few cases, the IMF has taken no or minimal actions to follow up on a recommendation endorsed by the Board (Stedman, 2012: 117).
While we should acknowledge that Stedman’s analysis is based on recommendations made before 2011, it is our opinion that her findings are also representative of the status of implementation of IEO recommendations during the most recent years.
It is our belief that though the official follow-up process for IEO evaluations is the most structured and organized of the three channels through which the IMF learns and incorporates new knowledge through evaluation, it is also the least efficient of the three. The follow-up process is not “owned” by those required to implement it, namely, IMF management and staff, and it obliges management and staff to take action on something they do not believe is necessary or that they may prefer to address in their own way. As a result, even with all of the progress made to date on the follow-up process, current procedures may look good on paper but are not backed by the incentives that would be needed to ensure that evaluation findings are appropriately followed up.
Formal Procedures Versus “Ownership” of Independent Evaluation
Once an Executive Board discussion of an IEO report has taken place, IMF management is now expected to produce, within a reasonable timeframe, an implementation plan for IEO’s Board-endorsed recommendations, and the staff is expected to put those into action. A process for monitoring the progress made in implementing the agreed recommendations is also in place. These mechanisms for follow-up are an improvement over the past, but still have some way to go to be considered optimal.
There were no formal procedures for follow-up on IEO recommendations when the office was first created. Instead, mechanisms were put in place in 2006 after the first external evaluation of the IEO and were revised following the second external evaluation.
The first external evaluation of the IEO (Lissakers, Husain, and Woods, 2006) found little evidence that findings and recommendations of specific IEO reports were being systematically followed up by senior management and the Board. The panel found that the Board was surprisingly passive in the follow-up of IEO evaluations, leaving any action largely to management. The report suggested that the Board needed to take charge and engage more systematically on follow-up.
As a result of the Lissakers Report, in 2006 the Board established a follow-up mechanism for IEO recommendations, comprising:
(i) a forward-looking Management Implementation Plan (MIP) that outlines actions to be taken by management to address Board-endorsed recommendations and is to be transmitted to Directors soon after the Board discussion of the IEO report; and
(ii) an annual Periodic Monitoring Report (PMR), prepared by the Strategy, Policy, and Review Department, to report on the status of implementation since the last report and any actions pending from prior reports as well as to propose any remedial actions or substitutes in the event of difficulties in implementation.2
Having reviewed the follow-up process that was put in place as a result of the Lissakers Report, the second external evaluation of the IEO (Ocampo, Pickford, and Rustomjee, 2013) concluded that it lacked strong ownership by the Board; involved a conflict of interest for management; was not well suited for responding to broader, more substantive recommendations from the IEO; and had become very bureaucratic. The report also picked up on evidence cited in the IEO’s ten-year review of independent evaluation at the IMF that the lag in producing MIPs had greatly increased over time and was by then nowhere near the “soon after” originally contemplated by Directors.
Hence the Ocampo Report called for a revamped follow-up process. In response, the Executive Board agreed that MIPs must be prepared no later than six months following the Board discussion of an IEO evaluation, and—to avoid staff ‘s conflict of interest in monitoring the implementation of IEO’s Board-endorsed recommendations—transferred the responsibility for preparing the PMR to the Office of Internal Audit. In addition, the Board Evaluation Committee has discussed the need for more careful attention to the monitoring phase of the follow–up process, in particular whether the underlying concerns raised by IEO evaluations are being addressed. In approving the MIP for the IEO report on Self-Evaluation at the IMF, the Evaluation Committee asked for a report within one year on the progress being made in implementing specified actions.
The procedural amendments made over the years have clarified the steps that take place once an IEO report is produced and certainly have improved the follow-up process. However, they do not address the lack of management and Board “ownership” of the required and agreed changes, and it remains to be seen whether issues raised by IEO evaluations will be better addressed. Moreover, the Executive Board, with the regular rotation of many Directors, faces challenges in consistently overseeing follow up, and thus leaves management with the important responsibility to “own” the process. Consequently, a clear willingness and decisive involvement by management will be required to move the process forward. Like both the Lissakers and the Ocampo panels, we conclude that management holds the key for the IMF to benefit more from the presence of the IEO.
Ideally, those whose work is evaluated should have the responsibility for deciding what to do about the findings and recommendations of an evaluation report and for implementing these decisions. As such, independent evaluation should lead to active engagement between those that evaluate and those that are evaluated. Unfortunately, this type of positive engagement between evaluator and evaluee seldom takes place.
It is our experience that when IEO findings are aligned with the institution’s operational or policy priorities or concerns, there seems to be common ground to move forward using the new knowledge afforded by the IEO. While progress may be slow, there is a sense of shared objectives and ownership in learning from independent evaluation that can propel the follow-up process.
But when IEO findings are not in keeping with the Fund’s priorities, it is our experience that IEO messages are sidelined, there may be pushback, and, therefore, the prospects for incorporating new knowledge into the Fund’s operations may be limited. In this case, the established procedures for follow-up on IEO recommendations can tend to be bureaucratic, take a long time, and result in deviating from the IEO and the Board’s intended purpose. Thus, lack of ownership obstructs the possibility of using new knowledge gained through independent evaluation to promote change.
What is the alternative? How can the IEO promote a dialogue of openness and learning? The answer to these questions has to do with the culture of the organization, a topic addressed in the next chapter.
For a broader discussion and assessment of self-evaluation at the IMF, see IEO (2015).
As noted in Abrams and Lamdany (2012), the Lissakers Report also recommended, and Directors discussed, establishing a system for the effectiveness of actions undertaken in response to Board-endorsed recommendations. At that time the Board decided to postpone such a mechanism until after more experience had been gained from the MIP/PMR experience, but to date such a mechanism has not been put in place.