Some common issues and challenges apply to many PIMAs. Chapter 4 discusses the main common issues, whereas chapters 5 through 8 discuss the detailed assessment of the different institutions.
How Are Institutional Design and Effectiveness Assessed in Practice?
PIMAs assess institutional design as well as effectiveness. The analysis of institutional design looks at the formal public investment management system, including legislation and regulations, to see whether its design is in line with international good practices. The institutional design assessment describes the potential effect of the current framework, provided that it is fully applied. When assessing effectiveness, the focus is on how well the system works in practice. Are the implementing agencies fully compliant with the different rules and procedures, and does the formal framework have the intended effects on project planning, resource allocation, and project implementation?
The PIMA framework can be described in a generic theory of change or intervention logic (Figure 4.1). This theory of change spells out the main elements in the logic underlying the different PIMA dimensions. The assumption is that clearly defined objectives, well-defined procedures, and effective communication mechanisms (key elements of institutional design) contribute to the successful completion of planned activities. These activities should result in the production of planned outputs and the realization of expected outcomes and effects (effectiveness). The quality of the process and the results will depend on the important cross-cutting features, including the legal framework, policies, information systems, and staff capacities, as well as on external factors.
Political considerations and decisions may have important effects at different stages of the result chain. Even the best legal and regulatory frameworks will rarely produce the expected results if they are routinely circumvented in the political process. Many PIMA institutions, including Institution 4 (Appraisal), emphasize the importance of independent review and transparency. This measure will reduce the risks of undue political influence. PIMA scores for effectiveness should reflect the actual outcomes of relevant processes, including the results of political interventions. Broader governance challenges can be addressed under the IMF’s new framework for enhanced engagement on governance (IMF 2019).
Similar theories of change can be specified for each dimension in the PIMA framework. Figure 4.2 provides an example of a theory of change for Dimension 4.a: Are major capital projects subject to rigorous technical, economic, and financial analysis?
The PIMA questionnaire largely focuses on the institutional design and highlights selected elements in the underlying theory of change. For Dimension 4.a, the questionnaire’s definition of a high score is as follows: Major projects are systematically subject to rigorous technical, economic, and financial analysis, and selected results of this analysis are published or undergo independent external review. This definition mainly relates to procedures and systems for project submission and appraisal, and to some extent to the publication of appraisal results (bolded text in Figure 4.2). Some parts of the theory of change for one dimension may be dependent on other dimensions. For Dimension 4.a, the outcome of the appraisal process (selection of high-value projects) is covered by Institution 10, and the realization of project benefits (the intended effect) is influenced by several PIMA institutions.
The assessment of effectiveness must be based on the analysis of the results that are achieved and will often go beyond the specific questions in the PIMA questionnaire. Effectiveness is related to, but not synonymous with, compliance. Compliance with formal rules should contribute to ensuring that the planned outputs and outcomes are realized. Weak compliance may be an important factor when explaining weak effectiveness, but compliance will often be insufficient to ensure effectiveness. The analysis of effectiveness should therefore not be limited to the verification of compliance. This handbook discusses relevant indicators for the assessment of effectiveness under each of the PIMA dimensions.
Assessments are based on current legal frameworks and current practice. Design scores should be made on the basis of literal interpretation of the dimension criteria. For example, if existence of a law, regulation, or policy is a scoring criterion, full credit should be given if it has been approved by the parliament even if it has not yet been implemented. The absence of results from the new legislation so far should be covered under the effectiveness assessment. If a law is under consideration, this can be mentioned in the narrative but should not impact the score on institutional design.
This handbook mentions several government documents and datasets that will be useful for a PIMA assessment. The documents will often be most useful for assessing institutional design and are listed under this heading, while the suggested datasets are listed under the effectiveness assessment. However, both documents and datasets will generally contribute to an understanding of both institutional design and effectiveness.
What Data and Data Sources Are Typically Available?
Effectiveness assessments should be evidence based. For each PIMA dimension, this handbook suggests what might be useful datasets for the assessment of the dimension, in particular for the effectiveness assessment, and gives suggestions for how these datasets may be used. The suggested quantitative thresholds are indicative, and assessment teams must cross-check these against other indications of effectiveness. There are several different categories of data that may be useful in this regard:
Time series of budget data, for instance, aggregate capital spending by sectors.
Data related to specific projects, for instance, how estimates have developed over time.
Comparative data, for instance, maintenance allocations compared with capital stock.
Case studies, for instance, summaries of external audit reports of project delays and cost overruns.
Lack of specific data can be an indication of low effectiveness. If data are not available at the time of the PIMA, this is important information in itself. To ensure the effectiveness of a process or a system, it will generally be necessary for relevant government bodies to generate and analyze the performance data. If such data are not available, it is an indication that the performance is not being monitored and that effectiveness is limited. Similarly, if there is inadequate information about or documentation of institutional features, including legislation and regulations, this would lead to a low score on institutional design.
Time series should cover at least three years. Effectiveness assessments should be based on practices over three years, so they are not unduly influenced by spurious developments in one specific year. If data are available, a five-year perspective may add additional depth to the assessment, but this is not a requirement.
PIMA focuses on capital spending; the development budget should not be used as a proxy for capital spending unless there is no reasonable alternative. Capital spending or expenditure is commonly defined as spending to acquire a physical asset or to extend the usable life of a physical asset.1 Under accrual accounting, capital spending is capitalized in the balance sheet. Many countries have a development budget rather than capital budget. A development budget commonly includes current as well as capital spending. Most countries with development budgets also identify capital spending as projects either within the development budget or in the economic classification, and this should be the basis of the assessment whenever possible. Each PIMA report should clarify which components of the budget have been covered by the assessment.
How Are Externally Financed Projects Assessed?
A high level of external financing may contribute to the fragmentation of practices under some institutions. This applies to institutions 2 (Planning), 10 (Selection), and 12 (Funding). For these dimensions, the PIMA questionnaire includes specific questions on the treatment of externally financed projects, and these may affect the scores for both institutional design and effectiveness.
Practices for project development, appraisal, selection, and monitoring may also differ based on whether a project is externally financed or domestically financed. This applies to three dimensions under institutions 4 (Appraisal), 11 (Procurement), and 14 (Project Implementation). International financial institutions (IFIs) and development partners (DPs) often have their own rules for project preparation and approval, which must be applied to projects when they contribute to the financing. This is not explicitly reflected in the PIMA questionnaire. External financing should not affect the institutional design scores for these three institutions, which should reflect the institutional framework established by national laws and regulations. However, practices related to externally financed projects may in a few cases have impacts on effectiveness assessments.
Some IFIs, in particular the major development banks, have rigorous project methodologies. However, many other financial institutions and bilateral DPs have less rigorous approaches. External financing schemes are often more focused on establishing a financial mechanism than on the specific projects to be financed under this scheme. In these cases, the DPs’ appraisal of the financial mechanism will not constitute appraisal of the specific investment projects. In some cases, IFIs and DPs carry out appraisal of specific investment projects only after a financial mechanism has been approved. Project development and management by IFIs and DPs will focus on the priorities and preferences of the institution, which are not always fully aligned with the government’s priorities and preferences.
In some cases, the PIMA effectiveness assessment may be influenced by the financing source. This would require that the share of major projects implemented by IFIs or DPs that apply stringent project development and management criteria, consistent with government priorities, is large enough to reach the relevant thresholds for better practices.2 If this is the case, the PIMA should provide documentation of which share of major projects are subject to systematic, rigorous, and consistent development, appraisal, selection, and management, and how this reflects the government’s priorities and preferences. The share of different IFIs’ and DPs’ financing of the capital budget should be presented, together with a summary description of the project management practices that are applied by each major financing source. The specific thresholds that need to be met for a higher effectiveness assessment are discussed under the relevant institutions.
PIMA reports should specify the role of external financing of public investment and how this has been treated. If external financing arrangements have impacted any scores, this should be clearly specified and explained in the report. Issues related to assessments of projects with external financing are discussed in more detail under the relevant institutions and dimensions. Table 4.1 gives an overview of potential impacts.
Possible Effects of External Financing on PIMA Scores
Possible Effects of External Financing on PIMA Scores
PIMA Dimension | Possible Effect of External Financing | |
---|---|---|
2.a | Does the government prepare national and sectoral strategies for public investment? | A high score (institutional design or effectiveness) requires that externally financed projects are fully reflected in national and sectoral strategies. |
4.a | Are major capital projects subject to rigorous technical, economic, and financial analysis? | If there are many externally funded projects and these are subject to rigorous analysis, this may affect the overall share of major projects that meet the requirements of this dimension (effectiveness). |
10.a | Does the government undertake a central review of major project appraisals before deciding to include projects in the budget? | A high score (institutional design or effectiveness) requires that the central review process include externally financed projects. |
11. a | Is the procurement process for major capital projects open and transparent? | If there are many externally funded projects and these are procured through open and transparent processes, this may affect the overall share of major projects that meet the requirements of this dimension (effectiveness). |
12.c | Is external (donor) funding of capital projects fully integrated into the main government bank account structure? | The score (institutional design or effectiveness) depends on the degree of integration of external financing in the government bank accounts. |
14. a | Do ministries/agencies have effective project management arrangements in place? | If there are many externally funded projects and these are based on effective project management practices, this may affect the overall share of major projects that meet the requirements of this dimension (effectiveness). |
Possible Effects of External Financing on PIMA Scores
PIMA Dimension | Possible Effect of External Financing | |
---|---|---|
2.a | Does the government prepare national and sectoral strategies for public investment? | A high score (institutional design or effectiveness) requires that externally financed projects are fully reflected in national and sectoral strategies. |
4.a | Are major capital projects subject to rigorous technical, economic, and financial analysis? | If there are many externally funded projects and these are subject to rigorous analysis, this may affect the overall share of major projects that meet the requirements of this dimension (effectiveness). |
10.a | Does the government undertake a central review of major project appraisals before deciding to include projects in the budget? | A high score (institutional design or effectiveness) requires that the central review process include externally financed projects. |
11. a | Is the procurement process for major capital projects open and transparent? | If there are many externally funded projects and these are procured through open and transparent processes, this may affect the overall share of major projects that meet the requirements of this dimension (effectiveness). |
12.c | Is external (donor) funding of capital projects fully integrated into the main government bank account structure? | The score (institutional design or effectiveness) depends on the degree of integration of external financing in the government bank accounts. |
14. a | Do ministries/agencies have effective project management arrangements in place? | If there are many externally funded projects and these are based on effective project management practices, this may affect the overall share of major projects that meet the requirements of this dimension (effectiveness). |
How Do We Define Major Projects?
Many dimensions of the PIMA institutions focus on major capital projects. The definition of major projects varies across countries and the assessment should generally be based on the national definition. However, the assessment team should verify that this definition is reasonable and consistent with national practices.
The most common definition of a major project is in terms of total project costs. All projects above a certain threshold (for example, 100 million currency units) are defined as major. National rules determine the thresholds and the definition of total project costs (for example, investment costs or lifecycle costs).
It is common that projects that are particularly complex and entail high risks are also defined as major projects, even if total costs are lower than the general threshold. In some countries, there are lower thresholds for certain project types (for example, IT investments).
Some projects are defined as major projects for political reasons. They may be part of government main priorities, have important regional impacts, or be particularly visible to the public.
In some countries, all projects with external financing or projects procured as public-private partnerships are defined as major projects, even if total costs are below the general threshold. These projects will often entail high risks and be politically important.
The number of major projects and their share of the total public investment will vary between countries. In many countries the number of major projects under preparation and implementation range from 30 to 100. Their share of the total investment budget will often be in the range of 50–75 percent.
How Do We Apply the Indicative Scoring Thresholds?
The PIMA questionnaire uses general terms to refer to the share of projects being subject to specific practices for many institutions and dimensions. This handbook recommends interpreting these terms in the following manner:3
All refers to 90 percent or more (by value).
Most refers to 75 percent or more (by value).
Majority or many refers to 50 percent or more (by value).
Some refers to 25 percent or more (by value).
A few refers to less than 25 percent (by value).
Little or no refers to less than 10 percent (by value).
This handbook suggests thresholds that may be used in the PIMA assessments. The recommendations related to institutional design are largely based on qualitative thresholds. Recommendations related to effectiveness are also qualitative, but there are also suggestions for quantitative thresholds in Appendix II.
The thresholds for institutional design focus on the legal basis for relevant provisions. Legal requirements are embedded in law. Regulatory requirements are included in regulations, typically issued by the the cabinet, the council of ministers, or the president. Other formal requirements include ministerial regulations and guidelines, including from the Ministry of Finance or Ministry of Planning. Government documents with lower legal status than regulations will generally be covered in the assessment of effectiveness. However, in some cases, these documents may also affect institutional design. This is specified under the relevant institutions—for example, Dimension 4.b assesses the use of central appraisal methodology, which would not necessarily be defined in the regulatory framework.
The suggested thresholds for effectiveness are indicative and should not be used mechanistically. The PIMA assessment team must verify that the proposed thresholds are appropriate for the country being analyzed and adjust the assessment when this is justified. These thresholds are for guidance and should not be applied indiscriminately. In many cases, mission teams will need to compile data from different sources or estimate some of the parameters that are included in the thresholds (or both). The key underlying assumptions for such compilation and estimation should be specified in the report. Relevant considerations for effectiveness assessments are discussed under the different institutions. Examples of quantitative thresholds for effectiveness are summarized in Appendix II.
Some thresholds include two conditions, both of which should be met to achieve the relevant score. If one of the threshold conditions is not met, the score will be one step lower, even if the other threshold condition meets the benchmark for the higher score. For example, high effectiveness on Dimension 5.c requires that “the review process covers at least the largest public corporations (PCs) measured by assets or the PCs covering most of the total PC assets and a consolidated report is published.” If it is not published, then the consolidated report and effectiveness score will be medium, even if the coverage of the review process is consistent with the first condition for a high score.
Assessments should be based on the data for at least three years and shares should be based on the total value or cost of investments. The relevant threshold should be met in the majority of the years: at least two of three years when data cover a three-year period. Many indicators refer to shares of total investments and this should generally be interpreted as shares of the value of investments.
Institution scores are the simple average of dimension scores. A low dimension score has a numeric al value of 1, medium has 2, and high has 3. The color coding used in the PIMA report follows normal mathematical rounding rules. Scores of 1.00–1.49 are shown as red (low score), 1.50–2.49 as yellow (medium score), and 2.50–3.00 as green (high score).
What is the Scope of the PIMA?
The PIMA focuses on investments by the central government sector. Institutions 3 and 5 cover the coordination with other levels of government and public corporations, as well as interfaces with the private sector. Dimensions 2a and 7b ask about the coverage of PCs in planning and budget disclosure. The assessments under the other institutions, including the cross-cutting issues, will generally be based on central government practices.
In some countries, SNGs or PCs are major contributors to public investment, and the PIMA mission team may choose to expand the assessment to cover practices in these sectors, either generally or for selected institutions. If this is done, the report should clearly specify where the scope of the assessment has been expanded and how this has impacted the findings.
Current spending that contributes to the creation of a government asset, for instance, project monitoring by own staff, may be included in the capitalized value of the asset. This will depend on national accounting standards.
Thresholds are discussed in the next section of the handbook.
If the use of value to classify project shares gives skewed results—for instance, if there is one mega-project that dominates the results—the assessment team can choose to base the analysis on the 10 largest projects.