Book
Share
Chapter

6 Performance Auditing and Performance Budgeting

Author(s):
Marc Robinson
Published Date:
October 2007
Share
  • ShareShare
Show Summary Details
Author(s)
David Shand

Performance auditing is now a well-established part of the work of national audit institutions in many OECD and other countries. As such it is part of the overall approach to performance management in the public sector of those countries. Its relationship to performance budgeting, as is discussed below, varies between countries, based on the scope and nature of the both the performance auditing function and performance budgeting. In some cases the link between performance auditing and performance budgeting is clear; in others it is less clear.

This chapter first sets the scene by examining a possible taxonomy of performance auditing. It then discusses the historical development of performance auditing, and the possible value added of performance auditing, its impact on the traditional staffing and independence of audit institutions, and then develops links with performance budgeting. Nine countries are used as the basis for this analysis, with a description of the performance auditing approach of each country being set out in an appendix. The analysis is based both on desk research and on discussions with audit officials in most of the nine countries surveyed.

Nature and boundaries of performance auditing

The International Organization of Supreme Audit Institutions (INTOSAI)1 defines performance auditing as being concerned with the audit of economy, efficiency, and effectiveness, and embracing:

  • audit of the economy of administrative policies in accordance with sound administrative principles and practices, and management policies

  • audit of the efficiency of utilization of human, financial, and other resources, including examination of information systems, performance measures, and monitoring arrangements, and procedures followed by audited entities for remedying identified deficiencies

  • audit of the effectiveness of performance in relation to the achievement of the objectives of the audited entity, and the audit of the actual impact of the activities compared with the intended impact (INTOSAI, 1992, Basic Principle 1.0.40).

The broad nature of the work covered by this definition reflects the fact that there is no unique concept of performance auditing. It can be regarded as a product line rather than a single product (see Barzelay, 1996a, p. 22) and as such it is difficult to generalize about either its form or impact. A major symposium on performance auditing convened by the OECD in 1995 noted then a substantial variety of different approaches between countries in terms of origin, scope, objectives, and processes (OECD, 1996a).

It can be seen from this definition and the individual country practices outlined in the appendix at the end of this chapter, that a taxonomy of performance auditing can be developed based on two major issues:

  1. The scope of the performance auditing mandate, that is, whether it covers only economy and efficiency or also extends to effectiveness.2

  2. The form of the performance audit. The audit may be a substantive review of the level of performance achieved in effect using results standards or targets, or it may be a systemic review—of the organizational management systems insofar as they will impact on performance, which may be described as based on process standards. Both of these types of audits may include evaluating the adequacy of systems to measure performance, which in a number of countries extends to the review and sometimes formal attestation of reported performance indicators through a short form audit report expressing a formal opinion on the appropriateness and reliability of the indicators.

The performance audit may also cover both approaches, and mixed arrangements may also exist. For example, the comprehensive auditing approach developed in the 1970s by the Canadian Comprehensive Auditing Foundation (CCAF) and implemented by the Canadian Auditor-General’s office involves substantive studies of (economy and) efficiency as well as reviewing the adequacy of systems to measure and promote efficiency but focuses only on the review of systems to measure effectiveness, that is, there is no substantive measurement of effectiveness.

It is perhaps useful to clarify this “product range” by looking at the way the public sector production chain is often portrayed, as set out similarly in Chapter 3:

Inputs → Processes → Outputs → Outcomes

That is, inputs are converted through management processes (which comprise staffing and assets and all information and control systems) into outputs, which in turn contribute to achieving the desired outcomes of the expenditure. A systemic performance audit evaluates the management processes and systems. A substantive efficiency audit evaluates whether a program or activity is being efficiently operated and a substantive effectiveness audit evaluates whether the outputs of a program or activity are achieving their intended outcomes. Auditing of performance indicators is a thread which potentially runs through all other aspects of performance audit. Performance indicators (or lack of them) are part of the management control systems evaluated in a systemic audit. Output indicators are also relevant to measuring performance under a substantive efficiency audit and outcome indicators assist in measuring performance under a substantive effectiveness audit.

Some relationship can be seen with the traditional financial and compliance (or regularity, as termed by INTOSAI) audit undertaken by national audit institutions which review compliance with applicable financial laws, regulations, and controls as well as providing a formal audit opinion on the reliability or fairness of presentation of the annual financial statements. For example, financial and compliance audits through their review of the internal control system may identify weaknesses in controls which have a material impact on performance. This is likely to reflect operational efficiency issues—for example, the adequacy of asset management, in particular inventory or plant assets, as discussed below in the section on the development of performance auditing. As Barzelay (1996b) notes, “efficiency has been one of the core values in all but the most legalistic of audit bodies.”

Performance auditing is almost invariably ex post, although there are at least two examples of an ex-ante approach. The US Government Accountability Office (GAO) may review programs in advance of their implementation and also reviews or scores the agency performance plans developed by the executive branch. The German Federal Court of Audit has a specific role to provide independent advice to assist the appropriations committee of parliament in its review of the proposed government budget.3 As part of this role the findings of its ex-post performance audits are used and special ex-ante performance studies may also be undertaken.

Development of performance auditing

In most OECD countries it appears that early developments in performance auditing have been separate from the development of other aspects of performance management in the public sector. However, Barzelay (1996a) notes that performance auditing is more developed in those countries which have more vigorously pursued what is described as the “new public management” or (NPM)4—with the exception of France and Germany, which have a long-established performance audit function but which have been relatively slow in adopting NPM concepts.

Performance issues as a legitimate audit concern first emerged in the 1950s.5 The development of program evaluation in the US included its development in the GAO as effectiveness auditing in the late 1960 and 1970s, reflecting the view of the then head of the GAO that Congress needed such information to exercise appropriate oversight of the executive branch. However, performance auditing appears to have been stimulated in some countries from the early 1970s through the introduction of Planning, Programming, and Budgeting Systems (PPBSs) in their various forms.

From the 1960s new legislation formally expanded the mandate of national audit institutions in a range of countries including Australia (1979) and the UK (1983). These moves appear to reflect a number of factors including the wishes of parliaments and government concerns to improve public sector performance, and in some cases formalized moves already undertaken by audit institutions into examining economy and efficiency issues, arising out of their existing financial and compliance audits. For example, it is a relatively small step from the role of a traditional financial and compliance audit review of the adequacy of inventory management systems to produce reliable information about the level of inventory held (which is reported in the financial statements) and to ensure adequate controls over inventory, to a performance audit using this information to evaluate whether the right inventory is held at optimal levels. In essence some external audit institutions actively sought to expand their work into performance auditing and successfully sought political support for this, these performance auditing developments thus being, at least in part, supply driven. Some of the political support for these developments may also have reflected the view that given the application of market type mechanisms in some parts of the public sector, which provided a surrogate for performance measurement, the need for performance measurement and review in the remainder of the public sector becomes of greater importance.

The partly supply-driven nature of some country’s moves to performance auditing is also suggested by the fact that in general, formal public sector management reform programs adopted by OECD governments and reflecting the NPM approach6 made little or no explicit reference to the role of external audit in the overall performance management system.

The moves by many OECD governments from the late 1980s and early 1990s to develop more formal systems of performance measurement and thus to better measure and report on performance, to a large extent associated with NPM, also facilitated the moves to performance auditing. As discussed in Chapter 1, NPM has aimed to achieve a greater focus on performance through providing greater managerial flexibility, in return for which managers are required to achieve agreed results. This development has seen a greater focus on measuring performance through performance indicators and more in depth program evaluation, with performance measurement becoming part of ongoing management systems. Changes in country budgeting systems have reflected this approach—particularly through less-detailed appropriation controls and the publication of performance information in the budget documents.7

However, performance auditing has not necessarily been perceived as an important or integral part of this development. Perhaps reflecting this, a 2005 OECD questionnaire on the use of performance information in the budget process, prepared for the annual meeting of its senior budget officials (SBO) network, did not raise the issue of the role of performance audits (either substantive or systemic), although an earlier OECD (2002) review noted that in over half of OECD countries, performance information in budget documents was subject to some form of audit.

The more recent focus on developing (and using) performance information has seen the development of an external audit focus on performance information (New Zealand, Brazil, Australia, the UK, Sweden, and France), although not necessarily extending as far as formal audit attestation of performance indicators, based on the view that claims about performance should be subject to some independent external review, particularly in view of the fact that most performance information is produced by the supplier of the services. In general, however, the development of performance auditing appears to have proceeded along a separate path from performance budgeting, although one exception is Sweden, as will be discussed later.

Substantive performance audits

Substantive performance audits are generally at organizational, program, or activity level, rather than whole of government. They examine performance (and usually by implication the performance of management) on an ex-post basis by reference to some type of results or process standards. They include:

  • review of program effectiveness, that is, the extent to which programs are achieving their objectives

  • review of government policy (but the audit mandate extends this broadly in only a few jurisdictions, which have a clear separation of powers between the parliament and the executive branch. Thus in Australia and the UK, which have traditions of strong executive government and where parliament is less at arm’s length from the executive, the performance audit mandate specifically excludes any review of government policies. By contrast, the broader performance audit mandate of the US GAO reflects the role of Congress as an equal partner in government with and independent of the executive branch)

  • review of operational efficiency, that is, whether outputs are being produced a lowest cost, consistent with quality requirements.

It can be noted that the auditor is not normally providing an overall opinion on the achievement of economy, efficiency, and effectiveness of the audited entity (INTOSAI Field Auditing Standard 4.0.23).

Studies of program effectiveness raise the issue of the boundaries between performance (effectiveness) audits and program evaluations. This also raises the question of whether effectiveness audits are an appropriate role for auditors, even recognizing that audit institutions have changed their personnel practices to acquire knowledge and skills for such auditing which go beyond those of a financial auditor. Pollitt and Summa (1997a) note the similarities between substantive performance auditing and external program evaluation as being a similar “toolkit” or methodology but note differences in terms of authority and processes, which are often reflected in different topic selection, relationship with the auditee, and “tone” and form of reporting.

A number of countries reviewed have a clear and significant effectiveness auditing role (Australia, Brazil, Netherlands, the UK, and the US) which exists in parallel with program evaluation carried out within the government. There are equally examples of countries where program evaluation is well developed but with no corresponding performance auditing role.8

Interestingly, INTOSAI has established separate working parties on performance auditing and program evaluation, but the basis of this distinction is not clear. As discussed later it seems clear that the methodologies of effectiveness auditing and program evaluation are similar, with the difference being mainly in concept of independent external review and the public reporting of evaluation results inherent in the audit function. However even these differences may not be a distinguishing feature of effectiveness audits. For example, all program evaluations in Chile are required to be carried out by independent “experts” and all evaluations must be published.

Reviews of performance indicators

The INTOSAI standards refer specifically to national audit institutions working toward “improving techniques for auditing the validity of performance measures” (Basic Principles 1.0.6(j) and 1.0.45) and comment that “the expanding audit role of auditors will require them to improve and develop techniques and methodologies to assess whether reasonable and valid performance measures are used by the audited entity” (Basic Principle 1.0.46).

External auditors appear to perceive the design and operation of information systems as an area in which they may have particular expertise, drawing from the evaluation of financial management systems as part of their financial audit work.

The scope of this type of systems review may vary—it may include review of the accuracy and validity of performance measures. This review may go so far as formal attestation as to whether the information is relevant and reliable, or be by exception—that is, reports on cases where the measures are perceived as inaccurate or invalid/misleading. In principle this role may also include reviewing the targets against which performance is assessed, although this is not common.9

In principle this may also extend to reviewing whether the performance indicators are appropriately used in decision-making—so as to actually improve performance. This has been a component of some substantive performance audits, as has also been the question of whether there is adequate performance focus in program implementation.10 If this addresses the use of performance information by elected officials, it may be pushing the boundaries of performance auditing into the political arena (and potentially compromise the auditor’s perceived political impartiality) in that it may question the appropriateness of elected officials making decisions on the basis of “political rationality,” rather than using supposedly “objective” performance information. It also raises the issue of how, given the inherent limitations of many performance indicators, it may be possible to determine whether information has been used “appropriately.”

The same issue arises in respect to the use of the other major types of performance information (in particular, program evaluation information and substantive performance audits) by decision-makers, including elected officials. However, this appears to have received less attention than the use of performance indicators.

The national audit institutions of New Zealand, the Netherlands, Brazil, and France have a formal role in reviewing of the validity and reliability of reported performance indicators, although the nature of the audit report may vary. For example, in the Netherlands and New Zealand there is a formal audit opinion on the performance information. Other audit institutions have a less formal review role (Sweden and the UK) but nevertheless an objective to see that reported performance information is at best not misleading.

There is also considerable work by external audit institutions in promoting good practice in performance measurement in the public sector (Australia, the Netherlands, New Zealand, Sweden, and the UK), in some cases through the audit institution itself developing guidance on good performance measurement or reviewing the level of adherence to other guidance issued on performance measurement.

Systemic performance audits

Pollitt and Summa (1997b) note that in five national audit institutions surveyed by them as part of a comparative study of performance auditing in Europe, the focus is on the adequacy of management systems—staffing, organizational structures, information systems, and other components of internal control, to enable the organization to operate effectively and efficiently, rather than on substantive performance measurement.

These systems are judged against generally accepted good practices, as developed by the management consultancy and other related professions. Individual organizations’ performance is increasingly benchmarked against that of other relevant organizations using a range of business excellence models. Such work may make use of the balanced score card approach under which the adequacy of measures to maintain the capacity and responsiveness of the organization are also reviewed, along with the perceived service delivery performance of the organization.

Value added of performance auditing

Based on presentations to the 1995 OECD symposium and on their own publications describing their work, external auditors appear to perceive two main objectives for their performance auditing work:

  1. enhancing the accountability of executive government to parliament and the public

  2. improving the performance of the public sector.

These goals of performance improvement and performance accountability are not mutually exclusive. They are related but not identical. Barzelay (1996a) suggests they may not even be synergistic and notes that, if they were, the distinction would not be important. However, it seems from the experiences of a number of countries that there may be some difference in emphasis between these two objectives, reflecting different “cultures” in different audit institutions ranging from a collaborative relationship between the auditor and the auditee (reflecting a focus on performance improvement) to a more adversarial one (reflecting a focus on accountability), where critical audit reports may evoke a defensive or contested response. This may also reflect in part the two main different traditions of institutional arrangements for external audit—the Court of Accounts tradition where the external auditor is legally a court, passing judgments on the actions and accounts of officials (represented by France and Brazil) and the Auditor-General tradition where the external auditor has no such status, but also has a special reporting relationship with the legislature (represented by Australia, Canada, New Zealand, Sweden, and the UK). There may be of necessity be a more “judgmental” approach where the audit institution has the status of a court.

On the objective of performance improvement it can be noted that in a number of cases the identification of budget savings is specifically identified as an objective of this work. For example, the UK National Audit Office has its own performance objective of identifying potential savings nine times the overall cost of its operations. The US GAO has a similar approach in identifying potential savings or value added from its reports. The Brazilian Court of Accounts (TCU) reports annually on the percentage of its performance auditing recommendations adopted by the executive and the likely budget savings. And the UK Audit Commission (which audits local government entities) has stated that its prime objective is to assist local governments in achieving greater value for money. However, such an approach may lead to difficult debates about the level of such savings and whether they are attributable to audit work or were known anyway, which has led some countries (Australia) to eschew such an approach. Determining potential savings also requires detailed costing of all audit recommendations, which may be difficult.

The value added by external auditors is perceived by them to be independence and the professional audit standards which are required to be applied to the work—competence, due care, standards of planning, supervision and review, evidence and reporting, including consultation. The external public sector auditor also operates under a Code of Ethics and professional auditing standards promulgated by INTOSAI.11 These codify such issues as required integrity, independence, objectivity, impartiality, professional secrecy, and competence. Arguably, such standards could also be developed by an “evaluation profession,” which is emerging in some countries, assisted by the work of national and regional evaluation organizations which seek to promote high-quality, professional evaluation. The Canadian, American, European, and Australasian evaluation societies are major players in this area, and some have active involvement of their external audit institutions. The need for independent review may also be reinforced by the self-evaluation concept which is part of NPM, and as mentioned above by the fact that most performance information is prepared by the government organization which is managing the programs or activities.

Given the scope of the audit and the lack of a single criterion for arriving at conclusions, the performance audit report necessarily differs from the “short form” audit report on the financial statements. As INTOSAI standards state, “performance audit is wide ranging in nature and is more open to judgment and interpretation. As a consequence performance audit reports are varied and contain more discussion and reasoned argument” (Reporting Standard 4.0.21). This gives rise to two other issues which may impact on the auditor’s independence—the making of recommendations for improvement and the impact on the relationship with auditees.

On the first issue, the performance improvement focus may include directly advising the auditee on how to improve performance measurement and management, as well as detailed audit recommendations on how performance might be improved. A traditional view of audit might perceive this as compromising the auditor’s independence, in that the auditor is subsequently auditing systems which he/she has been involved in establishing. However, this does not accord with a more modern view of auditing. INTOSAI reporting standards state, inter alia:

  • “Performance audits should not concentrate solely on criticism of the past but should be constructive. The auditors’ conclusions and recommendations are an important aspect of the audit and, where appropriate, are written as a guide for action” (Reporting Standard 4.0.25).

  • “Fairness implies the presentation of weaknesses or critical findings in such a way as to encourage correction, and to improve systems and guidance within the audited entity” (Reporting Standard 4.0.24).

Thus INTOSAI auditing standards now implicitly accept the importance of the external auditor adding value by contributing to performance improvement, and the making of recommendations for improvements is not perceived as compromising audit independence.12

In terms of relationships with auditees, both the Lima Declaration and INTOSAI standards refer to giving “due consideration to the points of view of the audited organizations in the audit findings and agreeing the facts with the audited entity to ensure they are complete, accurate and fairly presented in the audit report.”13 Thus there is a willingness by auditors to engage in a dialogue with the auditee on performance audit findings, which does not involve negotiating the conclusions of the audit. This is generally seen as enhancing the quality and likely impact of the audit, rather than compromising audit independence. Indeed it is the practice of many auditors to include auditee responses or comments in the audit report.

There is some debate, particularly among auditees, about the adequacy of audit skills to undertake substantive efficiency and, more particularly, effectiveness audits. However, it can be noted that to ensure a high-quality audit, performance audit staffing now routinely includes persons of varying professional backgrounds (for example, economics, management, engineering, social sciences), not just those with financial and compliance audit training and skills.14 Even if such diverse professional skills are not part of the audit institution’s staffing, they may be added to performance audit teams through consultancy arrangements. There is less debate on the adequacy of audit skills to undertake systemic performance audits, given traditional audit expertise in management and information systems.

Linking performance audits with the budget

A key issue is the link between performance audit activity and the budgetary process. To what extent do substantive and systemic performance audits feed into the budget dialogue, including any audit of performance indicators? In most countries there has been no direct study of the extent to which substantive performance audit studies have influenced budgetary decisions. In Australia an internal study by the Australian National Audit Office (ANAO) notes the difficulty of determining a causal relationship between performance audits and budget decisions. Nevertheless it documents a number of cases where performance audit reports have featured prominently in budget decisions, which appear to reflect performance audits focused on significant and topical issues with costed and specific recommendations for performance improvement. (See Box 6.1.)

Box 6.1.ANAO review

In 2005, the ANAO reviewed the impact of its performance audits and identified seven significant examples over the period 1999-2005 that demonstrated a direct link to budgetary initiatives. These included additional funding provided to the Taxation Office to realize efficiency gains and additional funding to strengthen quarantine operations, both following a detailed performance audit report. Other examples included changes in procedures for managing detention center contracts and the management of drought assistance programs.

The substantive performance audits of the UK Audit Commission, with their focus on identifying budgetary savings also appear to have a close link with the budgetary process.

It is clear that the link between performance indicators and budgetary allocations is necessarily an indirect one15 in which performance information informs the budget process but does not control it through some mechanistic or rational link. The original New Zealand budget model of output-based appropriations suggested a direct linkage with outputs, but neither funding during the year nor the dialogue on the following year’s budget appear to have been significantly influenced by the actual outputs produced.16

A key issue therefore is under what conditions public performance audit reports will feed into budget decision-making. The act of publication will in itself increase public and political pressure to implement performance audit recommendations. In addition, there are a number of other institutional factors which may increase the likelihood of recommendations being implemented, including:

  • the extent to which there are formal requirements for a follow-up or response by the executive to the performance audit report. For example, Westminster-system countries (Australia, Canada, New Zealand, and the UK) typically have a system of Treasury minutes under which the Ministry of Finance (MoF) or equivalent is required to make a formal response to all audit reports, with this response being tabled in the parliament. In any case there is a well established parliamentary committee structure charged with following up audit reports. Even in the Court of Accounts tradition of auditing, although there is no direct constitutional relationship between the Court and the parliament, parliament is able to pursue matters raised in audit reports

  • the extent to which cost savings or value gains are quantified in the audit report, and particularly whether they have been discussed and agreed with the organization being audited

  • the extent to which parliament has committee structures in place to facilitate a linkage into budget decision-making. In many countries the parliamentary committee responsible for the review and follow-up of audit reports is the public accounts committee (or its equivalent), with no direct role in reviewing the government’s proposed budget. In others countries (such as Germany), the roles of audit follow-up and budget scrutiny are combined

  • the extent to which parliaments, which have budget-making powers independent of the executive branch, are interested in issues of performance when reviewing budget proposals. It appears that neither the Brazilian nor the US Congresses evince much interest in performance information, including performance audit reports.

The work of audit institutions in seeking to improve the quality of reported performance indicators which feed into the budget process, as discussed above, is another dimension of the linkage between performance auditing and performance budgeting. This appears particularly strong in Sweden, where the development of performance auditing focused upon the review of performance information which feeds into the budget process played an important role in developing and implementing the new results-based budgeting system developed from 1993 (see Sandberg and Larsson, 1996). This may also reflect the former status of the Swedish National Audit Office as an autonomous part of the MoF, with explicit responsibilities to assist in improving financial management. In any case it is one example of an audit institution playing an important role in the development of performance budgeting. It is also strong in the Netherlands and Australia, whose national audit institutions have played a significant role in reviewing the quality of performance information included in budget documents, in the former case involving a formal audit opinion on the performance information.

The Brazilian Court of Accounts has a special role in monitoring the physical and financial execution of public works projects and reporting to Congress to assist in decisions on budget allocations. This has a performance (efficiency) dimension through comparing physical progress to expenditures and budget.

A number of external audit institutions have carried out broad reviews of the operation of performance management initiatives, including performance budgeting. For example, the Canada Auditor-General’s Office has had a long tradition of reviewing management reforms initiatives and seeking to support reforms to achieve a well functioning performance management system (see appendix). The US GAO has carried out reviews of such major performance management initiatives as the Government Performance and Results Act (GPRA) which is intended to bring about greater linkage between performance information and Congressional budget allocations, and of the Program Assessment and Review Tool (PART) used by the executive branch in formulating its budget proposals.

Conclusion

Performance auditing exhibits substantial differences between countries in terms of its origins, objectives, scope, and processes. While there has been significant discussion of performance auditing in professional and academic literature, there has been little discussion or analysis of its relationship with performance budgeting. However, in general it appears that performance auditing and performance budgeting have developed along separate paths, but recently have been brought more closely together through increasing emphasis on published performance indicators.

Clearly, rigorous and professional performance audits, on topical issues, whether substantive or systemic, can inform and improve a performance budgeting system. This appears to be the case in Sweden and also in Australia and the Netherlands. The linkage does not appear as strong in New Zealand, with its narrower focus on output measures in the formal budget system. In France it is too early to tell, given the very recent development of a performance budgeting approach (see Chapters 11 and 20). In Canada and the UK we do not as yet have enough evidence on the operation of performance budgeting and the impact of performance audit. In the US the position is complicated by the failure of Congress to embrace the performance budgeting concepts set out in its own legislation (the GPRA) which have been largely adopted by the executive branch. A similar position applies in Brazil, where the Congress has independent budget powers but has yet to embrace performance budgeting concepts, even though they are quite well developed in the executive branch.

Appendix: country summaries

Australia

The Australian National Audit Office has had a statutory mandate to carry out performance auditing since 1979, although at the earlier stages there was an emphasis on efficiency issues. It may review the economy, efficiency, and effectiveness of any public sector organization, program, or function—although, as in the case of the UK National Audit Office discussed below, it may not comment on the merits of government policy. Its performance audits will invariably include an assessment of the performance management system governing the organization, program, or function.

Reforms in the Australian budgetary process over the past 20 years have emphasized the importance of a performance culture supported by increased budgetary devolution and accountability. Performance information is presented in two main accountability documents: the portfolio budget statement (PBS) prepared by each department, and the annual reports of each department and agency. There is no formal requirement for this information to be audited. However, for both documents the ANAO has issued better practice guidance17 to assist in improving performance measurement, in the former case jointly with the Department of Finance and Administration (ANAO, 2002; ANAO and Department of Finance and Administration, 2004). Portfolio budget statements reflect internal managemen and at the same time provide parliament with information to explain the proposed funding in terms of outcomes (including whole of government outcomes) and output classes and outputs. This information has also been used in the past for market testing of government activities, to determine whether they might be better contracted out to the private sector. Recent moves have seen greater emphasis placed on formalizing outcome measures as part of the budgetary process.

The requirements for annual reports are issued by the Department of Prime Minister and Cabinet and generally reflect information contained in the PBS. The ANAO has also in the past prepared reports reviewing the quality of performance information (ANAO, 2001, 2004) which have focused on the organization’s performance framework, measurement and data quality, and quality of reporting.

Given the high profile of performance audit reports and the requirement for a formal response to them by the government, performance audits appear to have a significant impact on budget decisions. In an internal study the ANAO has identified a direct link between a number of its reports and budget decisions, including cases where it has identified the need for additional expenditures to improve performance.

It can be noted that in the State of Western Australia the Auditor-General has been required to provide a formal audit opinion on departmental and agency performance indicators in their annual reports since 1985 as well as to opine whether the organization has operated in an efficient manner. However, this mandate initially proved difficult to fulfill, given the evolving state of performance management and the perceived low value added of issuing large numbers of qualified audit reports. However, as performance management has developed to focus on a smaller number of key performance indicators (KPIs) and a closer linkage with the budget, moving from an output to an outcomes focus, the audit role has similarly developed. The Auditor-General reviews the appropriateness and quality of the performance indicators used in the budgetary process and the links between the indicators and budget decisions. In terms of the formal audit opinion, there has been a substantial reduction in the number of qualified audit reports on performance indicators in the past ten years.

Brazil

The Brazilian federal Court of Accounts (TCU) has a significant performance auditing role. First, it reviews the (four-volume) annual report of the President to Congress, which includes non-financial information as well as financial statements. Detailed performance indicators for each government program under the four-year National Plan (PPA) are primarily reviewed by the internal audit institution reporting to the MoF, the Federal Secretariat of Control (SFC), although the TCU may also review and comment on this work. The SFC operates its own database to monitor program objectives, expenditures, and results.

Second, the TCU carries out an extensive range of substantive effectiveness audits, in which it also makes detailed recommendations for improving performance. These focus on the priority social programs contained in the National Plan. The TCU’s annual report reviews the percentage of its recommendations adopted by the government and the estimated savings.

Finally the TCU has a special role to oversee the implementation of major public works projects. The TCU reports on the physical and financial execution of these projects to Congress, to assist in decisions on the allocation of funds to these projects.

Canada

The Auditor-General Act of 1977 requires the Auditor-General to report to parliament on cases where it is observed that “money has been spent without due regard to economy or efficiency, or satisfactory measures have not been established to measure and report on the effectiveness of programs, where such procedure could appropriately and reasonable be implemented.” This reflects the Canadian Comprehensive Auditing Foundation approach discussed earlier; there is substantive reporting on economy and efficiency, but not effectiveness, where the focus is on audit review of performance management systems. This performance auditing mandate followed the 1975 recommendations of an independent review committee.

Departments and agencies are required to monitor key aspects of the performance of their programs and operations as set out in key results commitments covering a three-year period, and to carry out appropriate evaluation and review. Central agencies have also regularly carried out major program and expenditure reviews. Performance management is the policy responsibility of the Treasury Board Secretariat, which prepares an annual report to parliament on the performance of departments and agencies, which also report performance in their annual reports to parliament. However, it does not appear that this reporting is against specific targets. Every department presents a Report on Plans and Priorities once the budget is passed, indicating how it intends to achieve the outcome results which have been set for it.

The Office of the Auditor-General (OAG) does not systematically audit the performance information presented to parliament, but may do so as part of a substantive performance audit and the review of a department or agency’s performance management systems. Auditor-General’s annual and special reports have commented generally on areas where performance management systems need improvement and cross-cutting issues and the OAG has a long history of proactive involvement in promoting improved performance management, as part of overall public sector management improvement. Examples are its 2000 report “Reporting Performance to Parliament: Progress too Slow” and its 2001 report on “Public Service Management Reform.” Earlier studies in the late 1980s and 1990s included “Constraints on Productive Management in the Public Sector” and “Attributes of Well-Performing Organizations.”

France

With the passing of the Loi Organique relative aux Lois des Finances (LOLF) in 2001, France has moved to restructure its budgetary system to one focusing on outcomes, with the first program budget being presented in 2004, and the system to be fully operational in the 2006 budget exercise (see Chapters by 11 and 20). The budget is broken down into some 47 missions and below that into around 150 programs and 500 sub-programs or activities. In return for increased budget and management flexibility program managers are required to produce agreed results, as set out in an annual performance plan (PAP) attached to the budget law, which contains main goals, performance indicators, expected results or targets, and expected costs. At year end actual results are reported in an annual performance report (RAP) attached to the budget review law. Overall some 1,500 performance indicators have been produced based on guidance produced by the Ministry of Finance, the parliament and the Cour des Comptes (Court of Accounts) for those who will produce, use, and verify the performance information. The indicators are required to be documented in such a way that they can be verified.

The Cour des Comptes has a broad performance auditing role, which includes substantive reviews of performance. Studies of the “good use of funds” have long been combined with studies of the “proper and legal use of funds.” Although these reports were not generally made public, a 1991 decision of the Conseil d’Etat empowered the Court to undertake and publish specific audit reports. With the new performance budgeting approach, its work in this area is expanding to include examination of the “coherence and reliability” of the information in the budget documents, both ex ante and ex post. It may also comment on significant variances between targets and results. Its work in this area is complemented with an interdepartmental program audit committee (CIAP), comprising Inspectors-General of Finance from ministries, which reviews the quality of the information and the analysis contained in the annual performance plans and annual performance reports.

The Netherlands

The Ministry of Finance is responsible for coordinating performance measurement and reporting, with individual ministries responsible for measuring the performance of their policies and management. From 2001 a system of outcomes-based budgeting has been introduced and performance information covering both efficiency and effectiveness is required by the Government Accounts Act to be included in the budget documents. In addition the annual report of each ministry and a consolidated annual report of the government focus on the question of the achievement of goals as set out in the budget and their costs. There is thus a focus on increasing the informational value of the two major accountability documents—the budget documents and the annual reports. The Government Accounts Act allows performance indicators to be developed from in-depth expenditure evaluations, as set out in a separate regulation on performance data and evaluation.

Ministers are required to present a statement on management control, which includes the adequacy of internal systems to convert the budget inputs to appropriate outputs and outcomes, and this is included in each ministry’s annual report.

The Court of Audit has a statutory mandate to review and report on efficiency and effectiveness, including reviews of performance measurement systems, administrative systems and program effectiveness, (but not questioning government policies) dating from 1976. It carried out program effectiveness audits and may also review those undertaken by ministries and agencies themselves. Under the Government Accounts Act it must express an opinion on the quality of non-financial information included in the government’s annual report, including information on outcomes and outputs, covering both their reliability and appropriateness/relevance. In doing this it would prefer to rely on the work of the internal audit directorate of each ministry as it does for the audit of financial information, but this internal role has not yet been developed. Overall the Court considers further work is needed within government on developing good performance information, particularly outcomes, and has indicated that it will actively participate in this work. It can be noted that the Court expressed reservations in a 1997 report about the decision-making relevance of performance indicators developed earlier under Ministry of Finance guidance.

New Zealand

The Office of the Auditor-General has a broad performance-auditing mandate which enables it to examine the extent to which a public entity is carrying out its activities effectively and efficiently and complying with its statutory obligations, whether any waste may have resulted and whether there may have been lack of probity. Fifteen performance audits were completed in 2004/05—a relatively small portion of the OAG work at less than 10 percent, but the OAG’s five-year strategy calls for an increase in this percentage and additional staff are being recruited to carry out performance audits.

As part of the financial audit, the OAG also expresses an audit opinion on the service performance (outputs) reports required from most public entities, which report actual performance covering quantity, quality, timing, and location for each class of output compared with targets, and form part of the financial statements of each public entity. The OAG’s work here is governed by its Auditing Standard 4, the Audit of Service Performance Reports and by the Institute of Chartered Accountants of New Zealand standards for reporting performance information. The OAG’s standard requires the auditor to form a judgment about the appropriateness of the performance measures, meaning that they must be relevant, complete, and understandable. The standard provides guidance on what tests might be applied to determine this. Reliability of the information is also to be tested through evaluating the systems and processes which produce the information. Where the auditor considers that the performance measures are “fundamentally misleading or senseless,” and where those performance measures are “significant” (guidance on determining significance is also provided) these concerns must be discussed with management. “Special circumstances” which are also set out in the standard must be taken into account in any decision to issue a qualified audit report on a performance report.

The OAG standard also requires the auditor to review draft performance targets and measures and the processes for establishing those measures in consultation with management before public consultation and formal approval of the measures, thus providing for “upstream” audit involvement.

The OAG has also been proactive in promoting improved reporting of public sector performance, and its January 2002 report on Reporting Public Sector Performance (second edition) aimed to stimulate discussion about how public entities set direction and measure and report performance, and thereby improve the level of accountability—going well beyond the statement of service performance into issues of outcomes. This guidance supplements other guidance provided by both the Treasury and the State Services Commission on statements of service performance and annual reports.

Sweden

Performance management in Sweden reflects its governmental system in which most public services are provided by autonomous agencies, subject to some general policy supervision by umbrella ministries. In return for their significant autonomy agencies are required to achieve results. The Ministry of Finance, together with the National Financial Management Authority (ESV) has overall responsibility for the development of performance measurement systems, with parliament and the government determining what is to be achieved or objectives within the 47 expenditure policy areas, ministries specifying what is to be measured and agencies determining the relevant performance measures. Performance measurement tends to focus on its use in performance improvement and resource allocation through the budget, as well an accountability mechanism. Performance monitoring and evaluation are closely linked with the budgetary process (ESV, 2003).

Agencies prepare an annual report which includes financial, performance, and other information—including how they have met the objectives set by government for their operations, which forms the basis of a performance dialogue between ministries and agencies. The government also prepares a consolidated annual report for the central government sector setting out objectives aimed at and results achieved, and agencies thereafter present their budget request which sets out their objectives and results achieved, which together with the annual reports form the basis of detailed budget reviews by the relevant parliamentary committee. Performance information is further integrated into the budgetary process through special performance reports to parliament on various expenditure areas during the year.

Performance auditing dates from the early 1990s and was the main task of the National Audit Office (RRV), after financial auditing was largely devolved to agencies themselves. Audit is seen as having a moral responsibility to ensure that action is taken on its reports and has had an important role in bedding down results-based management and budgeting through promoting the development of the related performance indicators for the results-based budgeting system introduced from 1993.

The RRV was reconstituted in 2003 as a body headed by three separate and equal Auditors-General reporting to parliament as well as the government, and has a major performance auditing role which reflects the responsibilities of agency management for efficient operations, reliable reporting, and compliance with regulations (RRV, 2000). The RRV may comment on material cases where efficiency has not been achieved and identify possible efficiency gains. It reviews (but does not formally attest) the performance information in terms of whether the information published is true and fair. It will indicate cases where it considers that the information is not truly and fairly presented, is not free from significant errors, or is incomplete in meeting decision-makers’ needs. That is, the reporting is on an exceptions basis. However, such adverse audit reports have been rare. This information is available to parliament when considering budget requests, so that there is considerable emphasis in ensuring that reliable and relevant information is available to decision-makers.

Finally, the RRV also plays a consultative role in assisting agencies to improve their performance measurement.

Although there has been no systematic study of the use of performance auditing findings in budget formulation, it can be noted that apart from the close integration of performance issues into the budget, the government is required to report to parliament annually on actions taken on RRV reports, and this may feed into parliament’s review of the budget.

The United Kingdom

The National Audit Office has since 1983 had a formal mandate to examine and report on the economy, efficiency, and effectiveness of government organizations, programs, activities, and systems, although it may not go so far as to question “the merit of [Government] policy objectives.” The 1983 mandate to some extent formalized a role which had already been developed. The formal extension of the NAO’s mandate was strongly supported by the then Prime Minister, Margaret Thatcher, with her focus on efficiency and budget savings. About 50 percent of NAO resources are devoted to performance auditing through studies of “value for money.”

The NAO has a Directorate of Performance Measurement to coordinate its work on performance measurement in financial audits, reviews of governance arrangements of central governance bodies, and value for money studies. As part of this role the NAO has provided guidance to public sector organizations on developing performance measurement systems as well as evaluations of these systems, as reflected in two major reports (NAO, 2000, 2001). It also collaborates with executive government in promoting good practices, through jointly issued guidance (HM Treasury et al., 2001, 2003) and through participation in the Performance Information Panel, a multi-stakeholder advisory group which has reviewed the robustness of draft performance measurement arrangements for departmental Public Service Agreement (PSA) targets (see Chapter 12). The Directorate also provides advice on the quality of performance data, covering both performance measures and the information systems which collect them, when requested by central government agencies.

For departments and ministries performance measurement now focuses on targets as set out in PSAs, begun in 1998 and negotiated with the Treasury, which cover a period of three years ahead or until the next envisaged expenditure review, which focus on desired outcomes, with related outputs set out in service delivery plans. The NAO may also validate these performance measurement systems.

The NAO plays a role in the external validation of performance measures. External validation to provide reasonable assurance of the quality of performance information for agencies and non-departmental public bodies is required where the achievement of targets affects the pay of staff. In all other cases it is left to the discretion of the responsible minister as to whether external validation should be required, but it is expected that validation will be undertaken where performance information is used to inform key strategic decisions such as allocation of resources or reporting to parliament. External validation can be provided by the NAO, by other external experts, or by the organization’s own internal audit unit.

Although there has been no systematic review of the use of value for money audits in budget decisions, it can be noted that the NAO measures its own performance partly on achieving the objective of identifying budgetary savings from its performance audits of nine times its annual operating costs. And departments agree their PSA targets as part of the spending review process, so that in principle, resource allocation decisions have a link with performance.

For local government the Audit Commission specifies the standard performance indicators that local authorities are required to produce and publish each year, and also publishes a comparative set of these indicators, covering all local government units, to enable the performance of different local authorities to be compared. Auditors appointed by the Commission check that the systems used are robust enough to produce reliable information. In addition “best value performance indicators” reflecting the national interest in the delivery of local services are required, with an emphasis on targets that reflect continuous improvement. Auditors review whether the systems are producing reliable information and whether the required “best value performance plans” have been prepared according to the legislation.

The Audit Commission also carries out major value for money studies of public sector services which identify areas where performance can be improved and savings made in the delivery of local government services.

The United States

The Government Accountability Office assists Congress in its oversight of the executive branch, and about 70 percent of its work is in response to specific requests by Congress. Its performance auditing role covers economy, efficiency, and effectiveness, and much of its work is in-depth program evaluation. The scope of this work is defined in its auditing standards, commonly known as “the yellow book,” which was updated in 1994. While there is no formal audit of the extensive performance information produced by agencies which cover effectiveness, efficiency, and service quality, the GAO’s performance audits may review performance management systems and the quality of performance information in individual organizations.

The GAO also reviews cross-cutting performance management issues; for example, it has carried out a major review of the implementation of the Government Performance and Results Act, 1993, under which agencies are required to develop five-year strategic plans, set annual program goals, and develop output measures through annual performance plans and reports and link them with budget submissions. It has also reviewed the Program Assessment and Review Tool initiated in 2002 by the executive branch to better link performance with the budget.

Its reporting to Congress, mainly on issues identified by Congress as requiring review, suggests that its performance audits should closely inform Congressional decisions on the budget. However, no detailed study has been made of the use of its performance audits by Congress in the budget process.

The GAO’s own performance measurement stresses the identification of savings and value for money improvements identified in its reports, which it endeavors to quantify in its own annual report to Congress.

Notes

This chapter focuses on external auditing. However, some reference is made to internal audit arrangements, and it can be seen that similar issues arise in internal audit.

Definitions of these three elements, commonly referred to as the “three E’s,” are not provided in the INTOSAI standards. However, there is extensive public sector management literature to assist in defining these concepts, some of it based on drawing a distinction between outputs (physical measures of goods or services delivered), which reflect efficiency and outcomes (which measure the achievement or otherwise of program objectives through those outputs), which reflect effectiveness. A new classification of service delivery quality, which may encompass elements of both efficiency and effectiveness, is also sometimes added. In all cases inputs, reflected in costs, are the other side of the performance coin.

See Zavelberg (1996, p. 201) for a description of this role. It can be noted that the President of the Federal Court of Audit is also designated as the Federal Commissioner for Efficiency.

NPM can be distinguished from earlier moves to results-based management concepts by its comprehensive approach, involving both management devolution and performance accountability. See OECD (1996b, 2005) for a fuller description of NPM developments in OECD member countries.

See Shand and Anand (1996) for a more detailed discussion of the origins of performance auditing. It can also be noted that INTOSAI does not appear to perceive performance auditing as a necessary part of the external audit function. Its discussion of the basic nature of audit in the Lima Declaration of Guidelines on Auditing Precepts adopted by INTOSAI in 1977 refers to legality, regularity, economy, efficiency of financial management (italics added), although this latter term is not defined. However, it also states that “the full scope of government audit includes regularity and performance audit.”

For example, the Australian Financial Management Improvement Program which commenced in 1983, the UK Next Steps initiative of 1990, the Canada Public Service 2000 initiative of 1990 and the US National Performance Review of 1994.

See OECD-PUMA (1997) for a more detailed description of some of these changes to budget systems.

Chile, which has a well developed formal system of program evaluation led by the central budget office of the Ministry of Finance, is a good example.

The Finnish National Audit Office and the New Zealand Office of the Auditor-General are two institutions which have such a role.

For example, a 1990 Australian Auditor-General report criticized the administration of a grant program for having no clear criteria for the allocation of grants beyond apparent political rationality.

INTOSAI, Code of Ethics and Auditing Standards, available at <http://www.intosai.org>. However, there is no mechanism available to INTOSAI to enforce these standards, unlike auditing standards promulgated by the international accountancy profession through the International Federation of Accountants (IFAC). Public sector external auditors may also operate under their own local auditing standards and the international standards on auditing promulgated by IFAC.

However, it can be noted that some audit organizations believe that such recommendations should be general rather than specific, that is, the recommendations should indicate the improvements needed but not necessarily the details of how to achieve or implement them.

Lima Declaration (see note 5 above) Section 20.2 and INTOSAI Reporting Standard 4.0.24 respectively.

Also, the Lima Declaration states that if special skills are not available among the audit staff, the national audit institution may call on external experts as necessary.

This issue is discussed in Moynihan (2003).

See Shand and Norman (2005) for a more detailed discussion of the New Zealand approach.

ANAO practice has been to develop better practice guides across a wide range of issues, so as to assist in improving public sector performance.

References

    ANAO2001Performance Information in Portfolio Budget Statements,ANAO Audit Report 18 (Canberra: ANAO) November.

    ANAO2002Better Practice Guide. Performance Information in Portfolio Budget Statements (Canberra: ANAO) May.

    ANAO2004Performance Management in the Australian Public Service,ANAO Audit Report 6 (Canberra: ANAO) August.

    ANAO and Department of Finance and Administration2004Better Practice Guide. Better Practice in Annual Performance Reporting (Canberra: ANAO and Department of Finance and Administration) April.

    BarzelayM.1996aPerformance Auditing and the New Public Management: Changing Roles and Strategies of Central Audit Institutions,” in OECD Performance Auditing and the Modernization of Government (Paris: OECD).

    BarzelayM.1996bCentral Audit Institutions and Performance Auditing: A Comparative Analysis of Organizational Strategies in the OECD.Unpublished paperOctober.

    ESV2003Performance Management in Swedish Central Government (Stockholm: ESV).

    HM Treasury Cabinet Office and National Audit Office2003Setting Key Targets for Executive Agencies: A Guide (London: HM Treasury) November.

    HM Treasury Cabinet Office National Audit Office Audit Commission and Office for National Statistics2001Choosing the Right Fabric: A Framework for Performance Information (London: HM Treasury) March.

    INTOSAI1992Auditing Standards (issued by the Auditing Standards Committee at the XlVth Congress of INTOSAI, 1992Washington DCamended by the XVth Congress of INTOSAI 1995Cairo).

    MoynihanD.2003Performance-Based Budgeting: Beyond RhetoricWorld Bank PREM Public Sector Note No. 78February (Washington: World Bank).

    NAO2000Good Practice in Performance Reporting in Executive Agencies and Non-Departmental Public Bodies (London: NAO) March.

    NAO2001Measuring the Performance of Government Departments (London: NAO) March.

    OECD1996aPerformance Auditing and the Modernization of Government (Paris: OECD).

    OECD1996bGovernance in Transition: Public Management Reforms in OECD Countries (Paris: OECD).

    OECD2002Overview of Results Focused Management and Budgeting in OECD Member Countries (Paris: OECD).

    OECD2005Modernizing Government—The Way Forward (Paris: OECD).

    OECD-PUMA1997In Search of Results: Performance Management Practices (Paris: OECD).

    PollittC. and H.Summa1997aPerformance Audit: Travelers Tales,” in Evaluation for the 21st Century: A Handbooked.E.Chelimsky and W.Shadish (Thousand Oaks, Calif., and London: Sage).

    PollittC. and H.Summa1997bPerformance Audits and Their Neighbours: Concepts and Practices in Five SAIs.Paper presented to the Conference of the European Evaluation Society Stockholm. RRV 2000 Audit Guide (Stockholm: RRV).

    SandbergB. and K.Larsson1996Role of Auditing in Performance Improvement in Sweden,” in OECD Performance Auditing and the Modernization of Government (Paris: OECD).

    ShandD. and P.Anand1996Performance Auditing in the Public Sector: Approaches and Issues in OECD Member Countries,” in OECD Performance Auditing and the Modernization of Government (Paris: OECD).

    ShandD. and R.Norman2005Performance Budgeting in New Zealand.Paper prepared for the IMF Conference on Performance BudgetingDecember.

    ZavelbergH.G.1996Legislatures and Performance Auditing: The German Experience,” in OECD Performance Auditing and the Modernization of Government (Paris: OECD).

    Other Resources Citing This Publication