Most project management forecasters are fools or liars, says Oxford professor
Professor Bent Flyvbjerg, an international expert in the management of major projects, delivered outspoken criticism of forecasters in a new paper which demonstrates the considerable inaccuracy of front-end estimates of costs and benefits that are typically used to support decisions on projects. “Estimates are commonly poor predictors of the actual value and viability of projects, and cannot be trusted as the basis for informed decision-making. These forecasts frequently misinform decision makers on projects instead of informing them. Some of the inaccuracy comes from genuine forecasting mistakes arising from over-optimism, but some estimates are deliberately misleading, designed to secure financial or political support for a project.”
Professor Flyvbjerg goes further and calls many of these forecasts ‘garbage’ — moreover, expensive garbage, both in terms of the fees of the consultancy firms involved, and in terms of the costs and risks to investors and other stakeholders, that arise from their misrepresentations. “Many forecasts are garbage and can be shown to be worse than garbage,” says Flyvbjerg. “These reports give the client, investors and others the impression that they are being informed about future demand, or the costs involved in a major project, when they are being misinformed. Instead of reducing risk, reports like this increase risk by systematically misleading decision-makers and investors about the real risks involved.”
Flyvbjerg calls for firm action to be taken by all involved in the sector to address the problem. Clients should ask for their money back when they receive reports which later prove to be significantly inaccurate and misleading. In some cases, he urges them to seek compensation for decisions taken and occasionally, criminal action would be justified, he says. “Merely firing the forecaster may be letting them off too easily. Some forecasts are so grossly misrepresented and have such dire consequences that we need to consider suing them for the losses incurred as a result. In a few cases where forecasters foreseeably produce deceptive forecasts, criminal penalties may be warranted.”
He calls upon professional organisations like the PMI, APM, APA, RTPI and others to use their codes of ethics to penalize and possibly exclude members who produce unethical forecasts, and chastises the associations for failing to address this adequately. “This needs to be debated openly within the relevant professional organisations. Malpractice in project management should be taken as seriously as malpractice in other professions like medicine and law.”
Above all, such inaccuracy in forecasting is unnecessary, demonstrates Flyvbjerg. “Recent research has developed the concepts, tools and guidance on incentives that could help curb both delusional and deceptive forecasts.” Flyvbjerg’s own work has been central in much of this and in this new paper he sets out a clear case for quality control and due diligence to be applied to the evaluation of front-end forecasts. In a carefully argued and detailed piece, he offers an eight point due diligence plan which will allow those reviewing project forecasts to evaluate their accuracy as a basis for decision-making and investment.
“Whether forecasters are unwittingly or deliberately under-estimating the costs, completion times, and risks of projects, and over-estimating their benefits, we need to have a systematic basis for evaluating their findings in order to make informed investment decisions. Given the high cost of major infrastructure projects, the irreversibility of decisions, and the limited availability of resources, this is clearly critical for both public sector and private sector projects. Significantly more accurate forecasts can be produced by looking at the evidence available from previous similar projects which have been already completed – what I call, taking an ‘outside view.’ This seems so simple, but in practice it is transformative and leads to much more accurate forecasting.”
Using the case of an A-train project and drawing on the largest data set of its kind (which includes all large transportation infrastructure projects for which valid and reliable data on forecasting accuracy is available), Flyvbjerg works through his eight step due diligence plan in detail, evaluating the forecasters findings and demonstrating how it is possible to critically evaluate the accuracy and reliability of forecasts to take decisions from an informed basis. The plan relies upon establishing a statistically reliable benchmark from data available from other similar projects and using this benchmark built on the average performance of the previous projects, to compare against the estimates for a proposed project. It quickly becomes apparent if the forecasts are claiming to meet or exceed the average results of previous similar projects and the reasons can then be scrutinized.
On the initiative of the US Congress, the US General Accounting Office (GAO) has recently adopted Flyvbjerg’s methodology to undertake due diligence on the decision to build a $98.5 billion high speed rail system in California, the most expensive civil construction project in US history. Flyvbjerg is working with the GAO in this task.
Flyvbjerg has also recently been invited to serve as an expert witness in a $150 million class action suit against a forecaster. This is the first such case and may well set a precedent in terms of holding forecasters to account.