New Data Tools for investigative Journalists

Reporting on public spending is key to accountability, but access and analysis of data is not always easy. This is why OpenBudgets.eu, Your Data Stories and DIGIWHIST, three H2020 projects, have focused on tracing public spending, data analysis and storytelling, in the name of accountability. Together the three projects provide an important set of resources for journalists in their work to hold governments accountable. This guide shows how the different tools can help journalists in their work, gives examples of stories, offers data trainings/tutorials and provides datasets that can be used for their next investigation.

OpenBudgets.eu
The OpenBudgets.eu toolbox provides journalists with training material, visualisation and deep analytical software to understand public budgets and government accounting. It also contains tutorials to give users a head start on researching public spending. Start off in a playful way and let the game “the good the bad and the accountant” provide you with the basics on how public spending, conflict of interest, and corruption may appear. OpenBudgets.eu supports a linked data repository for budgetary data, so that budget and spending data can be stored, visualised and analysed on the platform. This provides the first overviews and insights into where to look for stories.

DIGIWHIST
Public budget and spending data can then in turn be combined with the data provided by in DIGIWHIST: the largest database for public procurement data in Europe. DIGIWHIST provides portals in 35 jurisdictions across all EU-Member States for EU tender data. On opentender.eu journalists and researchers can access procurement data to have a closer look at projects and transactions. Using red-flag tools, they can find suspicious patterns and gain insights into the context, the rules and regulations and potential misuse of procurement procedures. This allows them to find leads on how public money is being spent and uncover potential misuse of public funds.

Your Data Stories
With the insights and data from OpenBudgets.eu and DIGIWHIST, the data can be used for storytelling with the tools of Your Data Stories. Your Data Stories allows journalists and data experts to interconnect, search, analyse, explore and understand available data, in order to extract insights, facts and material to support and shape their data stories. The YDS platform allows users to browse through and analyse datasets from various sources, which have been already aligned, interlinked and unified under a common vocabulary. The simple visual approach of YDS makes it possible for journalists and less skilled data researchers to delve into the data from any possible aspect and perspective finding new angles. The all-in-one workflow lets users go straight from finding the data to publishing their story.

All three project, OpenBudgets.eu, DIGIWHIST and Your Data Stories, have made these tools available on their websites. Whilst some are still in prototype stage, together they will further the field of financial data journalism, aiding in the analysis, visualisation and publication of financial datasets, from budgets and spending data, to procurement and statistics, leading to more investigative stories and more transparency.

This post is a co-creation of Anna Alberts (OpenBudgets.eu), Tilman Wagner, Anna Triantafillou (both Your Data Stories), Daniel Tan and Fiona Harrison (both DIGIWHIST). The publication can also be found on the respective project blogs.

1. GOVERNMENTS SHOULD SET UP A COMPREHENSIVE CENTRAL PUBLIC PROCUREMENT PORTAL

Currently, 23 of 34 DIGIWHIST countries already have a single national public procurement portal publishing all regulated tenders and contracts, albeit information content, usability, and reliability varies greatly. Providing comprehensive public procurement information free of charge in an easy-to-use format to all interested parties is expected to increase market transparency, decrease transaction costs, and facilitate government accountability. This has also been recommended by the G20. Hence, a well-functioning central public procurement platform should contribute to achieving value for money in public procurement as well as increase integrity throughout the public sector.

The DIGIWHIST portal www.opentender.eu featuring all of the above functionalities will be launched in 2018, filling the gap where source data quality allows.

2. GOVERNMENTS SHOULD COMMIT TO PUBLISHING PROCUREMENT DATA BY DEFAULT IN AN OPEN AND EASY-TO-UNDERSTAND DATA FORMAT

Publishing public procurement data in a timely and easy-to-understand format and publishing information as machine readable data are essential for lowering the barriers to data use and reuse by all stakeholders. Having a set range of few reporting formats makes the understanding and processing of public procurement information the least cumbersome and thus represents the optimal scenario.

As recommended by international civil society organisations like the Open Knowledge Foundation, the Sunlight Foundation or the Open Contracting Partnership, governments should adhere with machine-readable file formats such as CSV, JSON, and XML to ensure usability. Users should be also able to download data in bulk either as .csv or through an Application Programming Interface (API).

The number of data publication forms should be kept to the very minimum in order to minimize complexity, facilitating stakeholder engagement with the data. To ensure data format and accessibility meet user needs, governments should establish a monitoring, evaluation, and learning process involving data users as well as data producers.

3. GOVERNMENTS SHOULD REQUIRE LOW REPORTING THRESHOLDS WITH THE SAME REGULATORY FRAMEWORK FOR ALL PUBLIC BODIES AND SPENDING AREAS

Governments should implement low monetary publication thresholds and apply public procurement rules to all public bodies and spending areas. Ideally, monetary thresholds requiring publishing tendering information should be close 0€ so that all or most public spending through public procurement systems is transparently published. With the widespread use of e-procurement systems and electronic administration of public purchases the cost of such widespread transparency more or less equal to the fairly minimal cost of adapting government IT systems (most if not all information published in public procurement announcements must be recorded as part of standard record keeping anyways!). When it comes to applying procedural rules for example requiring open bidding, the public expectation for open competition, the expected benefits of competing bids, and the associated administrative burden of administering open and highly regulated tenders must be carefully balanced. A lighter procedural regime should be applied to the smallest value contracts, with full procedural and transparency rules required for higher value contracts starting from about 20-40,000€.

The scope of public institutions outside the remit of public procurement law and publication requirements such as public utilities or local governments and sectoral exceptions such as defence and national security should be minimised in order to reap full benefits of market transparency and minimise gaming of exceptional rules.

4. GOVERNMENTS SHOULD INCREASE THE DEPTH OF PROCUREMENT DATA PUBLISHED, IN PARTICULAR PUBLISHING CONTRACT IMPLEMENTATION DATA, RELIABLE DATA ON CORRECTED AND FAILED TENDERS, AND ORGANISATIONAL IDS BY DEFAULT.

Procurement processes typically span over a long period of time and include various stages, such as call for expression of interest or pre-announcement; a formal call for tenders; announcements of the winning bidder(s); contract implementation; and ex-post assessments of performance. Any of these stages may also be subject to modifications, cancellations, or legal challenge. As public procurement typically implies a highly structured and complex procedure, there are many variables or bits of information which should be reported at each stage in order to provide sufficient information to interested bidders and civil society.

Governments should increase the depth of public procurement data publication in order to allow for the comprehensive monitoring of public procurement processes underpinning good government, efficient competition, and government accountability. In particular, we recommend that governments:

• publish public procurement data relating to the whole procurement cycle on existing public procurement platforms, including at a minimum: call for tenders, contract awards, and contract completion/ implementation announcements
• publish information on amendments, modifications, and failed tenders in a structured and reliable format so that up-to-date information is available on all tenders
• publish at least a minimum set of variables essential for government accountability and transparency of bidding. Crucially, unique organisational identifiers linkable to external registries such as company registries is a fundamental precondition to monitoring organisational performance

DIGIWHIST proposes a minimal list of variables and their publication location for governments. This list is close to the Open Contracting Data Standard to connect to already existing international standards but contains some specific variables for the European context, underpinning effective monitoring across the continent.

5. GOVERNMENTS SHOULD FACILITATE THE LINK BETWEEN PUBLIC PROCUREMENT DATA AND FURTHER DATASETS

Storing public procurement data separately from linked datasets is problematic as public procurement is a cross-cutting government function with numerous links to organisational financial performance, such as public body budget deficit, and to sector-specific outputs, such as hospital mortality or road usage.

We recommend that governments establish the link between public procurement data and related datasets describing organisational behavior and performance through the use of common organisational and contract IDs across different data systems, such as public procurement, payments, company registry, or court rulings. Sectoral public sector outputs such as quality of roads is what ultimately matters for citizens, hence increasing the trust in complex public procurement systems and minimizing corruption risks should build on linked data. Linking datasets would not only allow for better understanding how public money is spent but also for better risk assessment of public procurement processes. Data on company ownership and the data on individuals involved in public procurement could be linked in order to measure and eventually control the risks of favouritism.

6. GOVERNMENTS SHOULD LINK PUBLIC PROCUREMENT ANNOUNCEMENTS TO ORIGINAL PROCUREMENT DOCUMENTS BY DEFAULT

Typically, public procurement data as published in public procurement platforms only contain summary information of the original full procurement documents like full tender specification or the signed contract. Directly linking announcements to the original full documentation greatly decreases transactions costs and decreases the probability of corruptly providing crucial tendering information to selected bidders.

Hence, governments should link procurement announcements to all relevant original documents. Those should include the full tender documentation and maps, plans, etc. Ideally, signed contracts should also be linked and easily available. Here, information on sub-contractors as well as contract amendments, invoices, and completion reports submitted should be linked to the dataset. Submitted bids, or at least parts of them, may be exempt from these stringent transparency rules for protecting commercially sensitive information or privacy of individuals.

7. GOVERNMENTS SHOULD INTRODUCE CONTROL MECHANISMS TO ENSURE DATA QUALITY IS MAINTAINED

Missing, incomplete, and erroneous data is one of the fundamental problems in European public procurement data systems. Even in countries with comprehensive templates for data reporting and a central public procurement authority overseeing the data generation process, the quality of the administrative data is low, predominantly due to poorly designed online platforms and the lack of enforcement.

Accordingly, we recommend that existing data reporting requirements are adequately enforced and data quality is increased to the legally mandated minimum throughout Europe. To ensure all required data fields are filled out with truthful information, governments should introduce centralised control mechanisms and penalties for non-compliance. Punishment of non-compliance in extreme cases can include the freezing of the tender or interrupting payments until records are corrected, practices employed. But more light-touch enforcement may also produce the desired data quality, such as sending automatic clarifying questions back to the reporting public bodies until all missing information and inconsistencies are resolved. While data completeness and adequate publication may not appear crucial for public procurement outcomes, more information in an easily accessible format leads to better competition and better outcomes.

8. GOVERNMENTS SHOULD LOWER BUREAUCRATIC BURDEN BY LINKING PUBLICATION SYSTEMS TO TENDER, CONTRACT AND PAYMENT MANAGEMENT SYSTEMS

Currently, public procurement data is only rarely automatically filled in from linked administrative databases such as corporate registries (see above on data linking), as such links are not established to start with. This means that in most European countries, standard elements of public procurement announcements and records such as winning company name and address have to be manually entered, which creates additional unnecessary work and increases the potential for error.

If administrative datasets are linked, the officially verified (and supposedly correct) information should be automatically added to public procurement records to lower public procurement administrators’ administrative burden as well as minimizing the risk of erroneous data entry. This is also recommended by the G20 and should be followed by the EU as a guideline. G20 Principles for promoting integrity in public procurement.

9. GOVERNMENTS SHOULD ENCOURAGE THE REGULAR USE OF PUBLIC PROCUREMENT DATA BOTH INSIDE AND OUTSIDE GOVERNMENTS

As there is very little structured data, there are only few government agencies and non-governmental organisations which actually make use of public procurement databases in a substantive way. Reuse of public data has the capacity to contribute to greater competitiveness and more accountable government if stakeholders can understand it, act on it, and pursue change.

Governments who are the principal data guardians should promote the use of public procurement data within government and facilitate data reuse by non-governmental stakeholders such as civil society watchdogs and data provider firms. The use of public procurement data should be facilitated first by creating direct feedback mechanisms throughout the entire procurement process cycle (i.e. planning, tendering, awarding, implementation) involving all stakeholders from within and outside government. Second, governments should support non-governmental organisations which monitor, analyse, and investigate issues in the process as a friendly ally to both procuring entities and monitoring bodies such as prosecutors.

There is plenty of global good practice of engaging stakeholders in verifying, monitoring, and acting on public procurement data and analytics. For example, European Commission (2017) Integrity Pacts and Social Witness programs in Mexico provide potentially effective ways of strengthening the importance of user views and civil society monitoring. Open Government Guide (2017) Country Example. In Mexico “social witnesses” oversee public procurement. In addition, stakeholders can be given the opportunity to provide feedback to existing procurement processes in the case of irregularities, for example in the implementation phase, but also on the data published (for instance, by marking incomplete datasets). Some countries like the UK have already started working on stakeholder engagement practices, with one example being the construction of Heathrow Terminal 5. OECD (2016) Public Procurement Toolbox. Country case: Stakeholder engagement during the construction of Heathrow Airport Terminal 5.

This is a condensed excerpt of the original blog post by our DIGIWHIST members Mihály Fazekas and Mara Mendes published here: https://opentender.eu/blog/2017-03-recommendations-for-implementation/

Besides more in-depth information the original blog post also offers interactive graphs based on DIGIWHIST research results as well as a downloadable PDF version of the recommendations.

BACKGROUND
Approximately 15% of the EU’s Gross Domestic Product is spent every year on procuring goods and services, and some estimates indicate that corruption increases the cost of government contracts by 20 – 25%. It is even more worrying that corruption in public procurement compromises widely supported public goals, such as building safe highways, high quality school buildings, or delivering medicine in time. These are a few of the main reasons why more research needs
to be done on how to make public procurement more efficient and transparent. Addressing this gap is what the EU-funded, large-scale project DIGIWHIST does. This policy paper presents key data challenges in public procurement and proposes recommendations to improve the state of data and data use for better outcomes.

OPEN DATA AND PUBLIC PROCUREMENT
In the 34 European countries examined by DIGIWHIST (DIGIWHIST deliverable D1.1), public procurement is regulated by national and supranational (EU) legislation. This means procurement processes and their publications are governed by monetary values (thresholds). Those thresholds determine the way a tender has to be published at the national level and whether it has to be advertised at the European Union level. The latter is done on the European procurement platform Tenders Electronic Daily (TED). Procurement procedures involve a variety of stakeholders: procurement officers who design and implement tenders, experts who advise on content (e.g. engineers, medical staff) and bidders who bid on the actual tender. In addition, public procurement procedures are interesting for citizens who may want to find out what progress has been made in the construction of a public building or about who won the contract to supply their school canteen, for example. Linking procurement data with other datasets such as budget data creates even richer information on how money is spent. Efficient public spending has increasingly become a focus of transparency advocates, with organisations across the globe launching projects aimed at increasing transparency in public procurement. The Open Contracting Partnership has developed a publication standard, and many NGOs have developed risk indicators across Europe. For its focus countries, DIGIWHIST has developed a set of variables under which procurement data for all 34 countries is analysed and published on one generic portal for easy comparison across borders. Indicators measuring transparency, corruption risks, and administrative quality are also applied to the datasets.

THE PROBLEM
Most countries that have been examined in DIGIWHIST research fail to publish their procurement data to an acceptable minimum standard. Many well-governed countries such as Sweden or Germany publish only those tenders which are regulated by EU Directives in a transparent and data-rich manner. Here, TED is the most reliable resource for open public procurement data. This is in striking contrast with Eastern European countries such as Romania or Croatia, which have introduced low reporting thresholds of only a couple of thousand euros. This has made their procurement spending not only transparent, but also more competitive. With a few exceptions such as Italy and Estonia, no government publishes information on contract implementation, making it impossible to know what happens after the contract is awarded — for example, did the suppliers deliver on time and budget?
In addition to the lack of publicized information on the whole tender cycle, the sources on which procurement data is published can vary greatly and may even require the payment of a fee, making it very difficult for citizens to find the information they are interested in. Even on TED, some of the required fields are either not filled out or not filled in a standardised way, which makes locating a given tender as well as comparing different tenders sometimes impossible. All these obstacles create an opaque environment in which procurement practitioners, bidders, and citizens find themselves.

OPPORTUNITIES AND BENEFITS
Publishing procurement data in an open data format opens a wide range of opportunities. It empowers governments to produce better analytics, which creates vast learning opportunities across authorities. Better and more accessible data can also be used by potential and actual bidders to assess opportunities and evaluate their own performance internally. This would ultimately lead to more competition and ideally better outcomes.

The availability of procurement spending statistics is also a challenge at the EU level, in spite of extensive EU-wide regulations. Such data would enable civil societies to better understand government performance and enable civil society to hold governments more accountable.

RECOMMENDATIONS

1. GOVERNMENTS SHOULD SET UP A COMPREHENSIVE CENTRAL PUBLIC PROCUREMENT PLATFORM

Providing comprehensive public procurement information free of charge in an easy-to-use format to all interested parties is expected to increase market transparency, decrease transaction costs, and facilitate government accountability. Hence, a well-functioning central public procurement platform should contribute to achieving value for money in public procurement as well as increase integrity throughout the public sector. The DIGIWHIST portal opentender. eu, featuring all of the above, is in development and will be launched in 2018.

2. GOVERNMENTS SHOULD COMMIT TO PUBLISHING PROCUREMENT DATA BY DEFAULT IN AN OPEN DATA FORMAT

Publishing public procurement data in a timely, simple, and easy-to-understand format and publishing information as machine readable data are essential for lowering the barriers to data use and reuse by all stakeholders. As recommended by international civil society organisations like the Open Knowledge Foundation, the Sunlight Foundation, or the Open Contracting Partnership, governments must adhere with machine-readable file formats such as CSV, JSON and XML to ensure usability. Users should be also able to download data in bulk either as .csv or through an API. The number of data publication forms should be kept to the very minimum in order to minimize complexity facilitating stakeholder engagement with the data.

3. GOVERNMENTS SHOULD PRESCRIBE LOW REPORTING THRESHOLDS WITH THE SAME REGULATORY FRAMEWORK FOR ALL PUBLIC BODIES AND SPENDING AREAS

Governments should implement low monetary publication thresholds and apply public procurement rules to all public bodies and spending areas. Ideally monetary thresholds should be between 0€ and 5,000€ so that most public spending through public procurement systems is transparently published and regulated. In order to carefully balance the demand for transparency and the associated administrative burden, a lighter regime could be applied to the smallest value contracts, with full procedural and transparency rules applied for higher value contracts starting from about 30-40,000€.

4. GOVERNMENTS SHOULD INCREASE THE DEPTH OF PROCUREMENT DATA PUBLISHED

This should include:

• publish public procurement data relating to the whole procurement cycle on existing public procurement platforms including at a minimum call for tenders, contract awards and contract completion/ implementation announcements.
• publish information on amendments, modifications, and failed tenders in a structured and reliable format so that up-to-date information is available on all tenders.
• publish at least a minimum set of variables essential for government accountability and transparency of bidding, such as the description of the purchase, information on bidders and subcontractors, final payments, contract performance, and unique organisational identifiers.

5. GOVERNMENTS SHOULD FACILITATE THE LINK BETWEEN PUBLIC PROCUREMENT DATA AND FURTHER DATASETS

We recommend that governments establish the link between public procurement data and related datasets describing organisational behavior and performance using common organisational and contract IDs throughout different data systems, such as public procurement, payments, company registry, and court rulings.

6. GOVERNMENTS SHOULD LINK PUBLIC PROCUREMENT ANNOUNCEMENTS TO ORIGINAL PROCUREMENT DOCUMENTS BY DEFAULT

Those should include the full tender documentation and maps, plans, etc. Ideally, signed contracts should also be linked and easily available. Here, information on sub-contractors as well as contract amendments, invoices, and completion reports submitted should be linked to the dataset. Submitted bids or at least parts of them may be exempt from these stringent transparency rules for protecting commercially sensitive information or privacy of individuals.

7. GOVERNMENTS SHOULD INTRODUCE CONTROL MECHANISMS TO ENSURE DATA QUALITY IS MAINTAINED

We recommend that existing data reporting requirements are adequately enforced and data quality is increased to the legally mandated minimum throughout Europe. To ensure that all required data fields are filled out with truthful information, governments should consider introducing centralised control mechanisms and penalties for non-compliance.

8. GOVERNMENTS SHOULD LOWER BUREAUCRATIC BURDEN BY LINKING PUBLICATION SYSTEMS TO TENDER, CONTRACT AND PAYMENT MANAGEMENT SYSTEMS

If administrative datasets are linked, the officially verified (and supposedly correct) information should be automatically added to public procurement records to lower public procurement administrators’ administrative burden as well as minimize the risk of erroneous data entry.

9. GOVERNMENTS SHOULD ENCOURAGE THE REGULAR USE OF PUBLIC PROCUREMENT DATA BOTH INSIDE AND OUTSIDE GOVERNMENTS

Governments who are the principal data guardians should promote the use of public procurement data within government and facilitate data reuse by non-governmental stakeholders such as civil society watchdogs and data provider firms. The use of public procurement data should be facilitated first by creating direct feedback mechanisms throughout the entire procurement process cycle (i.e. planning, tendering, awarding, implementation) involving all stakeholders from within and outside government.

The original text was written by Mihály Fazekas and Mara Mendes and published on opentender.eu. Follow this link to access interactive graphics and to download the text in PDF format.

When talking anti-corruption, the most common buzzwords flung around by civil society activists, researchers and development professionals alike are transparency and accountability. Transparency is seen as so key to the fight against corruption that its arguably most important advocate took it as part of its name: Transparency International. It should thus not come as a surprise that the EU Horizon 2020 DIGIWHIST project also aims at increasing transparency, specifically in the realm public procurement. But how do we increase transparency, and how does this contribute to more accountable governance?

A significant part of the project involves the collection of publicly available budget data for both national- and local-level governments across the EU countries and beyond. DIGIWHIST relies on countries following through on their commitments to budget transparency in order to find the necessary data, but it’s evident that not all countries are pursuing budget transparency in the same way.

Budget transparency – and how to get it right

The OECD definition of budget transparency is “the full disclosure of all relevant fiscal information in a timely and systematic manner.” Thus, what’s important to consider when we talk about this kind of transparency is both how promptly the information in question is released, as well as how predictable and orderly its release is.

The digital era has transformed the practice of budget transparency significantly. Ideally, citizens who wish to inform themselves about their government’s budget simply need to access an online portal where they can find all of the desired information presented to them in an accessible, understandable format. Regular users of online media should, however, not be surprised that the reality of budget transparency is more complex than this. The extent to which it is realized varies radically today. Still, there are some examples of how to do digital budget transparency “right”.

The German government, for example, has two main portals for people to turn to. One is Bundeshaushalt-info.de, which allows visitors to explore the national budget via colorful infographics and interactive tables. The second is Govdata.de, the government’s official open data portal, where a simple search of “Bundeshaushalt” returns machine-readable versions of the national budget from 2012-2015. Both portals cater to diverse audiences, ensuring that everyone from casually interested citizens to dedicated data analysts (like the kind of people working on DIGIWHIST) can find the desired information in a form that works for them. On a more local level, a great example of this can be found in Spain. The Presupuestos de Aragón website offers visitors a variety of interactive tools and visualizations for understanding the Spanish autonomous community’s budget. Their open data portal, Opendata.aragon.es, also provides machine-readable budget documents for the years 2006 – 2017.

So how can we evaluate who is doing budget transparently well? One of the main organizations focusing on this is the International Budget Partnership, which conducts the Open Budget Survey. The survey tends to focus on more traditional aspects of budget transparency rather than solely on the digital aspect of it. It contains an index measuring countries’ budget transparency (amount, level of detail and timeliness of budget information), budget participation (opportunities for civil society and the general public to participate in the budget-making process), and budget oversight (capacity of institutions to influence how public resources are raised and spent). The most recent survey, published in 2015, saw New Zealand, Sweden, South Africa, Norway, and the United States leading the pack.

Another view on evaluating budget transparency comes from the Open Knowledge Foundation, which publishes a ranked government budget dataset. They have a specific focus on the availability of government data in digital forms, and rankings are based on nine different factors, such as whether budget data is openly licensed, if it is available online, and if it is machine readable. Because this index is more narrowly focused than the Open Budget Survey, it is hard to compare the two data sources, and they often have strikingly diverging rankings (for example, the Open Budget Survey ranks Russia 11th overall, while the OKFN rankings have Russia tied for 105th place). Thus when evaluating budget transparency, it’s important to be clear about what aspects of it you are specifically interested in.

Transparency – and then what?

Just having the data in hand does not mean the battle is won. More transparency does not automatically equal more accountability. On the contrary: exerting accountability via budget transparency is no easy feat. The mere existence and general availability of budget data does not mean that it is immediately possible to make observations on and draw meaningful conclusions from the data. First of all, the way the data is published is often problematic. It does no good for a country to boast that all of its budget information is published online if that information is buried in a difficult-to-navigate finance ministry site, or in a chaotically-organized open data portal. Other countries still exclusively publish budgetary information in PDFs or publish only select portions of their budget data in machine readable formats. This is prohibitive to organizations like DIGIWHIST who want to automatically extract and analyze budget data, as extracting information from PDFs is much more difficult and error-prone.

Second, the actual analysis of budget data and what changes in allocations from year to year actually mean can be challenging without sufficient contextual background. Huge changes can be explained away by departmental consolidations, and small shifts in numbers may actually be indicative of changes that warrant scrutiny – the point is, a casual observer of this data can’t necessarily look at it and understand what is happening and identify potential causes for concern.

Third, the budget data needs to be sufficiently detailed to be useful from an accountability perspective. For DIGIWHIST’s work with public procurement data, for example, it is not sufficient to know how much money is being allocated or spent by a single ministry. Since the project is interested in matching contracting authorities from procurement tenders with the specific government agencies to which they correlate, a deeper layer of detail is needed; budget experts refer to this as the “economic classification.” But not all countries release budget information at levels this specific, making it that much more difficult for organizations like DIGIWHIST to hold governments accountable for their budget allocations.

Though progress at times feels slow, there is a clear trend toward greater budgetary transparency in governments and better provision of structured, accessible data. Projects like DIGIWHIST help in furthering this push thanks to the pressure they place on governments to be more accountable. Their researchers make transparency and accountability more than just buzzwords and help citizens and civil society activists in their fight against corruption.

By Tori Dykes

Billions of euros are spent on procuring for public goods. Our research shows that some of the procurement could be done more efficient and hence cheaper. Here, having information on the whole public procurement cycle is essential to evaluate the process and monitor it properly. Without information on contract implementation such as the final price paid and completion date, any assessment is incomplete hence can be misleading.

When most of the people think about public procurement they either think of the result (e.g. a highway or school being built) or the tender that they saw being announced. However, there is much more to it. Procurement processes sometimes span over long time periods and entail different stages. These stages usually include a call for expression of interest or pre-announcement; a formal call for tenders; announcements on modifications (if any); cancellations or clarifications of the calls (if any); announcements of the winning bidder(s) and details of the contract signature; and information on contract completion and assessment of companies’ performance during implementation. We exclude planning stages here, which are internal procedures and not published openly. While information on the full procurement process is crucial to assess its quality, publicly available information typically has considerable gaps in Europe.

Open Contracting Data Standard provides a good overview of the full tender cycle (see above).

A major shortfall in covering the whole procurement cycle is the lack of information about contract implementation. This information is key in monitoring efficiency, and preventing corruption in procurement. Without this information, calculations for future risk assessment and benchmarking are impossible. Having this information does not only mean that citizens and civil society are able to monitor procurement processes – it is also crucial for public authorities and the evaluation of their procurement performance (I recommend reading this working paper to understand the relationship between benchmarking and procurement performance better). Nevertheless, only 8 in 33 European countries provide information on contract completion and the performance of the supplier (Read more about here). In some cases we found that while governments plan to provide the information on tender completion, the actual database lacks a lot of information. Like in the case of Italy.

In addition to quality management in contract performance, information on the contract implementation provides us also with information on the actual cost of the project. In a lot of cases costs in public procurement overrun. This can be due to unforeseen changes in the project (e.g. a construction side requires additional or different material than initially envisaged), but the reason can also be bad planning. A study by the Hertie School of Governance found that “the energy and ICT sectors especially are facing significant cost overruns, with 136% and 394% on average for finished projects respectively”. With such significant cost overruns the question about why the project was more expensive than foreseen becomes even more important.

While some European countries publish a wide range of documents related to the full tender cycle, announcements on contract implementation and completion are often very hard to link to procurement documents of earlier stages of the tender. Without linking all the different announcements characterising the same tender, it is very hard to assess spending efficiency or corruption risk. For example, knowing the final contract value at the end of contract implementation is a relevant bit of information, while it is much more valuable if it can be compared to the original cost estimate or the initially awarded contract value. In such cases of fragmented public information laborious case-by-case linking or inside knowledge about the tender are required to get the full picture.

To make public procurement more transparent and more efficient, we recommend governments to:

1. publish all documents related to the tender cycle (this should include: call for expression of interest; formal call for tenders; announcements on modifications; cancellations or clarifications of the calls; announcements of the winning bidders; the details of the contract signature, information on contract completion and assessment of companies’ performance during implementation);
2. link procurement documentation related to each tender, to make assessment of the tendering process easier and less prone to errors; and
3. ensure the information is provided in a timely and complete manner.

This blogpost was written by DIGIWHIST member Mara Mendes (Open Knowledge Foundation Deutschland) and originally posted here: https://okfn.de/blog/2016/07/from-publication-to-award/

In the past decades, efforts to fight corruption have increased across the globe. At a closer look not much evidence can be found to prove these strategies to be successful. We are still facing the same questions: Why is it so difficult to overcome corruption and what measures need to be taken to majorly improve the situation?

On 28 April 2016 Brussels-based economic think tank Bruegel hosted an event titled “Fighting corruption: from headlines to real impact” featuring ANTICORRP and DIGIWHIST researchers Mihaly Fazekas and Alina Mungiu-Pippidi and Transparency International (TI) representative Carl Dolan to talk about these questions. Together, the experts discussed the meaning behind different measurements of corruption and what conclusions they offer to support policy makers in their attempts to design better anti-corruption instruments.

The international flagship measurement for corruption levels is still Transparency International’s Corruption Perception Index (CPI), a tool to measure the expert perception of the phenomenon of corruption in specific countries and how it changes over time. The index has helped immensely in raising awareness of the topic worldwide. Yet, it leaves open an evaluation of the development and diffusion of corruption in respective countries with more objective indicators. This would help to compare anti-corruption measures taken. The ANTICORRP project, looking into the last 40 years of anti-corruption efforts, could identify only seven countries that have achieved significantly better control of corruption.

In the discussion, Alina Mungiu-Pippidi pointed out the need for better instruments to measure change. Indicators need to be modified in order to have more objective insights into corruption. According to Mihaly Fazekas, the trend towards open and big data helps researchers systematically collect and analyze previously undisclosed data. He highlighted public procurement as a good proxy for uncovering corruption, as its data helps trace preferential treatment. The Horizon2020 DIGIWHIST project puts together data on companies who win many contracts with public funds. This data is then matched with connections between politics and business to identify companies with political ties and the conflict of interest (COI) is measured.

The need for more evidence based reforms is clear – favoritism is still seen as “rule of the game” by many citizens in Europe. Recent findings by the European Research Center for Anti-Corruption and State-Building (ERCAS) show a strong correlation between trust in governments and their efforts to control corruption. New measurements and insights are a first step for these reforms.

A summary of the event can also be found here: http://bruegel.org/events/fighting-corruption-from-headlines-to-real-impact/

The report on “Public Integrity and Trust in Europe” written in behalf of the Dutch Presidency of the European Union can be found here: https://www.government.nl/documents/reports/2016/01/18/public-integrity-and-trust-in-europe

What does whistleblowing look like in the digital age? What are its benefits and pitfalls? On 17 November 2015 the Hertie School of Governance and the Council of Europe hosted a discussion on these questions. The event was a satellite event to the World Forum for Democracy, taking place at the Council of Europe 18 – 21 November and focusing on finding the right balance between freedom and control in democratic societies. The Hertie School brought together a set of panellists from a variation of countries and fields to present their very own experiences of working with whistleblowing. The panel was moderated by Anne Koch, regional director for Europe and Central Asia at Transparency International.

The different panellists showcased various experiences of working in the context of whistleblowing: Marius Dragomir, a journalist and senior manager for the independent journalism programme of the Open Society Foundations in London, Maksymilian Czuperski, working the Atlantic Council which recently supported the collection of evidence for the presence of Russian troops in Eastern Ukraine by crowdsourcing information from citizens. It also featured Simona Levi, the founder of Xnet, a Spanish online journalism platform specialized on engaging citizen. Xnet actively calls upon citizens to become whistleblowers and leak undisclosed information in order to uncover corrupt behaviour.

The final two panellists were Mara Mendes, project manager for Open Knowledge Germany and Alina Mungiu-Pippidi, professor of democratization at the Hertie School and director of the European Research Center for Anti-Corruption and State-Building (ERCAS). They presented DIGIWHIST, a new EU Horizon 2020 project. The project aims at increasing transparency and efficiency of public spending. It will do this through the systematic collection, structuring, analysis, and broad dissemination of information on public procurement through online platforms. Alina Mungiu-Pippidi highlighted the centrality of procurement data in fighting corruption. Linked to information on aggregate asset and income declarations data, she hopes that this data will help detect potential conflicts of interest and identify systemic vulnerabilities. In this way DIGIWHIST is supposed to specifically support journalists in creating transparency within the procurement sector.

What is a whistleblower?

One reoccurring theme at the discussion was the actual definition of a whistleblower. Anne Koch opened the panel by describing it as “any person that wants to report wrong doing to someone who can do something against the problem.” This stood somehow in contrast to the experience of Xnet’s Simona Levi, who, for instance, collected emails from whistleblowers at big Spanish banks and reported on wrongdoings in these contexts. The panel agreed though, that a whistleblower does not necessarily have to be someone working for the government or a private enterprise and release information from the inside. For Alina Mungiu-Pippidi it was “a person who is aware of a situation the rest of the world is not and brings it to public attention.” It can also be a group of people collectively gathering information that the wider public is unaware of, or analyse data collectively in order to highlight important information.

Who has the right to decide?

The debate also looked at the pitfalls of whistleblowing and discussed the questions of what safeguards are needed to prevent harm to innocent individuals through whistleblowing. In many countries protection of whistleblowers is still deficient and there are no laws specifically protecting whistleblowers from prosecution. Often those willing to share information are unaware of technical tools which can be used to protect their identity. The participants highlighted tools such as GlobaLeaks, which provides anonymous channels for whistleblowers. Journalists in particular carry a twofold responsibility. On the one hand they need to protect their sources and those who entrust them with information, also by teaching them secure ways to share information. Journalists, however, are also responsible for the information they publish. When Anne Koch asked the panel who has the right to decide what publications are in the public interest, the panel generally agreed: journalists can decide, but they have to be aware of their special responsibility. They will, however, always be better placed to decide than civil servants who might incriminate themselves by publishing data.

In the end, the best kind of whistleblowing might be done collectively. Communities of people can uncover corrupt behaviour of local officials and document what is happening around them. Also, individual whistleblowers depend on those around them. As Alina Mungiu-Pippidi pointed out: “In the end, laws cannot protect whistleblowers, but public opinion can.” A similar conclusion was also taken at a panel on ‘safe whistleblowing’ at the World Forum for Democracy in Strasbourg, which representatives of ERCAS also attended. One conclusion to be taken from both discussions is that whistleblowing should not remain an exception, but it should become the norm for citizens to report wrongdoings they witness.

In the UK where ‘First-Past-the-Post’ electoral rules are the norm in local elections, with the exception of Scotland and Northern Ireland, lack of local electoral accountability poses a high risk to government integrity as voters may not be able to elect an alternative to the cor­rupt incumbent. The persistence of uncontested seats and one-party dominated councils at the local level is a cause for concern across England in terms of quality of public services, value for money, and government responsiveness to citizen needs.

DIGIWHIST project coordinator Mihály Fazekas published last week a report based on big data titled ‘The Cost of One-Party Councils: Lack of Electoral Accountability and Public Procurement Corruption’ for the UK based Electoral Reform Society. Mihály Fazekas investigates the impact of weak electoral accountability on corruption in public pro­curement, explored at the local level in England (excluding London). In this context, weak electoral accountability at local elections is understood 1) if a borough is hav­ing a large share of uncontested seats (i.e. at least 10% of seats) or 2) if a council is being over­whelmingly controlled by the same party for 10 years or more (i.e. same party controlling more than 2/3 of seats without interruption).

Fazekas uses administrative data on electoral outcomes as well as on public procurement tendering procedures from 2009-2013 for 132,000 public procurement contracts. It employs an innovative measure of corruption risks in public procurement capturing a set of ‘red flags’ such as single bid submissions or shortened lengths of time between advertisement and submission deadline, signaling the likely steering of government contracts to a favoured bidder.

Based on the comparison of matching contracts, the findings suggest that weak elector­al accountability may lead to substantially higher corruption risks and lower price savings in English local councils. Councils of weak electoral accountability are roughly 50% higher corruption risk than their competitive counterparts which also coincides with foregone savings of 1-4% of contract value. The foregone savings by councils with weak electoral accountability roughly amount to £2.6bn annually.
This is substan­tial given the total £45bn annual spending estimate.

Un-matched and Matched contract samples price savings Comparisons: one-party dominated and competitive councils compared, 2009-2013, England

Source: European Union’s Tenders Electronic Daily

The figure above shows that median differences between matched contracts (matching serves to correct the varying spending composition among different locality types e.g. contract value or main product group of purchase) of different council types in terms of cost savings achieved in procurement tenders are substantial: 6.2% versus only 2.2%. The results on the local level are very diverse though as not in every locality there is such a negative effect.

The Electoral Reform Society states: “The £2.6bn potential wastage is a damning indictment of an electoral system that gives huge artificial majorities to parties and undermines scrutiny. This kind of waste would be unjustifiable at the best of times. But during a period of austerity it is simply astonishing.”

Based on these findings, it is very commendable that European Commission decided that indicators such as single bidding and lack of publishing the call for tenders constitute key aspects of public procurement performance and have included them into its procurement scorecard.

DIGIWHIST members were also present at the ANTICORRP conference “Understanding Governance Virtuous Circles: Who Succeeds and Why” (July 8-12, 2015, organized by the Hertie School of Governance (HSoG) and the European Research Centre for Anti-Corruption and State-Building, ERCAS), which got off to an intriguing start here at the European Academy Berlin. The aim of the event – with the biggest names in the international anti-corruption discourse in attendance – is to identify patterns that have led to national success stories (“virtuous circles”) in fighting corruption.

ANTICORRP has identified seven success stories in good governance that have led to low levels of corruption in those countries. One of these is Estonia, the small Baltic country that liberated itself from the Soviet Union in 1989-90. Can it be copied? Or lessons learned from it? Was Estonia’s model an accident of history – or something that can be applied elsewhere? To what extent does human agency play a critical role? What makes these reforms sustainable?

In a video presentation of a wide-ranging interview with Alina Mungiu-Pippidi of the HSoG, Estonia’s former prime minister Mart Laar, who led the country’s transition out of communism in the early 1990s, outlined the elements that were critical to getting the Estonian Wirtschaftswunder off the ground and sustainable with anti-corruption elements embedded.

Briefly, Laar’s government: 1) made fewer, simpler laws; 2) liberalized (deregulated) everything; 3) introduced open (merit-based) public service; 4) started anticorruption fight at beginning of transition; 5) had laws ready to go (came into office with them in hand); 6) first government came from the anti-communist resistance, highly motivated and unified in purpose; 7) controlled the instruments of the main processes (safeguarding banks from mafias, for instance); 8) sought out local advice that proved better than Western (U.S. American) advice.

In the discussion that followed, the panelists chimed in with the following:

“It’s essential to look at what has worked in the real world but there are no cook book recipes” for beating corruption. Michael Johnston, Colgate University

“One-man shows come and go but collective action is always necessary in fighting corruption.” Philip Keefer, Inter-American Development Bank

“Free and fair elections, biometric registration, and a serious electoral commission made an important difference in Nigeria” [where a peaceful transfer of power happened this year]. “Perhaps a virtuous circle is unfolding with the new leader there, as could be the case in Mongolia, too, where the corrupt former president was convicted and sent to prison. But it’s still too early to tell for sure.” Larry Diamond, Stanford University

By Paul Hockenos

DIGIWHIST’s Consortium Members came together for the first time to discuss the details of the EU HORIZON 2020 project which will last the following three years. Some of the hot topics were how the data collection and cleaning procedure will take place, what good work had been done before by consortium members in the field of legal mapping and public procurement data analysis and finally, how the results will be made public. Since the partners of this project live several countries or even continents apart (USA, Czech Republic, Hungary, Germany, UK, Italy, etc.), yet, each one of them represents a part of the puzzle that will finally be DIGIWHIST, successful communication is bound to be a key component of the upcoming months.

Deirdre Furlong (Research Executive Agency, Brussels): Horizon 2020 at a glance

Dr Francesco Calderoni (Italy, Transcrime) taking notes

Dr Mihaly Fazekas, the “Father” of DIGIWHIST (University of Cambridge, Department of Sociology) explaining the idea of DIGIWHIST

Fiona Harrison (Project Manager): You have to be on time with your paperwork!

Dr Jiří Skuhrovec (Datlab) has ample experience in public procurement data collection.

István János Tóth (Corruption Research Center Budapest) knows from the Hungarian experience how tedious data collection and cleaning can be.

Cambridge Computer Laboratory will definitely be very busy cleaning data in the next 2-3 years.

Clare College in Cambridge is a very inspiring environment, indeed. Looking forward to the next three years!