Open Science Symposium: Open Practices and Transparency in Business Studies and Economics

by Guido Scherp

On 27 April 2023, the ZBW – Leibniz Information Centre for Economics invited researchers of business studies and economics to Berlin for a day of exchange on Open Practices. In the morning a Meet-Up took place and in the afternoon a symposium on “Open Science – Research Transparency in Economics” (German). Both formats are part of the ZBW’s commitment to creating a discussion and networking space on the topic of research transparency for economics and business studies researchers.

The symposium was particularly dedicated to the topics of reproducibility and replication. Four national and international business researchers were invited to share their experiences with open practices for this topic and related food for thought. The event was moderated by Marianne Saam, Professor of Economics, in particular Digital Economics, at the Department of Socioeconomics at the University of Hamburg and head of the programme area “Open Economics” at the ZBW.

An open and collaborative approach pays off

Melanie Schienle, Professor of Statistical Methods and Econometrics at the Karlsruhe Institute of Technology, presented the Covid19-Nowcasthub in her talk “Transparent collaborative forecasts and nowcasts in real time” (German).

This is a platform for forecasting (forecast = 4 weeks, nowcast = estimate now-state) the incidence of hospitalization, which is based on a collaborative and combinatorial approach. To improve forecast quality, multiple models from different teams were combined using a standardized framework and an open, collaborative space for stakeholders to share was created. Jointly, the prediction quality and combination of the individual prediction models were assessed, evaluated, and interpreted. An attempt was also made to quantify the uncertainty that is always present in forecasts. Interestingly, the overall quality of the forecast combined from the different models performed better than the individual forecasts. The forecast data as well as the nowcast models are openly available on GitHub. The studies evaluating the forecast models or their combination were preregistered. In addition to quality assurance, this form of transparency and collaboration naturally also serves to promote trust in the results, says Schienle.

Through reuse, Open Science unfolds its true impact

oachim Gassen, Professor of Accounting and Auditing at Humboldt University of Berlin, shared his experiences from the “Open Science Data Center” (German) in his talk on “Making Open Science Matter: How can we increase the reusability and impact of research?” (PDF, German).

The center was created as part of the Collaborative Research Center “Accounting for Transparency” which is funded by the German Research Foundation. Gassen believes that there are still a number of hurdles to overcome on the way to Open Science. On the one hand, there is a lack of know-how, fear of mistakes, excessive demands, and also unresolved issues of licensing and data protection. On the other hand, there is a great interest in reusable data and tools, which is hindered by a lack of interoperability and content that is difficult to find. However, there is also the question of the impact of Open Science, or the question of what one wants to achieve with Open Science. The real impact of Open Science, according to Gassen, will ultimately unfold where others can build on the results and methods of previous research. An important, but unfortunately difficult issue is the recognition of corresponding achievements in career decisions.

Transparency and reproducibility do not end with publication

Dr Lars Vilhuber, Data Editor of the American Economic Association (AEA), reflected in his presentation “Continuous Transparency and Reproducibility in Academic Publications: Approaches and Techniques,” the experience of the AEA’s “Data and Code Availability Policy”, in effect since 2019.

With this policy, authors must provide so-called “Replication Packages” when submitting to the Association’s journals. A replication package should contain or reference the data and code underlying a publication and sufficiently describe the runtime environment. On this basis, Vilhuber’s team can perform reproducibility checks with the aim of obtaining computationally identical results. Since 2019, 1,500 publications have been checked in this way. However, Vilhuber points out limitations in this process. For example, programming errors are not checked, or complex computer environments can make computational reproducibility difficult or even impossible. Even if it is sometimes difficult to achieve today, the goal must be an all-encompassing replication package for the sake of transparency. When creating this package, authors must also think from the perspective of the reviewers, for example, what they need to carry out a reproduction (“computational empathy”), says Vilhuber. And transparency and reproducibility do not end with the journal publication; one has to get away from the concept of the final publication. For example, building replications can follow, including in the form of replication games. Vilhuber also emphasized that the implementation of replications must be better anchored in academic training.

Self-reflection must become part of economic science work

In the last presentation, “Economics needs self-reflection: Are transparency standards enough?” (PDF, German), Professor Jörg Ankel-Peters of RWI – Leibniz Institute for Economic Research and founding member of the Institute for Replication addressed the resilience of modern empirical economics. In principle, great progress has been made in implementing research transparency, Ankel-Peters said, for example in the area of data and code sharing and prespecification. But the road does not end with the creation of transparency standards, he said. A culture of meta-analysis (how can everything be put together to form a consolidated, generalized picture) and organized skepticism (methodological questioning of published results) is lacking in economics, he said. In this respect, there are signs of a “replication crisis” in economics, Ankel-Peters said. He followed this up with the question of what economic researchers actually replicate for, and brought the concept of “policing replications” into play. A “policing replication” is one that directly addresses and “challenges” a previously published paper. This can have a “regulatory effect,” Ankel-Peters said. However, the proportion of “policing replications” in published replications is very small, at 1- 3%, he said. And this type of replication is not without problems, he said, since they are both costly and highly socially and culturally sensitive. Nevertheless, Ankel-Peters called for critical self-reflection to become part of work in economics.

Continuing the discourse on Open Practices in business studies and economics

In the presentations, it became clear that economics researchers are currently quite aware of a further development of research transparency in economics and business studies. The examples presented show how Open Practices are becoming increasingly entrenched. In the discussion that followed, however, it also became clear that the journey is not yet over and that the discourse must continue, for example, on what should be achieved by creating transparency. For example, should more collaboration and reuse be achieved, should systematic quality assurance be pursued through reproducibility standards, or should targeted use of replications be the focus? In this context, a change in the work culture, for example the development of a different error culture, as well as the systematic teaching of necessary skills, also belong. In line with this, two groups have already formed in the Meet-Up to further develop the topics “Incentives, Culture, Conflict Readiness” and “Education”.

This text has been translated from German.

You may also find interesting:

• Event page with links to presentations and a selection of photos (German)
• Open Science Meet-Up: Creating the Future Together
• Open Science in Economics: Selected Findings from the ZBW Awareness Analysis 2022
• Barcamp Open Science 2022: Connecting and Strengthening the Communities