Birds of a Feather Archive

Reproducibility and Trustworthiness of Computational Research

Authors: Hakizumwami Runesha (University of Chicago), Ana Trisovic (Harvard University)

Abstract: The advancement of scientific knowledge is driven by the ability to reproduce research findings. While there is an agreement about the importance of reproducibility and we continue to see a growing number of related initiatives, most researchers, however, still do not incorporate reproducibility practices in their work. It is also not uncommon to lose years of research progress when a researcher leaves a team or graduates. Building on PEARC22 BoF discussions, this BoF will aim at democratizing reproducibility and discussing opportunities and challenges for developing active community-driven services and shared training resources for the reproducibility and trustworthiness of scientific research.

Long Description: The advancement of scientific knowledge is driven by the ability to share and reproduce research findings. Reproducible computational results are necessary for establishing trust in science. The challenge, however, is that for most scientists reproducibility practices come as an afterthought upon publication, instead of incorporating them and benefiting from them in day-to-day work. The lack of incentives and the amount of effort required to put together reproducible research artifacts could hinder trustworthiness of their results. Furthermore, it isn’t uncommon for a faculty or group to lose valuable information or years of research progress when a researcher leaves the team or graduates because of the loss of contextual knowledge or the lack of research documentation resulting in no one in the group being able to reproduce the researcher’s work. While there is no disagreement about the importance of reproducibility for scientific research and we continue to see a growing number of initiatives related to reproducibility, most researchers, however, are still not incorporating reproducibility practices in their day-to-day work. There is a need to establish a culture for researchers that fosters incorporating reproducibility from the beginning to the end of a research project. To make progress, efforts and engagement of the research community will be required, from increasing training on our campuses to providing access to the right tools and platforms. While there are many tools that capture parts of the research process, we are unaware of any tools and environments that have reached a level of success to be available in a turnkey and sustainable way or to support most disciplines. Not only do we need to continue to develop tools, but we also need computational and data platforms that will facilitate the capturing of artifacts necessary to reproduce results and that fully integrate tools and processes that support reproducibility into computational environments.

The BoF has three main goals. First, it will discuss putting into practice the recommendations of the 2019 National Academies report on “Reproducibility and Replicability in Science,” and the Working Group on Reproducibility and Sustainability for the April 2022 NSF OAC Advisory Committee on Cyber Infrastructure (ACCI). Second, we aim to create an active community that focuses on facilitating reproducibility practices as part of the research lifecycle. Third, it will review the catalog of existing reproducibility tools, discuss how we can use them, share success stories, discuss experiences of existing reproducibility efforts aiming at defining what needs to be captured to assure reproducibility and trustworthiness of scientific research. The overall goal is to identify the most complete set of reproducibility tools and develop technical support services and shared training resources for these tools and platforms. The BoF will discuss opportunities and challenges for developing such support services at our universities, how to lower barriers for capturing artifacts while doing research, how to work together to increase the training of our researchers, and ultimately how to work towards a concerted effort to build an ecosystem of tools to support reproducibility.

URL:


Back to Birds of a Feather Archive Listing