AI Research Is Becoming More Transparent as Code and Data Sharing Rapidly Grow

The researchers analyzed papers from AAAI, ICLR, ICML, IJCAI, and NeurIPS, five conferences that sit at the center of modern AI research.

Artificial intelligence research moves at an extraordinary pace, with thousands of new papers appearing every year. Keeping up with that flood of studies is challenging enough, but an even bigger question hangs over the field: can other researchers actually reproduce the results being reported?

A new study titled 'The Shift Toward Open and Reproducible AI Research', published in Communications AI & Computing, examined how AI research practices have changed over the last decade. Instead of looking at a small sample of papers, the researchers analyzed an enormous dataset of 56,800 papers from five of the world's leading AI conferences between 2014 and 2024. The findings reveal a major transformation in how AI researchers document their work and share the resources needed for others to verify their results.

The study paints a picture of a research community that has steadily embraced open science. More papers now provide code, use public datasets, explain experimental procedures, and include the technical details needed for independent teams to repeat their work.

A Decade of Change in AI Research

The researchers analyzed papers from AAAI, ICLR, ICML, IJCAI, and NeurIPS, five conferences that sit at the center of modern AI research. During the study period, publication volume exploded from around 1,200 papers in 2014 to more than 12,000 papers in 2024, reflecting the extraordinary growth of the field. Among those papers, about 92% were empirical studies involving experiments and data.

To assess reproducibility, the team examined seven key elements that make research easier to verify. These included open code, open datasets, dataset split information, pseudocode, hardware descriptions, software dependency details, and experiment setup documentation.

The results showed a clear improvement across nearly every category. In 2014, only 8% of empirical papers documented at least five of these reproducibility factors. By 2024, that figure had climbed to 43%, indicating a substantial increase in research transparency and reporting quality.

Open Code and Public Datasets Have Become the New Normal

In 2014, only 13% of empirical AI papers provided open code; by 2024, that number had risen to 69%. Public dataset use also increased during the same period, growing from 68% to 91%. The combination of open code and open datasets is especially important because previous research has shown that studies become much easier to reproduce when both are available.

The new analysis found that papers sharing both resources increased from 11% in 2014 to 64% in 2024, nearly a sixfold increase. At the same time, papers sharing neither code nor data fell from 29% to just 4%. This shift suggests that transparency is becoming part of mainstream AI research rather than a practice limited to a small group of researchers.

Reproducibility Rates Have More Than Doubled

The researchers used previous replication research to estimate how documentation improvements may have affected the ability to reproduce findings. Based on those calculations, the estimated reproducibility rate of empirical AI research increased from 28% in 2014 to 64% in 2024. Although these figures are estimates rather than the result of direct replication attempts, they indicate that AI research has become substantially easier to verify than it was a decade ago.

The number of studies likely to be reproducible grew even faster than the field itself. As AI publications expanded tenfold, the estimated number of reproducible papers increased nearly twenty-four times, rising from roughly 305 papers in 2014 to more than 7,200 papers in 2024. The researchers also noted a sharp decline in the use of private datasets. Papers relying on non-public data dropped from 32% to 9% during the study period, removing one of the biggest obstacles to independent verification.

The Improvement Started Before Reproducibility Checklists

Many leading AI conferences introduced reproducibility checklists over the past several years, encouraging authors to disclose methodological details and research artifacts.

The study investigated whether these checklists were the main reason documentation improved. Surprisingly, the answer appears to be no. The upward trend in documentation quality had already started before most checklist requirements appeared. After comparing trends before and after checklist adoption, researchers found little evidence that checklists dramatically accelerated progress.

The findings suggest that AI's growing commitment to openness emerged from a broader cultural change across the research community. Researchers increasingly embraced sharing code, datasets, and technical details even before conferences formalized those expectations.

Challenges Still Stand in the Way

Around 27% of papers that use public datasets still do not share their code. This remains one of the biggest barriers preventing independent researchers from validating published findings. Code often contains implementation details that never make it into a paper, making reproduction much harder when repositories are unavailable.

The authors also point out that complete openness is not always possible. Some datasets contain private or sensitive information, making public release inappropriate. Even so, they argue that sharing should be the default expectation whenever practical, and researchers who cannot release code or data should explain why.

Looking ahead, the team believes future scientific papers should become more machine-readable and better structured. Such improvements could help emerging AI-powered research assistants automatically verify experiments, analyze findings, and support new scientific discoveries.

AI research has grown dramatically over the past decade, bringing concerns about trust and reliability along with its rapid progress. This large-scale analysis offers encouraging evidence that the field is moving toward greater transparency. More researchers are sharing their code, relying on public datasets, and documenting their methods in ways that make independent verification possible. The road to fully reproducible AI science remains incomplete, but the direction of travel is clear; openness is steadily becoming a core part of how AI research is conducted.