New guidance on using AI in meta-analysis
Not an easy task, but in a new note just out in the Journal of Economic Surveys we try to set a basic floor on the use of AI in meta-analysis.
https://onlinelibrary.wiley.com/doi/10.1111/joes.70105
Short version:
🔹 Human leadership. Humans direct the search, coding, and analysis, and record where they override the AI.
🔹 Human accountability. AI cannot be a co-author. If your name is on the paper, the errors are yours.
🔹 Human auditing. AI can serve as one of the coders, as long as humans audit at least 10% of screening records and 10% of coded studies (or 100 and 20, whichever is larger), and report a measure of agreement.
🔹 Human disclosure. Anything that shapes search, screening, coding, analysis, or conclusions should be disclosed, with prompts and model versions saved.
The effort was led by the amazing Nikolai Cook. It will be periodically updated at maer-net.org.
For the full discussion of how we agreed on these guidelines (scroll down to comments): https://www.maer-net.org/post/developing-guidelines-for-the-use-of-ai-in-meta-analysis-of-economics-research-guai-maer-and
