For my thesis work, I’ve been using the paid version of Rayyan to conduct a scoping review of existing literature. It’s a cloud-based platform designed specifically for systematic literature reviews that helps researchers organise and manage their review process. After using the tool for several months, I thought I’d share my thoughts on its strengths and limitations.
What Worked Well
Automated Duplicate Detection
By far, the most valuable feature I found in Rayyan was its automatic duplicate detection capability. The system identifies potential duplicates across your imported references and allows you to set your confidence threshold for what constitutes a match. I set mine to 95% confidence to ensure accuracy.
What I particularly appreciated about this feature was the transparency. Unlike many AI tools, where the “black box” problem makes it difficult to trust the results for academic work, Rayyan’s adjustable confidence settings gave me enough control and visibility to feel comfortable citing this process in my methodology.
Fast-Key Categorization
The ability to quickly categorise papers using keyboard shortcuts significantly streamlined my workflow. Rayyan allows you to:
- Tag papers with customised labels
- Document specific reasons for excluding papers from your review
- Include/exclude papers with a single keystroke
This system made the initial screening process much more efficient than traditional methods I’ve used in the past, such as spreadsheets or basic reference managers.
Meta-Study Tools
Although I didn’t end up needing them for my particular scoping review, Rayyan does offer tools for meta-analyses. For researchers conducting more quantitative systematic reviews, these might provide additional value.
Limitations and Challenges
Performance Issues
The most significant drawback I encountered was Rayyan’s performance. The system becomes noticeably sluggish after using it for an extended period in the same session. I found myself having to close the browser and clear cookies to regain reasonable performance speeds.
This performance degradation became particularly frustrating during intensive review sessions where I was processing large batches of papers at once.
Project Archiving
I discovered (somewhat unexpectedly) that Rayyan automatically archives projects after 30 days of inactivity. While you can recover archived projects, the retrieval process is time-consuming and adds unnecessary friction to the workflow.
Limited Audit Trail
When writing my methodology chapter, I struggled to reconstruct the exact sequence and parameters of my literature review process. Rayyan lacks a detailed change log or audit trail that would document:
- Exactly when which papers were imported
- Changes made to categorisations over time
- Specific criteria used for automated processes
For academic writing, where transparency and reproducibility are essential, this limitation meant I had to maintain additional documentation outside the system.
Is Rayyan Worth It?
Despite its limitations, I still found Rayyan to be more effective than trying to manage a complex literature review using general-purpose tools. The duplicate detection feature alone saved me hours of manual comparison work.
For researchers conducting systematic reviews, I would recommend Rayyan with a few caveats:
- Plan for performance management – Schedule regular breaks and browser refreshes during long sessions
- Maintain external documentation – Keep additional notes about your process for methodology writing
- Set calendar reminders – To prevent unexpected archiving, check in on your project at least once every 30 days
- Use the fast-keys – Invest time upfront to set the keyboard shortcuts as they significantly improve efficiency
Conclusion
Rayyan fills a niche in the academic workflow, addressing pain points in the systematic review process. While there’s certainly room for improvement in terms of performance and audit capabilities, its specialised features make it a valuable addition to a researcher’s toolkit, especially the duplicate detection functionality with adjustable confidence settings.
For my next literature review, I would use Rayyan again, but with more awareness of its limitations and better strategies to work around them.
Leave a Reply