MGnify's Nextflow Journey

MGnify, the metagenomics resource of EMBL-EBI, offers comprehensive analysis and integration of microbiome data. Leveraging cutting-edge bioinformatics tools and methods, MGnify enables researchers to gain deep insights into microbial communities across diverse environments. Our mission is to support the global research community by providing high-quality, accessible microbiome data and analysis services.
Over the last 10 months we have focussed on the migration of the MGnify production pipelines to Nextflow, adhering to nf-core best practices. Adopting nf-core's modern development practices and tools has progressively improved the robustness and versatility of the MGnify pipelines, allowing our team to utilize their time and development efforts more effectively. This transition has significantly enhanced our reliability, speed of development and deployment, platform support, community engagement, and overall support at EMBL-EBI.
Additionally, we are improving the automation of our services end-to-end using Prefect, with the aim of reducing the reliance on manual input. This system coordinates the execution of several pipelines and interfaces with multiple user-facing APIs and internal processes. This approach enhances the reliability and adaptability of the MGnify resource, improving its capability to scale effectively, thus serving the increasing needs of the research community.
In this work, we share our journey of integrating Nextflow into MGnify, highlighting the challenges we faced, the solutions we implemented, and the lessons we learned.