# tutorials

topic: microbiome
tutorial: clinical-mp-5-data-interpretation
title: Clinical Metaproteomics 5: Data Interpretation
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["[label-TMT11]"]
content_hash: 7734127a4a0739c599bc2cece10a8783
last_modified: 2026-02-26T21:12:17.465883
zenodo_link: https://doi.org/10.5281/zenodo.10105821
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/clinical-mp-5-data-interpretation/tutorial.html

content:
The final workflow in the array of clinical metaproteomics tutorials is the data interpretation workflow. Interpreting MaxQuant data using MSstats involves applying a rigorous statistical framework to glean meaningful insights from quantitative proteomic datasets. The MaxQuant output is explored to understand data distribution and variability. Subsequent normalization helps account for systematic variations. MSstats allows the user to define the experimental design, including sample groups and conditions, to perform statistical analysis. The output provides valuable information about differential protein expression across conditions, estimates of fold changes, and associated p-values, aiding in the identification of biologically significant proteins. Furthermore, MSstats enables quality control and data visualization, ultimately enhancing our ability to draw meaningful conclusions from complex proteomic datasets. Additional tutorial material for using MaxQuant and MSstatTMT for TMT data analysis can be found at [MaxQuant and MSstats for the analysis of TMT data](https://gxy.io/GTN:T00220).

![Data-Interpretation-workflow]({% link topics/proteomics/images/clinical-mp/clinical-mp-data-interpretation.JPG %})

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/10105821/files/Annotation.tabular
>    https://zenodo.org/records/10105821/files/Comparison_Matrix.tabular
>    https://zenodo.org/records/10105821/files/MaxQuant_Evidence.tabular
>    https://zenodo.org/records/10105821/files/MaxQuant_Protein_Groups.tabular
>    https://zenodo.org/records/10105821/files/Quantified-Peptides.tabular
>
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Add to each database a tag corresponding to ...
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Import Workflow


> <hands-on-title>Running the Workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/proteomics/tutorials/clinical-mp-5-data-interpretation/workflows/WF5_Data_Interpretation_Worklow.ga" title="Data Interpretation Workflow" %}
>
> 2. Run **Workflow** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *" Quantified Peptides * "*: `Quantified-Peptides.tabular`
>    - {% icon param-file %} *" MaxQuant Protein Groups"*: `MaxQuant_Protein_Groups.tabular`
>    - {% icon param-file %} *" MaxQuant Evidence"*: `MaxQuant_Evidence.tabular`
>    - {% icon param-file %} *" Annotation.txt "*: `Annotation.tabular`
>    - {% icon param-file %} *" Comparison Matrix "*: `Comparison_Matrix.tabular`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}

## Taxonomic analysis with **Unipept**
Unipept serves as a vital bioinformatics platform for the analysis of mass spectrometry-based shotgun proteomics data, especially in the study of microbial communities. Its primary utility lies in taxonomic and functional analyses, enabling researchers to identify and quantify microorganisms within diverse environments. The platform facilitates comparative studies across samples, conditions, or time points, shedding light on the dynamic responses of microbial communities to environmental changes. Unipept integrates with public databases like UniProt, ensuring access to comprehensive and updated information for annotations. Being community-driven and open source, Unipept fosters collaboration and transparency, with a user-friendly web interface that accommodates researchers of varying bioinformatics expertise. In essence, Unipept is an invaluable resource, offering tools for the exploration of metaproteomic data and contributing to advancements in our understanding of microbial ecology.


> <hands-on-title> Unipept 5.0 </hands-on-title>
>
>
> 1. {% tool [Unipept](toolshed.g2.bx.psu.edu/repos/galaxyp/unipept/unipept/4.5.1) %} with the following parameters:
>    - *"Unipept application"*: `peptinfo: Tryptic peptides and associated EC and GO terms and lowest common ancestor taxonomy`
>    - *"Peptides input format"*: `tabular`
>        - {% icon param-file %} *"Tabular Input Containing Peptide column"*: `output` (Input dataset)
>        - *"Select column with peptides"*: `c1`
>    - *"Match input peptides by"*: `Match to the full input peptide`
>    - *"Choose outputs"*: `Tabular with one line per peptide` `JSON Taxomony Tree` `Peptide GO terms in normalized tabular` `Peptide InterPro entries in normalized tabular` `Peptide EC terms in normalized tabular` `JSON EC Coverage Tree`
>
{: .hands_on}

![Data-Interpretation with Unipept]({% link topics/proteomics/images/clinical-mp/clinical-mp-data-interpretation-figure2.jpg %})

## Extraction of Microbial Sequences

> <hands-on-title> Extract Microbial sequences with Select </hands-on-title>
>
> 1. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `output` (Input dataset)
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_HUMAN)|(_REVERSED)|(REV_)|(CON)|(con)`
>    - *"Keep header line"*: `Yes`
>
>
{: .hands_on}


> <hands-on-title> Select sequences matching "HUMAN" </hands-on-title>
>
> 1. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `output` (Input dataset)
>    - *"the pattern"*: `(_HUMAN)`
>    - *"Keep header line"*: `Yes`
>
>
{: .hands_on}

> <hands-on-title> Select out reverse and contaminants </hands-on-title>
>
> 1. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `out_file1` (output of **Select** {% icon tool %})
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_REVERSED)|(REV_)|(CON)|(con)`
>    - *"Keep header line"*: `Yes`
>
>
{: .hands_on}


# MSstats TMT
MSstats TMT(Tandem Mass Tag) is a computational tool designed for the robust statistical analysis of mass spectrometry-based quantitative proteomics data using TMT labeling. TMT is a widely used method for multiplexed quantitative proteomics, enabling simultaneous identification and quantification of proteins across multiple samples. MSstats TMT plays a crucial role in this process by providing a statistical framework for analyzing TMT data, and facilitating accurate and reliable protein abundance measurements. The tool offers a range of features, including quality control, normalization, and statistical modeling, allowing researchers to identify differentially expressed proteins with confidence. MSstats TMT is particularly valuable in large-scale studies where quantifying protein expression across multiple conditions is essential for understanding complex biological processes. Its application contributes to advancing our understanding of proteomic changes in response to various experimental conditions or perturbations.

## Statistical Analysis of Microbial proteins with **MSstatsTMT**

> <hands-on-title> MSstatsTMT </hands-on-title>
>
> 1. {% tool [MSstatsTMT](toolshed.g2.bx.psu.edu/repos/galaxyp/msstatstmt/msstatstmt/2.0.0+galaxy1) %} with the following parameters:
>    - *"Input Source"*: `MaxQuant`
>        - {% icon param-file %} *"evidence.txt - feature-level data"*: `output` (Input dataset)
>        - {% icon param-file %} *"proteinGroups.txt"*: `out_file1` (output of **Select Microbial** {% icon tool %})
>        - {% icon param-file %} *"annotation.txt"*: `output` (Input dataset)
>    - In *"Plot Output Options"*:
>        - *"Select protein IDs to draw plots"*: `generate all plots for each protein`
>    - *"Compare Groups"*: `Yes`
>        - *"Use comparison matrix?"*: `Yes`
>            - {% icon param-file %} *"Comparison Matrix"*: `output` (Input dataset)
>        - *"Select outputs"*: `Group Comparison` `MSstats Volcano Plot` `MSstats Comparison Plot`
>        - In *"Comparison Plot Options"*:
>            - *"Select protein IDs to draw plots"*: `generate all plots for each protein`
>            - *"Select comparisons to draw plots"*: `Generate all plots for each comparison`
>        - *"Select outputs"*: `MSstatsTMT summarization log` `MSstatsTMT summarization MSstats` `Protein abundance`
>
>
{: .hands_on}


## Statistical Analysis of Human proteins with **MSstatsTMT**

> <hands-on-title> MSstatsTMT</hands-on-title>
>
> 1. {% tool [MSstatsTMT](toolshed.g2.bx.psu.edu/repos/galaxyp/msstatstmt/msstatstmt/2.0.0+galaxy1) %} with the following parameters:
>    - *"Input Source"*: `MaxQuant`
>        - {% icon param-file %} *"evidence.txt - feature-level data"*: `output` (Input dataset)
>        - {% icon param-file %} *"proteinGroups.txt"*: `out_file1` (output of **Select HUMAN** {% icon tool %})
>        - {% icon param-file %} *"annotation.txt"*: `output` (Input dataset)
>    - In *"Plot Output Options"*:
>        - *"Select protein IDs to draw plots"*: `generate all plots for each protein`
>    - *"Compare Groups"*: `Yes`
>        - *"Use comparison matrix?"*: `Yes`
>            - {% icon param-file %} *"Comparison Matrix"*: `output` (Input dataset)
>        - *"Select outputs"*: `Group Comparison` `MSstats Volcano Plot` `MSstats Comparison Plot`
>        - In *"Comparison Plot Options"*:
>            - *"Select protein IDs to draw plots"*: `generate all plots for each protein`
>            - *"Select comparisons to draw plots"*: `Generate all plots for each comparison`
>        - *"Select outputs"*: `MSstatsTMT summarization log` `MSstatsTMT summarization MSstats` `Protein abundance`
>
>
{: .hands_on}

The MSstats output typically includes essential information such as estimated fold changes, p-values, and other statistical measures that help identify differentially expressed proteins across experimental conditions or sample groups. It provides a clear picture of the variations in protein expression levels, aiding in the prioritization of biologically relevant targets. MSstats output also often includes visualizations and quality control metrics, making it a valuable resource for researchers in their quest to extract meaningful insights from complex proteomic datasets and understand the underlying biology of their experiments.
Example of our data interpretation:

![Data-Interpretation results with MSstats]({% link topics/proteomics/images/clinical-mp/clinical-mp-data-interpretation-figure3.jpg %})


# Conclusion
With the completion of this tutorial, you have successfully completed the clinical metaproteomics tutorials.

In conclusion, clinical metaproteomics tutorials represent an essential gateway to harnessing the power of advanced proteomic techniques in the realm of clinical research and applications. These bioinformatics tutorials serve as valuable guides for understanding the intricacies of metaproteomic workflows, from  data analysis to interpretation. By providing comprehensive knowledge and practical insights, they equip researchers and clinicians with the tools necessary to explore the rich diversity of the microbiome and its impact on health and disease or environment. As metaproteomic techniques continue to evolve and integrate with clinical practice, we hope these tutorials will be instrumental in shaping clinical research.


topic: microbiome
tutorial: metaplasmidome_query
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 4845b34a7f760a2d867a709a9614de05
last_modified: 2026-02-26T21:12:17.308019
zenodo_link: https://zenodo.org/records/14501567
workflows_json: []
title:
Query an annotated mobile genetic element database to identify and annotate genetic elements (e.g. plasmids) in metagenomics data

url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/metaplasmidome_query/tutorial.html

content:
Identifying and annotating mobile genetic elements (MGE) in metagenomic data can be tricky. To facilitate the process, we can use existing and custom annotated mobile genetic element database.

The *metaplasmidome* refers to the collection of all plasmids present within a given environment, typically identified through metagenomic sequencing. Plasmids are extrachromosomal genetic elements that often carry genes associated with antibiotic resistance, virulence, or metabolic functions, making them crucial for microbial adaptability. In the context of metagenomics, plasmids are identified alongside chromosomal DNA.

A common step in metaplasmidome analysis is searching sequencing reads against known plasmid databases to detect plasmid sequences within a metagenome, allowing researchers to map the diversity and abundance of plasmids in various environments. 

In this tutorial, we use a metaplasmidome database built from public air metagenomes and query it with assembled air metagenome data. A similar approach can be used for other mobile genetic elements.

> <details-title>How was built the air metaplasmidome database?</details-title>
>
> The air metaplasmidome is available from Zenodo ({% cite debroas_2024_11124657 %}) and was built from metagenomic data selected in Web of Science (Clarivate) on October 2022 using keywords: `txid655179[Organism:noexp] AND metagenome [Filter]; AIR Metagenome; Air microbiome; Troposphere; Aerosol; Atmosphere`. Data were manually curated to remove sequencing originated from metabarcoding data (i.e., 16S). The assembled data supplied by MetaSUB consortium ({% cite danko2021global %}) when available was used for air metagenome in the built environments. 
>
> Plasmid contents were predicted using the assembled data. Metagenomes sequencing by Illumina (paired-illumina reads) were assembled by using MEGAHIT 1.2.9 with metalarge option ({% cite li2015 %}) after cleaning data with bbduk2 (qtrim=rl trimq=28 minlen=25 maq=20 ktrim=r k=25 mink=11 and a list of adaptators to remove) from [bbtools suite](https://jgi.doe.gov/data-and-tools/software-tools/bbtools/)  
>
> Plasmids were predicted for each assembling by using PlasSuite scripts describing in-depth in Hilpert et al. {% cite hilpert2021reconstruction %} {% cite hennequin2022plasmidome %} and available on [GitHub](https://github.com/meb-team/PlasSuite/). Briefly, contigs were analyzed using both reference-based and reference-free approaches. The databases employed included those for chromosomes (archaea and bacteria) and plasmids from NCBI, as well as the MOB-suite tool ({% cite robertson2018mob %}), SILVA ({% cite quast2012silva %}) and phylogenetic markers harbored by chromosomes ({% cite wu2013systematic %}). Two reference-free methods were applied to contigs that were not affiliated with chromosomes (discarded) or plasmids (retained in the first step): PlasFlow ({% cite krawczyk2018plasflow%}) and PlasClass ({% cite pellow2020plasclass %}). Viruses were removed by using [viralVerify](https://github.com/ablab/viralVerify) ({% cite antipov2020metaviral %}) that provides in parallel provide plasmid/non-plasmid classification.
> 
> Eukaryotes contaminants were removed by aligning the sequences against NT databases and human chromosomes (GRCh38) with minimap2 with -x asm5 option ({% cite li2018minimap2 %}). Contigs mapping with an identity of 95% and a coverage of 80% were removed.  the final plasmidome set was clustered by mmseqs ({% cite mirdita2019mmseqs2 %}) with 80% of coverage and 90% of identity (--min-seq-id 0.90 -c 0.8 --cov-mode 1 --cluster-mode 2 --alignment-mode 3 --kmer-per-seq-scale 0.2). 
{: .details}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Galaxy and data preparation

Any analysis should get its own Galaxy history. So let's start by creating a new history and import the data (plasmidome database and query metagenomes) into it.

> <hands-on-title>Prepare Galaxy and data</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 3. Import the metaplasmidome reference database from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library
>
>    ```
>    https://zenodo.org/records/14501567/files/air_metaplasmidome.fasta.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 4. Rename `Air plasmidome database`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 5. Import the reads to query against the reference database from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library
>
>    ```
>    https://zenodo.org/records/14501567/files/air_metagenome_assemblies.fasta.gz
>    ```
>
> 6. Rename `Air metagenomes`
{: .hands_on}


> <question-title></question-title>
>
> 1. How many reads are in the query dataset?
> 2. How many plasmid sequences are in the metaplasmidome database?
>
> > <solution-title></solution-title>
> >
> > 1. 16,637 sequences
> > 2. 674,495 sequences
> {: .solution}
{: .question}


# Read mapping against the metaplasmidome database

> <hands-on-title>Map reads against metaplasmidome database</hands-on-title>
>
> 1. {% tool [Map with minimap2](toolshed.g2.bx.psu.edu/repos/iuc/minimap2/minimap2/2.28+galaxy0) %} with the following parameters:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>        - {% icon param-file %} *"Use the following dataset as the reference sequence"*: `metaplasmidome database`
>    - *"Single or Paired-end reads"*: `Single`
>        - {% icon param-file %} *"Select fastq dataset"*: `query`
>        - *"Select a profile of preset options"*: `Long assembly to reference mapping (-k19 -w19 -A1 -B19 -O39,81 -E3,1 -s200 -z200 --min-occ-floor=100). Typically, the alignment will not extend to regions with 5% or higher sequence divergence. Only use this preset if the average divergence is far below 5%. (asm5)`
>    - In *"Set advanced output options"*:
>        - *"Select an output format"*: `PAF`
>
{: .hands_on}

PAF is the default output format of minimap2. It is TAB-delimited with each line consisting of the following predefined fields:
1. Query sequence name
2. Query sequence length
3. Query start coordinate (0-based)
4. Query end coordinate (0-based)
5. `+` if query/target on the same strand and `-` if opposite
6. Target sequence name
7. Target sequence length
8. Target start coordinate on the original strand
9. Target end coordinate on the original strand
10.	Number of matching bases in the mapping
11.	Number bases, including gaps, in the mapping
12.	Mapping quality

> <question-title></question-title>
>
> 1. How many lines are in the file?
> 2. Can a query sequence be found several times in the target?
> 3. Are all alignments of good quality?
>
> > <solution-title></solution-title>
> >
> > 1. 29,796 lines
> > 2. There are 16,637 query sequences. Several query sequences are found several times in the file: SRR17300492_75807 is found 16 times. It means they have been mapped to several locations or sequences in the target.
> > 3. The first alignment (SRR17300492_75807 mapping on SRR17300667-707) has a score of 60, the highest Phred score. The third alignment has a score of 0, so not good.
> >
> {: .solution}
>
{: .question}

# Exploration of the mapping results

Let's look at the distribution of the mapping score (column 12)

> <hands-on-title>Plot score distribution</hands-on-title>
>
> 1. Change datatype to tabular
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
> 
> 2. {% tool [Cut columns from a table](Cut1) %} with:
>    - *"Cut columns"*: `c12`
>    - *"Delimited by"*: `tab`
>    - {% icon param-file %} *"From"*: Output of **Map with minimap2** {% icon tool %}
>
> 2. {% tool [Histogram with ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_histogram/ggplot2_histogram/3.4.0+galaxy0) %} with the following parameters:
>    - *"Input should have column headers - these will be the columns that are plotted * "*: Output of **Cut** {% icon tool %}
>    - *"Label for x axis"*: `Score`
>    - *"Label for y axis"*: `Distribution`
>    - In *"Advanced Options"*:
>      - *"Legend options"*: `Hide legend`
>
{: .hands_on}

![Distribution of mapping quality score (on x-axis, value between 0 and 60) with density (y-axis)](./images/mapping_score_histogram.png)

> <question-title></question-title>
>
> Are all alignments of good quality?
>
> > <solution-title></solution-title>
> >
> > There is a pic at 60 but many alignments have a score below 40.
> >
> {: .solution}
>
{: .question}

We should remove alignments with a score below 40. Before that, let's look at the plasmid coverage. We first need to compute it, i.e. the ratio between the number of matching bases in the mapping (Column 10) and plasmid length (target sequence length - Column 7).

> <hands-on-title>Compute plasmid coverage</hands-on-title>
> 1. {% tool [Cut columns from a table](Cut1) %} with:
>    - *"Cut columns"*: `c1-c12`
>    - *"Delimited by"*: `tab`
>    - {% icon param-file %} *"From"*: Output of **Map with minimap2** {% icon tool %}
>
> 2. {% tool [Compute on rows](toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1) %} with the following parameters:
>    - *"Input file"*: Output of **Cut** {% icon tool %}
>    - In *"Expressions"*:
>      - *"Add expression"*: `float(c10)/float(c7)`
>
> 3. Rename `Mapping stats + plasmid coverage`
{: .hands_on}

A new column has been added (column 13) with the plasmid coverage. Let's now plot its distribution.

> <hands-on-title>Plot plasmid coverage distribution</hands-on-title>
> 1. {% tool [Cut columns from a table](Cut1) %} with:
>    - *"Cut columns"*: `c13`
>    - *"Delimited by"*: `tab`
>    - {% icon param-file %} *"From"*: `Mapping stats + plasmid coverage`
>
> 2. {% tool [ Histogram with ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_histogram/ggplot2_histogram/3.4.0+galaxy0) %} with the following parameters:
>    - *"Input should have column headers - these will be the columns that are plotted * "*: Output of **Cut** {% icon tool %}
>    - *"Label for x axis"*: `Plasmid coverage`
>    - *"Label for y axis"*: `Distribution`
>    - *"Bin width for plotting"*: `0.1`
>    - In *"Advanced Options"*:
>      - *"Legend options"*: `Hide legend`
>
{: .hands_on}

![Distribution of plasmid coverage (on x-axis, value between 0 and 1) with density (y-axis)](./images/plasmid_coverage_histogram.png)

> <question-title></question-title>
>
> 1. What is the distribution of the plasmid coverage?
> 2. What could be a good threshold to filter?
>
> > <solution-title></solution-title>
> >
> > 1. There are 2 peaks: one around 0 (i.e. no plasmid coverage) that slowly decreases until 0.8 and a pic at 1 (full plasmid coverage)
> > 2. 0.8 seems to be a breaking point and could be a good value to filter.
> >
> {: .solution}
>
{: .question}

# Filtering

We will now filter the alignment to keep only the ones with a plasmid coverage (column 13) of at least 0.8, i.e. a read mapping to a plasmid covering at least 80% of the plasmid.

> <hands-on-title>Filter alignments based on plasmid coverage</hands-on-title>
>
> 1. {% tool [Filter data on any column using simple expressions](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `Mapping stats + plasmid coverage`
>    - *"With following condition"*: `c13>=0.8`
>
> 2. Rename `Alignments with plasmid coverage >= 0.8`
{: .hands_on}

> <question-title></question-title>
>
> 1. How many lines have been kept?
> 2. Which percentage of lines does that correspond to?
> 3. What does the distribution of the mapping score look like for these alignments?
> 4. Is there any extra filter we should do on the data?
>
> > <solution-title></solution-title>
> >
> > 1. 5,577 (over the 29,796)
> > 2. 18.73%
> > 3. To plot the distribution of the mapping score for filtered alignments, we need to run the series of tools as done earlier:
> >    
> >    1. {% tool [Cut columns from a table](Cut1) %} with:
> >        - *"Cut columns"*: `c12`
> >        - *"Delimited by"*: `tab`
> >        - {% icon param-file %} *"From"*: `Alignments with plasmid coverage >= 0.8`
> >
> >     2. {% tool [Histogram with ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_histogram/ggplot2_histogram/3.4.0+galaxy0) %} with the following parameters:
> >        - *"Input should have column headers - these will be the columns that are plotted * "*: Output of **Cut** {% icon tool %}
> >        - *"Label for x axis"*: `Score`
> >        - *"Label for y axis"*: `Distribution`
> >        - In *"Advanced Options"*:
> >          - *"Legend options"*: `Hide legend`
> >    
> >     ![Distribution of mapping quality score (on x-axis, value between 0 and 60) with density (y-axis)](./images/mapping_score_histogram_after_coverage_filtering.png)
> >
> >     Most of the alignments are of good quality (pic at 60). There is also a pic at 0, meaning that some alignments are of bad quality
> >    
> > 4. It might be good to add a filter on the score.
> {: .solution}
>
{: .question}

After filtering on the plasmid coverage, we also add a filter on the score to be sure that we keep only the best alignments (i.e. a score - column 12 - of at least 40).

> <hands-on-title>Filter alignments based on score</hands-on-title>
>
> 1. {% tool [Filter data on any column using simple expressions](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `Alignments with plasmid coverage >= 0.8`
>    - *"With following condition"*: `c12>=40`
>
> 2. Rename `Alignments with plasmid coverage >= 0.8 and score >= 40`
{: .hands_on}

> <question-title></question-title>
>
> 1. How many lines have been kept?
> 2. Which percentage of lines does that correspond to?
>
> > <solution-title></solution-title>
> >
> > 1. 5,249 (over the 5,577)
> > 2. 94.12%
> {: .solution}
>
{: .question}

# Extract sequences mapping to plasmids

Let's now extract the sequences mapping to plasmids (coverage higher than 80%).

First, we need to extract the names of the reads (in `air metagenomes`) mapping to plasmids in the metaplasmidome database. For that, we need to extract column 1 (query names, i.e. names of reads in `Air metagenomes`) and 6 (reference names, i.e. names of sequences in `Air plasmidome database`).

> <hands-on-title>Get sequences matching to plasmids</hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1,c6`
>    - {% icon param-file %} *"From"*: `Alignments with plasmid coverage >= 0.8 and score >= 40`
>
> 2. Rename to `Names for alignments with plasmid coverage >= 0.8 and score >= 40`
>
> 2. {% tool [Sort](sort1) %} with the following parameters:
>    - {% icon param-file %} *"Sort Dataset"*: output of **Cut** {% icon tool %}
>    - *"on column"*: `c2`
{: .hands_on}

Second, we need to convert the `Air metagenomes` FASTA file to a tabular so we can join it with `Names for alignments with plasmid coverage >= 0.8 and score >= 40`

> <hands-on-title>Get query sequence as a table</hands-on-title>
>
> 1. {% tool [Convert FASTA to Tabular](CONVERTER_fasta_to_tabular) %} with the following parameters:
>    - {% icon param-file %} *"Fasta file"*: `Air metagenomes`
>
> 2. Rename to `Air metagenome as table`
{: .hands_on}

Let's now join the 2 datasets on the names of the reads in `air metagenomes` so column 1 in `Names for alignments with plasmid coverage >= 0.8 and score >= 40` and column 1 in `Air metagenome as table`.

> <hands-on-title>Get sequences matching to plasmids</hands-on-title>
>
> 1. {% tool [Join two Datasets](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: output of **Sort** {% icon tool %}
>    - *"using column"*: `Column: 1`
>    - {% icon param-file %} *"with"*: output of **Convert FASTA to Tabular** {% icon tool %}
>    - *"and column"*: `Column: 1`
>    - *"Fill empty columns"*: `No`
>
{: .hands_on}

We have now a table with 4 columns: names of the reads in `air metagenomes`, names of sequences in the reference database, names of the reads in `air metagenomes`, sequence of reads in `air metagenomes`. We will now create 2 outputs:
- A table with metaplasmidome database sequence name, metagenomic sequence names and the metagenomic sequences
- A FASTA file with metagenomic sequences

## Create the table with air metagenomes identified as plasmids

Let's remove the duplicated column (column 3) and reorganize the columns

> <hands-on-title>Keep non duplicated columns</hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c2,c1,c4`
>    - {% icon param-file %} *"From"*: output of **Join two Datasets** {% icon tool %}
{: .hands_on}

Let's now add column names to the generated table.

> <hands-on-title>Add column names</hands-on-title>
>
> 1. {% tool [Add Header](toolshed.g2.bx.psu.edu/repos/estrain/add_column_headers/add_column_headers/0.1.3) %} with the following parameters:
>    - *"List of Column headers"*: `Metaplasmidome database sequence name,Metagenomic sequence name,Metagenomic sequence`
>    - {% icon param-file %} *"Data File (tab-delimted)"*: output of **Replace Text** {% icon tool %}
>
>
> 3. Rename the output to `Air metagenomes identified as plasmids`
{: .hands_on}

## Create the FASTA file with air metagenomes identified as plasmids

We can now generate a FASTA file with the identified sequences.

From the output of **Join two Datasets** {% icon tool %}, let's keep only metagenomic sequence names and the metagenomic sequences and remove duplicated sequences

> <hands-on-title>Keep unique metagenomic sequences</hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1,c4`
>    - {% icon param-file %} *"From"*: output of **Join two Datasets** {% icon tool %}
> 2. {% tool [Sort](sort1) %} with the following parameters:
>    - {% icon param-file %} *"Sort Dataset"*: output of **Cut** {% icon tool %}
>    - *"on column"*: `Column: 1`
>    - *"with flavor"*: `Alphabetical sort`
> 3. {% tool [Unique occurrences of each record](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sorted_uniq/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to scan for unique values"*: output of **Sort** {% icon tool %}
{: .hands_on}

> <question-title></question-title>
>
> How many lines have been kept?
>
> > <solution-title></solution-title>
> >
> > 4,055 (over 5,249)
> >
> {: .solution}
>
{: .question}

We have now a table with unique metagenomic sequences. Let's transform it into a FASTA file.

> <hands-on-title>Convert to a FASTA file</hands-on-title>
>
> 1. {% tool [Tabular-to-FASTA](toolshed.g2.bx.psu.edu/repos/devteam/tabular_to_fasta/tab2fasta/1.1.1) %} with the following parameters:
>    - {% icon param-file %} *"Tab-delimited file"*: output of **Unique occurrences of each record** {% icon tool %}
>    - *"Title column(s)"*: `Column: 1`
>    - *"Sequence column"*: `Column: 2`
>
> 2. Rename the output to `Air metagenome sequences identified as plasmids`
{: .hands_on}

# Annotate features on the identified plasmids
 
Let's now annotate features on the identified plasmids. For that, we will use the annotation of the air metaplasmidome sequences that have been done with Prokka (citation) and assume that the annotations are similar for the identified plasmids.

We will import the GFF generated by Prokka.

> <hands-on-title>Import the GFF with metaplasmidome reference database annotation</hands-on-title>
>
> 1. Import the metaplasmidome reference database annotation from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>
>    ```
>    https://zenodo.org/records/14501567/files/air_metaplasmidome_annotations.gff
>    ```
>
> 2. Inspect it.
{: .hands_on}

This file is a GFF: it describes genes and other features of DNA, RNA and protein sequences. It is a tab-delimited file with 9 fields per line:

1. **seqid**: The name of the sequence where the feature is located. 
2. **source**: The algorithm or procedure that generated the feature. 
3. **type**: The feature type name, like "gene" or "exon". 
4. **start**: Genomic start of the feature, with a 1-base offset. 
5. **end**: Genomic end of the feature, with a 1-base offset. 
6. **score**: Numeric value that generally indicates the confidence of the source in the annotated feature. 
7. **strand**: Single character that indicates the strand of the feature.
8. **phase**: Phase of CDS features.
9. **attributes**: A list of tag-value pairs separated by a semicolon with additional information about the feature. 

The **seqid** corresponds here to the ID of the sequences in the metaplasmidome reference database. So to filter, we need to compare `Metaplasmidome database sequence name` in `Air metagenomes identified as plasmids`
to **seqid** by joining the 2 datasets on column 1.

The file has with ~85 Million lines but many are comments (lines starting with `##`).

> <question-title></question-title>
>
> How many features are in the GFF file?
>
> > <solution-title></solution-title>
> >
> > 2,951,015 features
> >
> > To get this number, we run {% tool [Count GFF Features](toolshed.g2.bx.psu.edu/repos/devteam/count_gff_features/count_gff_features/0.2) %} with the following parameters:
> >  - {% icon param-file %} *"GFF Dataset to Filter"*: imported GFF
> >
> {: .solution}
>
{: .question}

Let's now filter the GFF to keep only information related to the sequences matching to plasmids. 
For that, we join the GFF on the SeqID column (column 1) with the `Air metagenomes identified as plasmids` file on the ` Metaplasmidome database sequence name` (Column 1)

> <hands-on-title>Extract information about the sequences matching to plasmids</hands-on-title>
>
> 1. {% tool [Join two Datasets](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: imported GFF
>    - *"using column"*: `Column: 1`
>    - {% icon param-file %} *"with"*: `Air metagenomes identified as plasmids`
>    - *"and column"*: `Column: 1`
>    - *"Fill empty columns"*: `No`
>
> 2. Inspect the generated file
{: .hands_on}

> <question-title></question-title>
>
> 1. How many lines have been kept?
> 2. Why is there more lines than in the `Air metagenomes identified as plasmids` file?
> 2. What are the columns? 
> 3. Which columns should we keep if we want to keep the Metagenomic sequence name, the feature and the attributes?
>
> > <solution-title></solution-title>
> >
> > 1. 26k+ lines
> > 2. Some sequences (e.g. SRR17300493-2380) have several lines: several features (CDS, rRNA, etc) annotated on it
> > 3. The 9 columns of the GFF file + the 3 columns of the `Air metagenomes identified as plasmids` file
> > 4. Columns to keep:
> >    1. Metagenomic sequence name (Column 11)
> >    2. Feature (Column 3)
> >    3. Attributes (Column 9)
> {: .solution}
>
{: .question}

Let's now cut the columns.

> <hands-on-title>Cut and filter </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c11,c3,c9`
>    - {% icon param-file %} *"From"*: output of **Join two Datasets** {% icon tool %}
>
> 2. Inspect the generated file
{: .hands_on}

> <question-title></question-title>
>
> What are the different identified features?
>
> > <solution-title></solution-title>
> >
> > Using {% tool [Group data by a column](Grouping1) %} to group and count on 2nd column, we find 26,329 CDS and 307 tRNA.
> >
> {: .solution}
>
{: .question}

## Extract the CDS

Let's filter to keep only the CDS and extract the gene names.

> <hands-on-title>Keep CDS</hands-on-title>
>
> 1. {% tool [Filter](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: output of **Cut** {% icon tool %}
>    - *"With following condition"*: `c2=='CDS'`
>
> 2. Inspect the generated file
{: .hands_on}

We have now a file with 26,329 lines. The 2nd column is not useful so we will remove it. Column 3 (**atributes** in GFF) lists tag-value pairs separated by a semicolon with additional information about the feature.
The first lines seems to be hypothetical proteins. If we scroll down, we can find some annotated genes (with `gene` keyword).

It would be good to create a tabular file with:
1. Metagenomic sequence name
2. Gene ID
3. Gene name
4. Gene product

As not all genes are annotated (with `gene` keyword), we first need to split the file between hypothetical proteins and annotated genes and process the two generated files independently.

> <hands-on-title>Keep CDS</hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1,c3`
>    - {% icon param-file %} *"From"*: output of **Filter** {% icon tool %}
>
> 2. {% tool [Filter by keywords and/or numerical value](toolshed.g2.bx.psu.edu/repos/proteore/proteore_filter_keywords_values/MQoutputfilter/2021.04.19.1) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: output of last **Cut** {% icon tool %}
>    - *"Does file contain header? "*: `No`
>    - In *Filter by keywords*:
>      - *"Column number on which to apply the filter"*: `c2`
>      - In `Enter keywords`:
>        - `copy/paste`
>        - *"Copy/paste keywords to find (keep or discard)"*: `gene=`
>
> 2. Inspect the generated files
{: .hands_on}

> <question-title></question-title>
>
> 1. How many lines have been kept (non-hypothetical proteins) and how many have been discarded (hypothetical proteins)?
> 2. Which information do we have for the 2nd identified gene?
>
> > <solution-title></solution-title>
> >
> > 1. There are:
> > - 25,001 hypothetical proteins CDS (lines in the discarded lines file)
> > - 1,329 annotated CDS (lines in the other file)
> > 
> > 2. For the 2nd gene, the column 3 is `ID=BAFOEJEB_01977;eC_number=1.8.5.7;Name=yqjG_1;dbxref=COG:COG0435;gene=yqjG_1;inference=ab initio prediction:Prodigal:2.6,similar to AA sequence:UniProtKB:P42620;locus_tag=BAFOEJEB_01977;product=Glutathionyl-hydroquinone reductase YqjG`, which mean:
> >    - Gene name is `yqjG_1`
> >    - The product is `Glutathionyl-hydroquinone reductase YqjG`
> >
> {: .solution}
>
{: .question}

Let's extract gene ID, gene name and the product in different columns from the annotated gene output and add a header to the file.

> <hands-on-title>Prepare the annotated CDS file</hands-on-title>
>
> 1. Change annotated CDS output of **Filter by keywords** {% icon tool %} datatype to tabular
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
> 2. {% tool [Replace Text in a specific column](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: output of **Filter by keywords** {% icon tool %}
>    - In *Replacement*:
>      - *"in column"*: `Column: 2`
>      - *"Find pattern"*: `ID=([^;]*);.*;gene=([^;]*).*;product=([^;]*).*`
>      - *"Replace with"*: `\\1\t\\2\t\\3`
>
> 3. {% tool [Add Header](toolshed.g2.bx.psu.edu/repos/estrain/add_column_headers/add_column_headers/0.1.3) %} with the following parameters:
>    - *"List of Column headers"*: `Metagenomic sequence name,Gene ID,Gene name,Gene product`
>    - {% icon param-file %} *"Data File (tab-delimited)"*: output of **Replace Text** {% icon tool %}
>
{: .hands_on}

Let's now prepare the hypothetical protein CDS file

> <hands-on-title>Prepare the hypothetical CDS gene file</hands-on-title>
>
> 1. Change hypothetical CDS gene output of **Filter by keywords** {% icon tool %} datatype to tabular
>
> 2. {% tool [Replace Text in a specific column](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: output of **Filter by keywords** {% icon tool %}
>    - In *Replacement*:
>      - *"in column"*: `Column: 2`
>      - *"Find pattern"*: `ID=([^;]*);.*`
>      - *"Replace with"*: `\\1\t\t`
>
{: .hands_on}

We can now merge both files.

> <hands-on-title>Concatenate the annotated and hypothetical protein gene files</hands-on-title>
>
> 1. {% tool [Concatenate datasets](cat1) %} with the following parameters:
>    - {% icon param-file %} *"Concatenate Dataset"*: output of **Add Header** {% icon tool %}
>    - In *Dataset*:
>      - {% icon param-file %} *"Select"*: output of last **Replace Text** {% icon tool %}
>
> 2. Rename `CDS in metagenomes identified as plasmids`
{: .hands_on}

## Add KO and PFAM annotation

Let's expand annotation with **[KO](https://www.genome.jp/kegg/ko.html) (KEGG Orthology)** ({% cite kanehisa2016kegg%}), a database of molecular functions
represented in terms of functional orthologs, and **[PFAM](http://pfam.xfam.org/)** ({% cite mistry2021pfam %}), a large collection of protein families. The files are 

> <hands-on-title>Import KO and PFAM annotations</hands-on-title>
>
> 1. Import the KO and PFAM annotations from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>
>    ```
>    https://zenodo.org/records/14501567/files/KOsignificatifNR.tsv
>    https://zenodo.org/records/14501567/files/pfamsignificatifNR.tsv
>    ```
>
> 2. Inspect both files
{: .hands_on}

Both are tabular files with several columns including the gene ID, KO/PFAM ID, and some extra annotation.
Let's now join the files with `Annotated genes in air metagenomes sequences identified as plasmids` to extend the annotations using gene ID.

> <hands-on-title>Join with KO and PFAM annotation files</hands-on-title>
>
> 1. {% tool [Join two Datasets](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: `Annotated genes in air metagenomes sequences identified as plasmids`
>    - *"using column"*: `Column: 2`
>    - {% icon param-file %} *"with"*: imported KO file
>    - *"and column"*: `Column: 2`
>    - *"Keep lines of first input that do not join with second input"*: `Yes`
>
> 2. Inspect the generated file
>
> 3. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1-c4,c7,c11`
>    - {% icon param-file %} *"From"*: output of **Join** {% icon tool %}
>
> 4. {% tool [Join two Datasets](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: output of **Cut** {% icon tool %}
>    - *"using column"*: `Column: 2`
>    - {% icon param-file %} *"with"*: imported PFAM file
>    - *"and column"*: `Column: 1`
>    - *"Keep lines of first input that do not join with second input"*: `Yes`
>
> 5. Inspect the generated file
>
> 6. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1-c6,c9,c10,c25`
>    - {% icon param-file %} *"From"*: output of **Join** {% icon tool %}
>
> 7. {% tool [Select last lines from a dataset (tail)](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_tail_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Text file"*: output of **Cut** {% icon tool %}
>    - *"Operation"*: `Keep everything from lines on`
>    - *"Number of lines"*: `2`
>
> 8. {% tool [Add Header](toolshed.g2.bx.psu.edu/repos/estrain/add_column_headers/add_column_headers/0.1.3) %} with the following parameters:
>    - *"List of Column headers"*: `Metagenomic sequence name,Gene ID,Gene name,Gene product,KO ID,KO annotation,PFAM name,PFAM ID,PFAM annotation`
>    - {% icon param-file %} *"Data File (tab-delimited)"*: output of **Cut** {% icon tool %}
>
> 9. Rename `CDS in metagenomes identified as plasmids + KO + PFAM`
{: .hands_on}

Let's look at the KO and PFAM statistics.

> <question-title></question-title>
>
> 1. How many genes have been extended with KO information?
> 2. How many different KO are found?
> 5. Which KO is the most found?
> 3. How many genes have been extended with PFAM information?
> 4. How many different PFAM are found?
> 5. Which PFAM is the most found?
>
> > <solution-title></solution-title>
> >
> > 1. Using {% tool [Filter](Filter1) %} with `c5!='.'`, we find that 426 CDS (1.62%) have been extended with KO information.
> > 2. 277 KO ({% tool [Group data by a column](Grouping1) %} to group and count on 5th column)
> > 3. K14572 (Ribosome biogenesis in eukaryotes) is found 9 times ({% tool [Sort](sort1) %} by descending order on the **Group data by a column** output on column 2)
> > 3. 4,817 CDS (18.29%) have been extended with PFAM information ({% tool [Filter](Filter1) %} with `c8!='.'` on `CDS + KO + PFAM`)
> > 4. 410 PFAM ({% tool [Group data by a column](Grouping1) %} to group and count on 8th column)
> > 5. PF00961.22 (Cytochrome c oxidase subunit 1) is found 89 times ({% tool [Sort](sort1) %} by descending order)
> >
> {: .solution}
>
{: .question}

Let's extract an overview of the annotations per metagenomic sequence by grouping on the 1st column and counting the number of distinct values on gene ID, gene name, KO ID, PFAM ID.

> <hands-on-title>Annotation per metagenomic sequences</hands-on-title>
>
> 1. {% tool [Group data by a column](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `CDS + KO + PFAM`
>    - *"Group by column"*: `Column: 1`
>    - In *"Operation"*:
>       - In *"1: Operation"*:
>         - *"Type"*: `Count distinct`
>         - *"On column"*: `Column: 2`
>       - In *"2: Operation"*:
>         - *"Type"*: `Count distinct`
>         - *"On column"*: `Column: 3`
>       - In *"3: Operation"*:
>         - *"Type"*: `Count distinct`
>         - *"On column"*: `Column: 5`
>       - In *"4: Operation"*:
>         - *"Type"*: `Count distinct`
>         - *"On column"*: `Column: 8`
> 
> 2. {% tool [Select last lines from a dataset (tail)](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_tail_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Text file"*: output of **Cut** {% icon tool %}
>    - *"Operation"*: `Keep everything from lines on`
>    - *"Number of lines"*: `2`
>
> 3. {% tool [Add Header](toolshed.g2.bx.psu.edu/repos/estrain/add_column_headers/add_column_headers/0.1.3) %} with the following parameters:
>    - *"List of Column headers"*: `Metagenomic sequence name,Number of CDS,Number of annotated CDS,Number of associated KO,Number of associated PFAM`
>    - {% icon param-file %} *"Data File (tab-delimited)"*: output of **Select last** {% icon tool %}
>
> 4. Rename `CDS annotation overview per metagenomic sequences`
{: .hands_on}

# Conclusion


topic: microbiome
tutorial: qiime2-cancer-microbiome-intervention
title: QIIME 2 Cancer Microbiome Intervention
difficulty: intermediate
time_estimation: 10H
tools_json: []
requirements_json: []
tags_json: ["metabarcoding", "16S", "taxonomic profiling", "diversity", "microgalaxy"]
content_hash: d41d8cd98f00b204e9800998ecf8427e
last_modified: 2026-02-26T21:12:17.364419
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/qiime2-cancer-microbiome-intervention/tutorial.html


topic: microbiome
tutorial: metatranscriptomics-short
title: Metatranscriptomics analysis using microbiome RNA-seq data (short)
difficulty: introductory
time_estimation: 3H
content: {% include topics/microbiome/tutorials/metatranscriptomics/content.md short=true %}
tools_json: []
requirements_json: []
tags_json: ["metatranscriptomics", "microgalaxy"]
content_hash: beed5d3e14894c43ff6cb10e5d043b66
last_modified: 2026-02-26T21:12:17.312128
zenodo_link: https://zenodo.org/record/4776250
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/metatranscriptomics-short/tutorial.html


topic: microbiome
tutorial: metagenomics-binning
title: Binning of metagenomic sequencing data
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: ["type: internal"]
tags_json: ["binning", "metagenomics", "microgalaxy"]
content_hash: be36ac834274cefc678af10524f0369a
last_modified: 2026-02-26T21:12:17.306413
zenodo_link: https://zenodo.org/records/17660820
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/metagenomics-binning/tutorial.html

content:
Metagenomics is the study of genetic material recovered directly from environmental samples, such as soil, water, or gut contents, without the need for isolating or cultivating individual organisms. Metagenomics binning is a process used to classify DNA sequences obtained from metagenomic sequencing into discrete groups, or bins, based on their similarity to each other.

The goal of metagenomics binning is to assign DNA sequences to the organisms or taxonomic groups from which they originate, allowing for a better understanding of the diversity and functions of the microbial communities present in the sample. This is typically achieved through computational methods that include sequence similarity, composition, and other features to group the sequences into bins.

> <comment-title></comment-title>
> Before starting this tutorial, it is recommended to do the [**Metagenomics Assembly Tutorial**]({% link topics/microbiome/tutorials/metagenomics-assembly/tutorial.md %})
{: .comment}

## Binning approaches

There are several approaches to metagenomics binning, including:

- **Sequence composition-based binning**: This method is based on the observation that different genomes have distinct sequence composition patterns, such as GC content or codon usage bias. By analyzing these patterns in metagenomic data, sequence fragments can be assigned to individual genomes or groups of genomes.

- **Coverage-based binning**: This method uses the depth of coverage of sequencing reads to group them into bins. Sequencing reads that originate from the same genome are expected to have similar coverage, and this information can be used to identify groups of reads that represent individual genomes or genome clusters.

- **Hybrid binning**: This method combines sequence composition-based and coverage-based binning to increase the accuracy of binning results. By using multiple sources of information, hybrid binning can better distinguish closely related genomes that may have similar sequence composition patterns.

- **Clustering-based binning**: This method groups sequence fragments into clusters based on sequence similarity, and then assigns each cluster to a genome or genome cluster based on its sequence composition and coverage. This method is particularly useful for metagenomic data sets with high levels of genomic diversity.

- **Supervised machine learning-based binning**: This method uses machine learning algorithms trained on annotated reference genomes to classify metagenomic data into bins. This approach can achieve high accuracy but requires a large number of annotated genomes for training.

## Binning challenges

Metagenomic binning is a complex process that involves many steps and can be challenging due to several problems that can occur during the process. Some of the most common problems encountered in metagenomic binning include:

- **High complexity**: Metagenomic samples contain DNA from multiple organisms, which can lead to high complexity in the data.
- **Fragmented sequences**: Metagenomic sequencing often generates fragmented sequences, which can make it difficult to assign reads to the correct bin.
- **Uneven coverage**: Some organisms in a metagenomic sample may be more abundant than others, leading to uneven coverage of different genomes.
- **Incomplete or partial genomes**: Metagenomic sequencing may not capture the entire genome of a given organism, which can make it difficult to accurately bin sequences from that organism.
- **Horizontal gene transfer**: Horizontal gene transfer (HGT) can complicate metagenomic binning, as it can introduce genetic material from one organism into another.
- **Chimeric sequences**: Sequences that are the result of sequencing errors or contamination can lead to chimeric sequences, which can make it difficult to accurately bin reads.
- **Strain variation**: Organisms within a species can exhibit significant genetic variation, which can make it difficult to distinguish between different strains in a metagenomic sample.

## Common binners

There are different algorithms and tools performing metagenomic binning. The most widely used include:

* **MaxBin** ({% cite maxbin2015 %}): A popular de novo binning algorithm that uses a combination of sequence features and marker genes to cluster contigs into genome bins.
* **MetaBAT** ({% cite Kang2019 %}): Another widely used de novo binning algorithm that employs a hierarchical clustering approach based on tetranucleotide frequency and coverage information.
* **CONCOCT** ({% cite Alneberg2014 %}): A de novo binning tool that uses a clustering algorithm based on sequence composition and coverage information to group contigs into genome bins.
* **MyCC** ({% cite Lin2016 %}): A reference-based binning tool that uses sequence alignment to identify contigs belonging to the same genome or taxonomic group.
* **GroopM** ({% cite Imelfort2014 %}): A hybrid binning tool that combines reference-based and de novo approaches to achieve high binning accuracy.
* **SemiBin** ({% cite Pan2022 %}): A command-line tool for metagenomic binning with deep learning, handling both short and long reads.
* **Vamb** ({% cite nissen2021improved %}): An algorithm that uses variational autoencoders (VAEs) to encode sequence composition and coverage information.
* **ComeBin** ({% cite Wang2024COMEBin %}): A metagenomic binning tool that integrates both composition and abundance features with machine learning-based clustering to improve binning accuracy across complex microbial communities.

## So many options, what binner to use ?

Each of these binning methods has its own strengths and limitations, and the choice of a binning tool often depends on the characteristics of the metagenomic dataset and the research question. Practical guidance on which binner to use for specific datasets and environments can be drawn from benchmark studies such as {%cite NatureBinner2025%}.

![The image displays four heatmaps (labeled g, h, i, j) comparing the performance of various binning methods across different datasets related to human gut and activated sludge microbiomes. Each heatmap shows numerical values representing metrics for methods such as CONCOCT, MaxBin 2, MetaBAT 2, VAMB, CLMB, MetaDecoder, BinnY, MetaBinner, SemiBin 2, and COMEBin. The columns correspond to different sequence types: short co-assembly, short single, short multi-sample, long single, long multi-sample, hybrid single, and hybrid multi-sample. The color gradient from light to dark red indicates the magnitude of the values, with darker shades representing higher values. The heatmaps provide a visual comparison of how each method performs across these different sequence types and datasets.](./images/Binning_Benchmark.png "Benchmark of multiple binners on activated sludge and human gut microbiome ({%cite NatureBinner2025%})"){:width="60%"}

Another useful approach to investigate the performance of binners is to use simulated datasets. Therefore, the CAMI challenges ({%cite Meyer2022%}) were introduced. CAMI, which stands for Critical Assessment of Metagenome Interpretation, is an international community-driven initiative that organizes benchmarking challenges to objectively evaluate the performance of metagenomic tools. This includes the benchmarking of binners based on standardized, realistically simulated microbial communities that vary in complexity, strain diversity, and abundance.

![This image visually compares the performance of various metagenomic binning tools across multiple environments, including Marine GSA, Marine MA, Strain-madness GSA, Strain-madness MA, Plant-associated GSA, and Plant-associated MA. The top row (a) uses box plots to show the distribution of genome counts recovered from each environment, with sample sizes (n) indicated for each category. The middle section (b) features horizontal bar charts illustrating the number of genomes recovered by different binning tools—such as MetaBinner, UltraBinner, CONCOCT, MetaWRAP, Vamb, MaxBin, MetaBAT, and Autometa—specifically for the Marine GSA environment. Similarly, the bottom section (c) presents bar charts for the Marine MA environment, highlighting the number of genomes recovered by each tool. The image effectively summarizes the efficiency and effectiveness of these tools in different metagenomic contexts.](./images/CAMI_Binners.png "Benchmark of multiple binners on the CAMI datasets ({%cite Meyer2022%})"){:width="60%"}

A general approach is to perform binning using multiple binners that have shown good performance for the specific dataset, followed by bin refinement to generate an improved bin set that retains the best bins from the analysis.

Does using more binners always improve results? In practice, one must also consider computational resources and time constraints. Running many binners can be very time-consuming and resource-intensive, especially for large studies. In some cases, adding extra binners does not lead to a meaningful increase in bin quality, so the choice of binners should be made carefully. Overall, identifying the optimal combination of binners remains an active area of research, and clear, widely accepted guidelines are still being established.

## Bin refinement

There are also bin refinement tools, which can evaluate, combine, and improve the raw bins produced by primary binners such as MetaBAT2, CONCOCT, MaxBin2, or SemiBin. These tools help remove contamination, merge complementary bins, and recover higher-quality MAGs.

* **MetaWRAP** ({% cite Uritskiy2018 %}): A comprehensive metagenomic analysis pipeline that includes modules for quality control, assembly, binning (wrapping multiple binners), refinement, reassembly, and annotation. It provides an easy-to-use framework for producing high-quality MAGs from raw reads.

* **DAS Tool** ({% cite Sieber2018DASTool %}): A bin-refinement tool that combines results from multiple binners (e.g., MetaBAT2, MaxBin2, CONCOCT, SemiBin) into a consensus set of optimized, non-redundant bins. DAS Tool improves overall bin quality by integrating the strengths from several algorithms.

* **Binnette** ({% cite Mainguy2024Binette %}): A fast and accurate bin refinement tool that constructs high-quality MAGs from the outputs of multiple binning tools. It generates hybrid bins using set operations on overlapping contigs — intersection, difference, and union — and evaluates their quality with CheckM2 to select the best bins. Compared to metaWRAP, Binette is faster and can process an unlimited number of input bin sets, making it highly scalable for large and complex metagenomic datasets.

**Anvi’o** ({% cite Eren2015 %}) is a platform for **interactive visualization and manual refinement** of metagenomic bins. While it can run automated binning (defaulting to **CONCOCT**), its main strength lies in allowing users to:

* Inspect contig-level coverage, GC content, and single-copy gene presence
* Visualize connections between contigs in a network view
* Manually merge, split, or reassign contigs to improve bin completeness and reduce contamination
* Annotate bins and link them to taxonomic or functional information

This interactive approach is particularly useful when automated binning produces ambiguous or low-quality bins, enabling **high-confidence MAG reconstruction**.

## Mock binning dataset for this training

Read mapping and binning real metagenomic datasets is a computationally demanding and time-consuming task. To demonstrate the basics of binning in this tutorial, we generated a small mock dataset that is just large enough to produce bins for all binners in this tutorial. The same binners can be applied for any real-life datasets, but as said, plan in some time, up to weeks in some cases.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare analysis history and data

Metagenomic binners typically take two types of data as input: 

- A fasta file containing the assembled contigs, which can be generated from raw metagenomic sequencing reads using an assembler such as MEGAHIT, SPAdes, or IDBA-UD.

- A BAM file containing the read coverage information for each contig, which can be generated from the same sequencing reads using mapping software such as Bowtie2 or BWA.

> <comment-title>Can Bins be generated without coverage information?</comment-title>
>
> Not all binners require coverage information. Some, like MetaBAT2, can operate using only genomic composition (e.g., tetranucleotide frequencies) when coverage files are not available. This is especially useful for single-sample datasets or legacy data where coverage cannot easily be calculated.
>
> Other tools that support composition-only binning include:
> - **MaxBin 2**, which can run with composition alone, but performs better with depth.
> - **SolidBin**, which supports single-sample binning based on sequence features.
> - **VAMB**, which primarily uses deep learning on composition, coverage optional.
>
> That said, including coverage information generally increases binning accuracy, especially for:
> - Differentiating closely related strains
> - Datasets with uneven abundance
> - Multi-sample metagenomics workflows (e.g., differential coverage binning)
>
> In summary: yes, it’s possible to bin without coverage, but coverage-aware workflows are recommended when available, as they reduce contamination and improve completeness.
>
{: .comment}


To run binning, we first need to get the data into Galaxy. Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Prepare the Galaxy history</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

We need to get the data into our history.

In case of a not very large dataset it's more convenient to upload data directly from your computer to Galaxy.

> <hands-on-title>Upload data into Galaxy</hands-on-title>
>
> 1. Import the contig file from [Zenodo]({{ page.zenodo_link }}) or a data library:
>
>    ```text
>    {{ page.zenodo_link }}/files/MEGAHIT_contigs.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    > <comment-title></comment-title>
>    > In case of large dataset, we can use FTP server or the [Galaxy Rule-based Uploader]({% link topics/galaxy-interface/tutorials/upload-rules/tutorial.md %}).
>    {: .comment}
>
> 2. Create a collection named `Contigs`
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
> 3. Import the raw reads in fastq format from [Zenodo]({{ page.zenodo_link }}) or a data library:
>
>    ```text
>    {{ page.zenodo_link }}/files/reads_forward.fastqsanger.gz
>    {{ page.zenodo_link }}/files/reads_reverse.fastqsanger.gz
>    ```
>
> 4. Create a paired collection named `Reads`
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
{: .hands_on}

> <comment-title>Why do we use collections here?</comment-title>
> In this tutorial, collections are not strictly necessary because we are working with only one contig file and its paired-end reads. However, in real metagenomic studies, it is common to process many samples—sometimes hundreds or even thousands—and in those cases, collections become essential for managing data efficiently.  
>  
> It is generally good practice to first test a workflow on a small subset of the data (for example, a collection containing only a single sample) to ensure that the tools run correctly and the parameters are appropriate before launching thousands of jobs on Galaxy.
{: .comment}

# Preparation for binning

As explained before, we need coverage information in a BAM file as a requirement for all binners. Some binners need a specific format for the coverage information, but this will be covered in the version specific to the desired binner. For now, we will map the quality-controlled reads to the contigs to get a BAM file with the coverage information. This BAM file also needs to be sorted for the downstream binners.

> <comment-title></comment-title>
> Make sure the reads are quality-controlled, e.g., by following the [**Quality Control** tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %})
{: .comment}

> <hands-on-title> Map reads to contigs </hands-on-title>
>
> 1. {% tool [Bowtie2](toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.4+galaxy0) %} with the following parameters:
>    - *"Is this single or paired library"*: `Paired-end`
>        - {% icon param-collection %} *"FASTQ Paired Dataset"*: `Reads` (Input dataset collection)
>        - *"Do you want to set paired-end options?"*: `No`
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from the history and build index`
>        - {% icon param-collection %} *"Select reference genome"*: `Contigs` (Input dataset collection)
>    - *"Set read groups information?"*: `Do not set`
>    - *"Select analysis mode"*: `1: Default setting only`
>    - *"Do you want to tweak SAM/BAM Options?"*: `No`
>    - *"Save the bowtie2 mapping statistics to the history"*: `Yes`
>
{: .hands_on}

Let's now sort the BAM files.

> <hands-on-title> Sort BAM files </hands-on-title>
>
> 1. {% tool [Samtools sort](toolshed.g2.bx.psu.edu/repos/devteam/samtools_sort/samtools_sort/2.0.7) %} with the following parameters:
>    - {% icon param-file %} *"BAM File"*: output of **Bowtie2** {% icon tool %}
>    - *"Primary sort key"*: `coordinate`
>
{: .hands_on}

The sorted BAM file can be used as input for any of the binning tools.

# Binning

As explained before, there are many challenges to metagenomics binning. The most common of them are listed below:

- High complexity.
- Fragmented sequences.
- Uneven coverage.
- Incomplete or partial genomes.
- Horizontal gene transfer.
- Chimeric sequences.
- Strain variation.

![The image illustrates the concept of metagenomic binning, where sequences called contigs are grouped into distinct bins based on their similarities. On the left side, a set of colored contigs (represented as horizontal lines in red, yellow, and green) is shown. An arrow labeled "Binning" points to the right, where these contigs are organized into three separate ovals labeled Bin 1, Bin 2, and Bin 3. Each bin contains contigs of similar colors, indicating that they have been grouped together based on shared characteristics, such as origin or genetic similarity. This process helps in categorizing and analyzing genetic material from complex metagenomic datasets.](./images/binning.png "Metagenomic binning involves grouping contigs into 'bins' based on sequence composition, coverage, or other properties."){:width="60%"}

In this tutorial, we offer several versions, each highlighting a different binners:

{% include _includes/cyoa-choices.html option1="MetaBAT2" option2="MaxBin2" option3="SemiBin" option4="CONCOCT" option5="COMEBin" default="MetaBAT2" %}

<div class="MetaBAT2" markdown="1">
{% include topics/microbiome/tutorials/metagenomics-binning/metabet2_version.md %}
</div>
<div class="MaxBin2" markdown="1">
{% include topics/microbiome/tutorials/metagenomics-binning/maxbin2_version.md %}
</div>
<div class="SemiBin" markdown="1">
{% include topics/microbiome/tutorials/metagenomics-binning/semibin_version.md %}
</div>
<div class="CONCOCT" markdown="1">
{% include topics/microbiome/tutorials/metagenomics-binning/concoct_version.md %}
</div>
<div class="COMEBin" markdown="1">
{% include topics/microbiome/tutorials/metagenomics-binning/comebin_version.md %}
</div>

# Bin refinement

Now that we have produced bins with our favorite binner, we can refine the recovered bins. 
Therefore, we need to 

1. Convert the bins into a contig-to-bin mapping table,
2. Combine the tables from each binner into one collection, and
3. Use **Binette** ({% cite Mainguy2024Binette %}) to create consensus bins.

   An alternative tool would be {% tool [DAS Tool](toolshed.g2.bx.psu.edu/repos/iuc/das_tool/das_tool/1.1.7+galaxy1) %}  ({% cite Sieber2018DASTool %}) which is also available in Galaxy.

For the refinement, we will use the bins created by all the binners used before. If you did not run them all by yourself, we have provided the results here as well. 

> <hands-on-title>Get result bins</hands-on-title>
>
> 1. Import the bin files from [Zenodo]({{ page.zenodo_link }}) or a data library:
>
>    ```text
>    {{ page.zenodo_link }}/files/semibin_0.fasta
>    {{ page.zenodo_link }}/files/maxbin_0.fasta
>    {{ page.zenodo_link }}/files/maxbin_1.fasta
>    {{ page.zenodo_link }}/files/metabat_0.fasta
>    {{ page.zenodo_link }}/files/concoct_1.fasta
>    {{ page.zenodo_link }}/files/concoct_2.fasta
>    {{ page.zenodo_link }}/files/concoct_3.fasta
>    {{ page.zenodo_link }}/files/concoct_4.fasta
>    {{ page.zenodo_link }}/files/concoct_5.fasta
>    {{ page.zenodo_link }}/files/concoct_6.fasta
>    {{ page.zenodo_link }}/files/concoct_7.fasta
>    {{ page.zenodo_link }}/files/concoct_8.fasta
>    {{ page.zenodo_link }}/files/concoct_9.fasta
>    ```
>
> 2. Create a collection for each binner by selecting only the bins created by this binner (e.g., `maxbin_0.fasta` and `maxbin_1.fasta` together) and creating a collection:
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
{: .hands_on}

Once each bin set is converted into one collection, it can be converted into a contig-to-bin mapping table. We need to perform this step for every bin set.

> <hands-on-title> Convert the bins into a contig-to-bin mapping table </hands-on-title>
>
> 1. {% tool [Converts genome bins in fasta format](toolshed.g2.bx.psu.edu/repos/iuc/fasta_to_contig2bin/Fasta_to_Contig2Bin/1.1.7+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Bin sequences"*: `bins` (output of any of the binners {% icon tool %})
>
{: .hands_on}


We can now build a list of the tables

> <hands-on-title> Build a list of the binning tables </hands-on-title>
>
> 1. {% tool [Build list](__BUILD_LIST__) %} with the following parameters:
>    - In *"Dataset"*:
>        - {% icon param-repeat %} *"Insert Dataset"*
>            - {% icon param-file %} *"Input Dataset"*: `contigs2bin` (output of **Converts genome bins in fasta format** of SemiBin {% icon tool %})
>            - *"Label to use"*: `Index`
>        - {% icon param-repeat %} *"Insert Dataset"*
>            - {% icon param-file %} *"Input Dataset"*: `contigs2bin` (output of **Converts genome bins in fasta format** of MetaBAT2 {% icon tool %})
>            - *"Label to use"*: `Index`
>        - {% icon param-repeat %} *"Insert Dataset"*
>            - {% icon param-file %} *"Input Dataset"*: `contigs2bin` (output of **Converts genome bins in fasta format** of MaxBin2 {% icon tool %})
>            - *"Label to use"*: `Index`
>        - {% icon param-repeat %} *"Insert Dataset"*
>            - {% icon param-file %} *"Input Dataset"*: `contigs2bin` (output of **Converts genome bins in fasta format** of CONCOCT {% icon tool %})
>            - *"Label to use"*: `Index`
>
{: .hands_on}

We can now refine the bins with Binette.

> <hands-on-title> Refine with Binette </hands-on-title>
>
> 1. {% tool [Binette](toolshed.g2.bx.psu.edu/repos/iuc/binette/binette/1.2.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input contig table"*: `output` (output of **Build list** {% icon tool %})
>    - {% icon param-collection %} *"Input contig file"*: `output` (Input dataset collection)
>    - *"Select if database should be used either via file or cached database"*: `cached database`
>
{: .hands_on}

> <question-title>Bin refinement</question-title>
>
> 1. How many bins are left after refinement ?
>
> > <solution-title></solution-title>
> >
> > 1. Two bins are left. Most contigs from different bins where combined into one bin. There is still one single contig bin left.
> >
> {: .solution}
>
{: .question}

# Checking the quality of the bins

Once binning is done, it is important to check its quality.

Binning results can be evaluated with **CheckM** ({% cite Parks2015 %}). CheckM is a software tool used in metagenomics binning to assess the completeness and contamination of genome bins. 

CheckM compares the genome bins to a set of universal single-copy marker genes that are present in nearly all bacterial and archaeal genomes. By identifying the presence or absence of these marker genes in the bins, CheckM can estimate the completeness of each genome bin (i.e., the percentage of the total set of universal single-copy marker genes that are present in the bin) and the degree of contamination (i.e., the percentage of marker genes that are found in more than one bin).

This information can be used to evaluate the quality of genome bins and to select high-quality bins for further analysis, such as genome annotation and comparative genomics. CheckM is widely used in metagenomics research and has been shown to be an effective tool for assessing the quality of genome bins. Some of the key functionalities of CheckM are:

- *Estimation of genome completeness*: CheckM uses a set of universal single-copy marker genes to estimate the completeness of genome bins. The completeness score indicates the proportion of these marker genes that are present in the bin, providing an estimate of how much of the genome has been recovered.

- *Estimation of genome contamination*: CheckM uses the same set of marker genes to estimate the degree of contamination in genome bins. The contamination score indicates the proportion of marker genes that are present in multiple bins, suggesting that the genome bin may contain DNA from more than one organism.

- *Identification of potential misassemblies*: CheckM can identify potential misassemblies in genome bins based on the distribution of marker genes across the genome.

- *Visualization of results*: CheckM can generate various plots and tables to visualize the completeness, contamination, and other quality metrics for genome bins, making it easier to interpret the results.

- *Taxonomic classification*: CheckM can also be used to classify genome bins taxonomically based on the presence of specific marker genes associated with different taxonomic groups.

Based on the previous analysis we will use **CheckM lineage_wf**: *Assessing the completeness and contamination of genome bins using lineage-specific marker sets*

`CheckM lineage_wf` is a specific workflow within the CheckM software tool that is used for taxonomic classification of genome bins based on their marker gene content. This workflow uses a reference database of marker genes and taxonomic information to classify the genome bins at different taxonomic levels, from domain to species.

Now we can investigate the completeness and contamination of any of your previously generated genome bins, as well as the refined set.

> <hands-on-title>Assessing the completeness and contamination of genome bins using lineage-specific marker sets with `CheckM lineage_wf`</hands-on-title>
> 1.  {% tool [CheckM lineage_wf](toolshed.g2.bx.psu.edu/repos/iuc/checkm_lineage_wf/checkm_lineage_wf/1.2.0+galaxy0) %} with parameters:
>     - *"Bins"*: `Folder containing the produced bins`
{: .hands_on}


The output of "CheckM lineage_wf" includes several files and tables that provide information about the taxonomic classification and quality assessment of genome bins. Here are some of the key outputs:

- **CheckM Lineage Workflow Output Report (Bin statistics)**: This report provides a summary of the quality assessment performed by CheckM. It includes statistics such as the number of genomes analyzed, their completeness, contamination, and other quality metrics.

- **Lineage-specific Quality Assessment**: CheckM generates lineage-specific quality assessment files for each analyzed genome. These files contain detailed information about the completeness and contamination of the genome based on its taxonomic lineage.

- **Marker Set Analysis**: CheckM uses a set of marker genes to estimate genome completeness and contamination. The tool produces marker-specific analysis files that provide details on the presence, absence, and copy number of each marker gene in the analyzed genomes.

- **Visualizations**: CheckM generates various visualizations to aid in the interpretation of the results. These include plots such as the lineage-specific completeness and contamination plots, scatter plots, and other visual representations of the data.

- **Tables and Data Files**: CheckM generates tabular data files that contain detailed information about the analyzed genomes, including their names, taxonomic assignments, completeness scores, contamination scores, and other relevant metrics. These files are useful for further downstream analysis or data manipulation.

It should be noted that "CheckM lineage_wf" offers a range of optional outputs that can be generated to provide additional information to the user.

To keep it simple, we will check the bin statistics to investigate the performance of our previous binning efforts.

> <question-title>Binning performance</question-title>
>
> 1. Which binner created the best bins?
> 2. How well did the bin refinement work?
>
> > <solution-title></solution-title>
> >
> > 1. Let's combine the stat:
> >
> >    Binner | Bin ID | Marker lineage | # genomes | # markers | # marker sets | 0  | 1  | 2 | 3 | 4 | 5+ | Completeness | Contamination | Strain heterogeneity
> >    --- | --- | --- | --- | --- | --- | --- | --- | ---| --- | --- | --- | --- | --- | ---
> >    **MetaBAT2** | 1 | k__Bacteria (UID203) | 5449 | 103 | 58 | 89 | 14 | 0  | 0  | 0  | 0  | 15.67 | 0.00  0.00
> >    **SemiBin** |  SemiBin_0 | k__Bacteria (UID203) | 5449 | 103 | 58 | 89 | 14 | 0 | 0 | 0 | 0 | 15.67 | 0.00 | 0.00
> >    **MaxBin** | 001 | k__Bacteria (UID203) | 5449 | 103 | 58 | 92 | 11 | 0 | 0 | 0 | 0 | 10.50 | 0.00 | 0.00
> >    **MaxBin** | 002 | k__Bacteria (UID203) | 5449  | 103 | 58 | 99 | 4 | 0 | 0 | 0 | 0 | 6.90| 0.00 | 0.00
> >    **CONCOCT** | 0 | root (UID1) | 5656 | 56     | 24 | 56 | 0  | 0 | 0 | 0 | 0 | 0.00 | 0.00 | 0.00
> >    **CONCOCT** | 1 | root (UID1) | 5656 | 56        | 24 | 56 | 0 | 0 | 0 | 0 | 0 | 0.00 | 0.00 | 0.00
> >    **CONCOCT** | 2 | root (UID1) | 5656 | 56 | 24 | 56 | 0 | 0 | 0 | 0 | 0 | 0.00 | 0.00 | 0.00
> >    **CONCOCT** | 3 | root (UID1) | 5656 | 56 | 24 | 56 | 0 | 0 | 0 | 0 | 0 | 0.00 | 0.00 | 0.00
> >    **CONCOCT** | 4 | root (UID1) | 5656 | 56 | 24 | 56 | 0 | 0 | 0 | 0 | 0 | 0.00 | 0.00 | 0.00
> >    **CONCOCT** | 5 | root (UID1) | 5656 | 56 | 24 | 56 | 0 | 0 | 0 | 0 | 0 | 0.00 | 0.00 | 0.00 
> >    **CONCOCT** | 6 | root (UID1) | 5656 | 56 | 24 | 56 | 0 | 0 | 0 | 0 | 0 | 0.00 | 0.00 | 0.00
> >    **CONCOCT** | 7 | root (UID1) | 5656 | 56 | 24 | 56 | 0 | 0 | 0 | 0 | 0 | 0.00 | 0.00 | 0.00
> >    **CONCOCT** | 8 | root (UID1) | 5656 | 56 | 24 | 56 | 0 | 0 | 0 | 0 | 0 | 0.00 | 0.00 | 0.00
> >    **CONCOCT** | 9 | k__Bacteria (UID203) |5449 | 103 | 58 | 89 | 14 | 0 | 0 | 0 | 0 | 15.67 | 0.00 | 0.00
> >    **MetaBAT2** and **SemiBin** generated the best‐performing bins in this dataset, each recovering one bin with ~15.7% completeness and no detectable contamination. 
> >
> >    **MaxBin** produced bins of lower completeness (10.5% and 6.9%), but still without contamination. 
> >
> >    **CONCOCT** largely failed to recover meaningful bins in this dataset: most bins show 0% completeness. This often occurs with CONCOCT on small assemblies or uneven coverage, because its Gaussian clustering model struggles when the number of contigs is low or the variation in coverage is insufficient.
> > **Binette**
> >
> > | Bin ID        | Marker lineage        | # genomes | # markers | # marker sets | 0  | 1  | 2 | 3 | 4 | 5+ | Completeness | Contamination | Strain heterogeneity |
> > |---------------|------------------------|-----------|------------|----------------|----|----|----|----|----|----|--------------|---------------|-----------------------|
> > | binette_bin1  | k__Bacteria (UID203)   | 5449      | 103        | 58             | 89 | 14 | 0  | 0  | 0  | 0  | 15.67        | 0.00          | 0.00                  |
> > 2. Let's look at **Binette** stats:
> >    Bin ID | Marker lineage | # genomes | # markers | # marker sets | 0 | 1 | 2 | 3 | 4 | 5+ | Completeness | Contamination | Strain heterogeneity
> >    --- | --- | --- | --- | --- | --- | --- | ---| --- | --- | --- | --- | --- | ---
> >    binette_bin1 | k__Bacteria (UID203) | 5449    | 103 | 58 | 89 | 14 | 0 | 0 | 0 | 0 | 15.67 | 0.00 | 0.00
> >    binette_bin2 | root (UID1) | 5656 | 56 | 24 | 56 | 0 | 0 | 0 | 0 | 0 | 0.00 | 0.00 | 0.00
> >
> > **Binette** produced one bin essentially identical in quality to the MetaBAT2/SemiBin bins (~15.7% completeness), plus a root-level bin with no completeness. Binette could recover the most completely successful bin. The additional
loq quality bin, would have normally been filtered by Binette if the `Set minimus completeness` parameter were set to a reasonable value, often > 75 % for true biological data.
> {: .solution}
>
{: .question}

> <comment-title>CheckM2 vs CheckM</comment-title>
> CheckM2 ({%cite Chklovski2023CheckM2%}) is the successor of CheckM, but CheckM is still widely used, since its marker-based logic can be more interpretable in a biological sense. E.g., to date (2025-11-21), NCBI still allows submitting MAGs to GenBank if either checkM or checkM2 has a completeness of > 90% (see the [NCBI WGS/MAG submission guidelines](https://www.ncbi.nlm.nih.gov/genbank/wgsfaq/#metagen)).
>
> **Key differences compared to CheckM1**:
>
> * CheckM1 relies primarily on lineage-specific single-copy marker genes to estimate completeness and contamination of microbial genomes.
> * CheckM2 uses a machine-learning (gradient boost / ML) approach trained on simulated and experimental genomes, and does *not* strictly require a well-represented lineage in its marker database. 
> * CheckM2 is reported to be more accurate and faster for both bacterial and archaeal lineages, especially when dealing with novel or very reduced-genome lineages (e.g., candidate phyla, CPR/DPANN) where classical marker-gene methods may struggle. 
> * The database of CheckM2 can be updated more rapidly with new high-quality reference genomes, which supports scalability and improved performance over time.
>
> If you're working with MAGs from underrepresented taxa (novel lineages) or very small genomes (streamlined bacteria/archaea), CheckM2 tends to give more reliable estimates of completeness/contamination. For more “standard” microbial genomes from well-studied taxa, CheckM1 may still work well, but you may benefit from the improved performance with CheckM2.
>
{: .comment}

# De-replication

De-replication is the process of identifying sets of genomes that are the "same" in a list of genomes, and removing all but the “best” genome from each redundant set. How similar genomes need to be to be considered “same”, how to determine which genome is “best”, and other important decisions are discussed in [Important Concepts](https://drep.readthedocs.io/en/latest/choosing_parameters.html).

A common use for genome de-replication is the case of individual assembly of metagenomic data. If metagenomic samples are collected in a series, a common way to assemble the short reads is with a “co-assembly”. That is, combining the reads from all samples and assembling them. The problem with this is that assembling similar strains can severely fragment assemblies, precluding the recovery of a good genome bin. An alternative option is to assemble each sample separately, and then “de-replicate” the bins from each assembly to make a final genome set.

![Image shows the process of individual assembly on two strains and five samples, after individual assembly of samples two samples are chosen for de-replication process. In parallel, co-assembly on all five samples is performed](./individual-assembly.png "Individual assembly followed by de-replication vs co-assembly"){:width="80%"}

Several tools have been designed for the process of de-replication. **dRep** ({% cite olm2017drep %}) is a software tool designed for the dereplication of genomes in metagenomic datasets. The goal is to retain a representative set of genomes to improve downstream analyses, such as taxonomic profiling and functional annotation.

An typical workflow of how dRep works for dereplication in metagenomics includes:

- *Genome Comparison*: dRep uses a pairwise genome comparison approach to assess the similarity between genomes in a given metagenomic dataset.

- *Clustering*: Based on the genome similarities, dRep performs clustering to group similar genomes into "genome clusters." Each cluster represents a group of closely related genomes.

- *Genome Quality Assessment*: dRep evaluates the quality of each genome within a cluster. It considers factors such as completeness, contamination, and strain heterogeneity.

- *Genome Selection*: Within each genome cluster, dRep selects a representative genome based on user-defined criteria. This representative genome is considered the "dereplicated" version of the cluster.

- *Dereplication Output*: The output of dRep includes information about the dereplicated genomes, including their identity, completeness, and contamination. The user can choose a threshold for genome similarity to control the level of dereplication.

> <hands-on-title>General list of actions for de-replication</hands-on-title>
> 1. Create new history
> 2. Assemble each sample separately using your favorite assembler
> 3. Perform a co-assembly to catch low-abundance microbes
> 4. Bin each assembly separately using your favorite binner
> 5. Bin co-assembly using your favorite binner
> 6. Pull the bins from all assemblies together
> 7. Run {% tool [dRep dereplication](toolshed.g2.bx.psu.edu/repos/iuc/drep_dereplicate/drep_dereplicate/3.6.2+galaxy1) %} on them
> 8. Perform downstream analysis on the de-replicated genome list
>
{: .hands_on}

# Conclusions

In summary, this tutorial shows a step-by-step guide on how to bin metagenomic contigs using various binners, including bin refinement.

It is critical to select the appropriate binning tool for a specific metagenomics study, as different binning methods may have different strengths and limitations depending on the type of metagenomic data being analyzed. By comparing the outcomes of several binning techniques, researchers can increase the precision and accuracy of genome binning.

There are various binning methods available for metagenomic data, including reference-based, clustering-based, hybrid approaches, and machine learning. Each method has its advantages and disadvantages, and the selection of the appropriate method depends on the research question and the characteristics of the data.

Comparing the outcomes of multiple binning methods can help to identify the most accurate and reliable method for a specific study. This can be done by evaluating the quality of the resulting bins in terms of completeness, contamination, and strain heterogeneity, as well as by comparing the composition and functional profiles of the identified genomes.

Overall, by carefully selecting and comparing binning methods, researchers can improve the quality and reliability of genome bins, which can ultimately lead to a better understanding of the functional and ecological roles of microbial communities in various environments.


topic: microbiome
tutorial: host-removal
title: Remove contamination and host reads
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: []
content_hash: e5f1059a1b8dbabe4d6512ed61334186
last_modified: 2026-02-26T21:12:17.290447
zenodo_link: https://zenodo.org/records/17829290
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/host-removal/tutorial.html

content:
Metagenomic sequencing captures all DNA present in a sample, including the **microbial community**, **host DNA**, and potential **environmental or external contaminants** (such as human DNA introduced during sample handling or sequencing).  
Before performing taxonomic or functional analysis, it is essential to remove these host-drived or contaminant reads to avoid misleading downstream interpretations.

In this tutorial, we will learn how to identify and remove host or contaminant reads using Galaxy.
We will:
- Map raw reads to a **host reference genome** using Bowtie2 and extract unmapped reads.
- Repeat the process with unmapped reads against a **human reference genome** to remove potential human contamination.
- Generate a final set of **clean, non-host reads** suitable for downstream analyses such as assembly, binning, or profiling.

To test and illustrate the process, we use bee gut metagenome samples and focus on filtering out both host (bee) sequences and potential human contamination from the reads. 


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## Prepare Galaxy and data
Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

Now, we need to import the data

> <hands-on-title>Import datasets</hands-on-title>
>
> 1. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library:
>
>    ```text
>    {{ page.zenodo_link }}/files/SRR24759598_1.fastq.gz
>    {{ page.zenodo_link }}/files/SRR24759598_2.fastq.gz
>    {{ page.zenodo_link }}/files/SRR24759616_1.fastq.gz
>    {{ page.zenodo_link }}/files/SRR24759616_2.fastq.gz
>    ```
>    
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Create a paired collection named 'Raw reads' that includes both pairs.
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
{: .hands_on}

## Map reads to a host genome with Bowtie2
To remove host contamination, we start by mapping the reads to the host genome using Bowtie2 to detect and remove host-derived sequences.

> <hands-on-title>Remove host reads</hands-on-title>
>
> 1. {% tool [Bowtie2](toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.3+galaxy1) %} with the following parameters:
>    - *"Is this single or paired library"*: `Paired-end Dataset Collection`
>        - {% icon param-collection %} *"FASTQ Paired Dataset"*: `Input reads`
>        - *"Write unaligned reads (in fastq format) to separate file(s)"*: `Yes`
>        - *"Do you want to set paired-end options?"*: `Yes`
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>        - *"Select reference genome"*: `A. mellifera genome (apiMel3, Baylor HGSC Amel_3.0)`
>    - *"Set read groups information?"*: `Do not set`
>    - *"Select analysis mode"*: `1: Default setting only`
>    - *"Do you want to tweak SAM/BAM Options?"*: `No`
>    - *"Save the bowtie2 mapping statistics to the history"*: `Yes`
>
> 2. Run the tool. The outputs will include:
>    - Mapping statistics report (`bowtie2.log`)
>    - Unaligned (unmapped) forward and reverse reads
>
> 3. These unmapped reads represent sequences **not belonging to the host** and will be used in the next step. 
>
> > <comment-title>Tip</comment-title>
> > Host reference genomes vary depending on the study organism. You can upload a FASTA file of your host genome if it is not available as a built-in index.
> {: .comment}
>
{: .hands_on}

## Re-pair unmapped reads
We now combine the unmapped forward and reverse reads into a new paired-end dataset for further processing.

> <hands-on-title> Combine unmapped forward and reverse reads into a paired collection </hands-on-title>
>
> 1. {% tool [Zip collections](__ZIP_COLLECTION__) %} with the following parameters:
>    - {% icon param-file %} *"Input 1"*: `output_unaligned_reads_l` 
>    - {% icon param-file %} *"Input 2"*: `output_unaligned_reads_r` 
>
>    2. This step creates a new paired-end collection that represents all reads **not aligned to the host genome**. Rename this output collection to 'Reads without bee reads'
>
>    > <comment-title> Note </comment-title>
>    >
>    > Zipping restores the normal paired-end structure, which is required for downstream tools or for rerunning the workflow on another reference.
>    {: .comment}
>
{: .hands_on}

## Summarize mapping statistics
Once the host mapping is complete, we use MultiQC to summarize and visualize the mapping statistics, helping us assess how many reads were removed and how many remain.

> <hands-on-title> Evaluate host read removal results </hands-on-title>
>
> 1. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3) %} with the following parameters:
>    - In *"Results"*:
>        - {% icon param-repeat %} *"Insert Results"*
>            - *"Which tool was used generate logs?"*: `Bowtie 2`
>                - {% icon param-file %} *"Output of Bowtie 2"*: `mapping_stats` (output of **Bowtie2** {% icon tool %})
>    - *"Report title"*: `Host Removal`
>
> 2. Run the tool and open the generated HTML report.
> 3. Review the mapping percentage, number of reads aligned, and number of unmapped reads.
>
>    > <comment-title>Tip</comment-title>
>    >
>    > Low mapping percentages in the report confirm that most host reads were successfully removed.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What percentage of reads mapped to the bee reference genome?  
> 2. What percentage of reads were removed?
>
> > <solution-title></solution-title>
> >
> > 1. 2.8% for SRR24759598 and 1.1% for SRR24759616. 
> > 2. 2.8% for SRR24759598 and 1.1% for SRR24759616.
> >
> {: .solution}
>
{: .question}

## Remove potential human contamination
After removing host reads, we can run the **same workflow again** to eliminate possible **human contamination** that may remain in the dataset.

> <hands-on-title> Rerun the workflow using the human genome as reference</hands-on-title>
>
> 1. Use the unmapped reads **Reads without bee reads** as the input for this second run.
> 2. In the **Bowtie2** step:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>    - *"Select reference genome"*: `Human (GRCh38)`
>    - Keep all other parameters the same as in the first run.
>
> 3. Continue through the **Zip collections** and **MultiQC** steps as before.
> 4. The output of this second run represents your **final cleaned reads**, free from both host and human sequences. Rename this collection to 'Clean metagenomic reads'
>
>    > <comment-title>Note</comment-title>
>    > Rerunning the same workflow maintains reproducibility. 
>    > Only the reference genome and the input data change between the two runs.
>    {: .comment}
>
{: .hands_on}

> <question-title>Verify your final dataset</question-title>
>
> 1. How does the mapping percentage differ between the host and human filtering runs?
> 2. How many reads have been removed?
>
> > <solution-title></solution-title>
> >
> > 1. The second run (against the human genome) usually removes only a small number of additional reads.
> > 2. 0.1% in both samples.
> >
> {: .solution}
>
{: .question}


# Conclusion

In this tutorial, you learned how to:

- Identify and remove reads originating from host or contaminant genomes using **Bowtie2**.  
- Combine unmapped forward and reverse reads into a paired collection for reuse.  
- Summarize mapping statistics and verify host-read removal using **MultiQC**.  
- Rerun the same workflow with a human reference genome to remove residual human contamination.  

The resulting **clean reads** are now ready for downstream metagenomic analyses such as:
- **Assembly**   
- **Binning**   
- **Functional or taxonomic profiling** 

These preprocessing steps are essential to ensure accurate microbial community reconstruction without interference from host DNA.


topic: microbiome
tutorial: diversity
title: Calculating α and β diversity from microbiome taxonomic data
difficulty: introductory
time_estimation: 20M
tools_json: []
requirements_json: []
tags_json: ["metagenomics", "diversity"]
content_hash: 2064f938924987a9304fa49105f19251
last_modified: 2026-02-26T21:12:17.275198
zenodo_link: https://zenodo.org/records/13150694
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/diversity/tutorial.html

content:
A **diversity index** is a quantitative measure that is used to assess the level of diversity or variety within a particular system, such as a biological community, a population, or a workplace. It provides a way to capture and quantify the distribution of different types or categories within a system.

In various fields, diversity indexes are employed to understand and compare the composition and richness of various elements. Apart from ecology, fields such as social and cultural science are interested in the diversity within a population or workplace. In these cases, the indexes may consider factors like age, gender, ethnicity, or other relevant characteristics to assess the diversity and inclusiveness of a group or organization.

To study microbiome data, indirect methods like **metagenomics** can be used. Metagenomic samples contain DNA from different organisms at a specific site, where the sample was collected. Metagenomic data can be used to find out which organisms coexist in that niche and which genes are present in the different organisms.

Once we know which taxons are present in a metagenomic sample ([Tutorial on Taxonomic Profiling and Visualization of Metagenomic Data]({% link topics/microbiome/tutorials/taxonomic-profiling/tutorial.md %})), we can do diversity analyses.

Related to ecology, the term **diversity** describes the number of different species present in one particular area and their relative abundance. More specifically, several different metrics of diversity can be calculated. The most common ones are $$\alpha$$, $$\beta$$ and $$\gamma$$ diversity:

- **$$\alpha$$ diversity** describes the diversity within a community

    It considers the number of different species in an environment (also referred to as species **richness**). Additionally, it can take the abundance of each species into account to measure how evenly individuals are distributed across the sample (also referred to as species **evenness**). 

- **$$\beta$$ diversity** compare the diversity between different communities by measuring their distance

- **$$\gamma$$ diversity** is a measure of the overall diversity for the different ecosystems within a region.

![The image illustrates the concepts of alpha (α), beta (β), and gamma (γ) diversity using two landscapes, each containing different sites populated with various colored fish. Landscape AB includes Site A and Site B, while Landscape CD includes Site C and Site D. Alpha diversity refers to the variety of species within a single site, as indicated by the different colored fish within each site. Beta diversity measures the difference in species composition between sites within a landscape, shown by comparing the fish species between Site A and Site B, or Site C and Site D. Gamma diversity encompasses the overall diversity across all sites within a landscape, highlighting the total variety of species present in Landscape AB or Landscape CD. The image visually explains how these three levels of diversity are interconnected, emphasizing the importance of considering different scales when studying biodiversity.](./images/diversity_differences.png)

In this analysis we will use Galaxy for calculating different alpha diversity indexes and the Bray-Curtis dissimilarity index for β diversity. 

# Background on data

The dataset we will use for this tutorial comes from an oasis in the Mexican desert called Cuatro Ciénegas ({% cite Okie.2020 %}). The researchers were interested in genomic traits that affect the rates and costs of biochemical information processing within cells. They performed a whole-ecosystem experiment, thus fertilizing the pond to achieve nutrient enriched conditions.

Here we will use 2 datasets:
- `JP4D`: a microbiome sample collected from the Lagunita Fertilized Pond
- `JC1A`: a **control** samples from a control mesocosm.

The datasets differ in size, but according to the authors this doesn't matter for their analysis of genomic traits. Also, they underline that differences between the two samples reflect trait-mediated ecological dynamics instead of microevolutionary changes as the duration of the experiment was only 32 days. This means that depending on available nutrients, specific lineages within the pond grow more successfully than others because of their genomic traits. The samples have been analysed as explained in the [Taxonomic profiling tutorial]({% link topics/microbiome/tutorials/taxonomic-profiling/tutorial.md %}).

In a nutshell, taxonomic labels have been assigned to the metagenomics data using [Kraken2](https://ccb.jhu.edu/software/kraken2/) to find out which species are present in the samples. Finally, species abundance was estimated using [Bracken](https://ccb.jhu.edu/software/bracken/). For this tutorial, we will use the output file of Bracken.

To get an overview, you can find a Krona chart visualizing the different species present in the two samples.

<iframe id="krona" src="krona-kraken.html" frameBorder="0" width="100%" height="900px"> ![Krona chart with multi-layered pie chart representing the community profile with in the center the higher taxonomy levels (i.e. domain) and on the exterior the more detailed ones (i.e. species)](./images/krona-kraken.png) </iframe>

The dataset we will work with in this tutorial is the output file of Bracken, which estimates species abundance.

```
name	taxonomy_id	taxonomy_lvl	kraken_assigned_reads	added_reads	new_est_reads	fraction_total_reads
Paracoccus sp. MC1862	2760307	S	98	4	102	0.00169
Paracoccus sp. AK26	2589076	S	85	8	93	0.00154
Paracoccus sp. Arc7-R13	2500532	S	67	13	80	0.00133
Paracoccus sp. BM15	1529068	S	27	1	28	0.00046
Paracoccus sanguinis	1545044	S	142	37	179	0.00297
Paracoccus contaminans	1945662	S	87	18	105	0.00174
Paracoccus aminovorans	34004	S	86	26	112	0.00186
```

> <question-title></question-title>
>
> What information do the different columns contain?
>
> > <solution-title></solution-title>
> >
> >  1. species name
> >  2. taxonomy ID
> >  3. taxonomic level: K_kingdom, P_phylum, C_class, O_order, F_family, G_genus, and S_species
> >  4. reads assigned by Kraken
> >  5. additional reads added by Bracken: In order to estimate species abundance, Bracken reestimates the reads assigned by Kraken using bayesian reestimation. For details on the procedure, have a look into the [Bracken publication](https://peerj.com/articles/cs-104/).
> >  6.  sum of column 4 and column 5
> >  7. fraction of the reads assigned to the particular species and the total reads
> >
> {: .solution}
{: .question}


> <details-title>More details on using input other than Bracken</details-title>
>
> It is possible to calculate $$\alpha$$ and $$\beta$$ diversity also on other datasets than the Bracken output. Any tool that outputs taxonomy abundances can be used prior to the diversity analysis. To use **Krakentools** (as described here), the respective output file needs to be converted into the correct table format and filtered for the taxonomic rank "species". This step is not necessary when using Bracken output as already only the species level is listed.
>
{: .details}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare Galaxy and data

Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

We need now to import the data

> <hands-on-title>Import datasets</hands-on-title>
>
> 1. Import the following samples via link from [Zenodo]({{ page.zenodo_link }}) or Galaxy shared data libraries:
>
>    ```text
>    {{ page.zenodo_link }}/files/JC1A_Estimate_Abundance_at_Species_Level.tsv
>    {{ page.zenodo_link }}/files/JP4D_Estimate_Abundance_at_Species_Level.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Create a collection.
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
{: .hands_on}


# Calculating $$\alpha$$ diversity

## Theory of $$\alpha$$ diversity

**$$\alpha$$ diversity** describes the diversity within a community. There are several different indexes used to calculate $$\alpha$$ diversity because different indexes capture different aspects of diversity and have varying sensitivities to different factors. These indexes have been developed to address specific research questions, account for different ecological or population characteristics, or highlight certain aspects of diversity. 

![The image explains the concept of alpha diversity, which refers to the variety of species within a specific community or ecosystem. It compares three communities: Community A, Community B, and Community C. Both Community A and Community B contain five different shapes, each representing a different microbial species, indicating that they have the same level of alpha diversity. Community C, however, consists of only three different shapes, suggesting it has lower alpha diversity compared to Communities A and B. The image uses arrows and labels to highlight that Communities A and B have higher alpha diversity than Community C, with a note clarifying that each shape symbolizes a distinct microbial species.](./images/alphadiversity_metrics.png)(Source: [Pedro J Torres](https://medium.com/pjtorres-high-gut-alpha-diversity-and-health/high-alpha-diversity-and-health-65e5eca7fa36))

Metrics of $$\alpha$$ diversity can be grouped into different classes:

![The image is a conceptual diagram illustrating the concepts of richness and evenness in ecological communities using colored dots within four quadrants. The vertical axis represents richness, increasing upwards, while the horizontal axis represents evenness, increasing to the right. The top left quadrant shows moderate richness and evenness with a mix of colored dots. The top right quadrant, with a high number of evenly distributed colored dots, indicates high richness and evenness. The bottom left quadrant, having fewer and unevenly distributed dots, suggests lower richness and evenness. The bottom right quadrant displays moderate richness but lower evenness, with a somewhat uneven distribution of dots. This diagram visually explains how different communities can vary in species diversity and abundance.](./images/alpha_diversity_richness_evenness.png)

- **Richness** estimates the quantity of distinct species within a sample

   - **Margalef’s richness** indicates the estimated species richness, accounting for the community size. This metric takes into account that a larger community size can support a greater number of species ({% cite Margalef.1969 %})
                                          
      $$ D = \frac{(S - 1)}{\log(n)} $$

      With
      - $$S$$ the total number of species,
      - $$n$$ the total number of individuals in the sample                                                                                                                                                          
   - **Chao1** estimates the true species richness or diversity of a community, particularly when there might be rare or unobserved species. Chao1 estimates the number of unobserved species based on the number of singletons and doubletons. It assumes that there are additional rare species that are likely to exist but have not been observed. The estimation considers the number of unobserved singletons and doubletons and incorporates them into the observed species richness to provide an estimate of the true species richness ({% cite Chao.1992 %}).
   
      $$ 
      S_{chao1} = S_{obs} + \frac{n_{1}(n_{1} - 1)}{2(n_2 + 1)} 
      $$
   
      With:
      - \\(S_{obs}\\) the observed species richness, 
      - \\(n_{1}\\) the number of species represented by a single individual (singletons), 
      - \\(n_{2}\\) the number of species represented by two individuals (doubletons).
   
   - **ACE** (Abundance-based Coverage Estimator) takes into account the abundance distribution of observed species and incorporates the presence of rare or unobserved species. ACE estimates the number of unobserved species based on the abundance distribution and incorporates it into the observed species richness. It takes into account the relative rarity of observed species and uses this information to estimate the true species richness.    
                                                                              

- **Evenness** evaluates the relative abundances of species rather than their total count
  
   - **Pielou’s evenness** quantifies how close the community's diversity is to the maximum possible diversity. This index is calculated by taking the Shannon Diversity Index (which measures the overall diversity of the community) and dividing it by the maximum possible diversity given the observed species richness ({%cite Pielou.1966 %}).  

      $$ J = \frac{H'}{ln(S)} $$
                                                                                 
      With:
      - $$H'$$ the Shannon Weiner diversity Index
      - $$S$$ the total number of species in a sample, across all samples in dataset.
 

- **Diversity** incorporates both the relative abundances and total count of distinct species
  
   - **Shannons** index calculates the uncertainty in predicting the species identity of an individual that is selected from a community ({% cite Shannon.1948 %}).

      $$ 
      H' = - \sum_{i=1}^{S} p_i \ln (p_i) 
      $$
      
      With \\(p_i\\) the proportion of individuals of species $$i$$

      The index ranges from 0 to a maximum value that depends on the number of species and their relative abundances. The higher the Shannon index value, the greater the diversity within the community. 
                                                                                                                                    
   - **Berger-Parker** index expresses the proportional importance of the most abundant type. This index is highly biased by sample size and richness ({% cite Berger.1970 %} ).

      $$ 
      D = \frac{n_{max}}{N} 
      $$
   
      With:
      - \\(n_{max}\\) the abundance of the most dominant species
      - $$N$$ the total number of individuals (sum of all abundances)

      In contrast to the Shannon index, which considers both species richness and evenness, the Berger-Parker index emphasizes the dominance of a particular species.

   - **Simpsons** index calculates the probability that two individuals selected from a community will be of the same species. Small values are observed in datasets of high diversity and large values in datasets of low diversity ({% cite SIMPSON.1949 %}).

      $$ 
      D = \sum_{i=1}^{S} \left( \frac{n_i}{N} \right)^2 
      $$
      
      With:
      - \\(n_i\\) the number of individuals in species $$i$$
      - $$N$$ total number of individuals of all species
      - \\(\frac{n_i}{N} = p_i\\) the proportion of individuals of species $$i$$
      - $$S$$ the species richness.

      The index ranges from 0 to 1, with 1 representing maximum diversity. A value of 0.6 indicates that two randomly selected individuals from the community have 60% probability to belong to different species.

   - **Inverse Simpons** index is the transformation of Simpsons index that increases with increasing diversity. A higher Inverse Simpson's index value signifies a community with a greater number of species and a more even distribution of individuals among those species. The index ranges from 1 to the total number of species in the community, with higher values indicating higher diversity. 

   - **Fishers alpha** index describes the relationship between the number of species and the number of individuals in those species. Parametric index of diversity that assumes that the abundance of species follows a logarithmic series distribution ({% cite Fisher.1943 %}).

      $$ 
      S = a \cdot \ln \left(1+\frac{n}{a}\right) 
      $$

      With:                                                      
      - $$S$$ the number of taxa
      - $$n$$ the number of individuals 
      - $$a$$ the the Fisher's alpha. 

      A higher Fisher's alpha value indicates greater species diversity within the sampled area.

## Computing α diversity using Galaxy                                                  

**KrakenTools** is a suite of scripts designed to help Kraken users with downstream analysis of Kraken results. The Krakentool **Calculate alpha diversity** offers the possibility to calculate five different $$\alpha$$ diversity indexes:
1. Shannon's $$\alpha$$ diversity
2. Berger Parker's $$\alpha$$
3. Simpson's diversity
4. Inverse Simpson's diversity
5. Fisher's index

> <hands-on-title>Calculate α diversity with Krakentools</hands-on-title>
> 1. {% tool [Krakentools: Calculate alpha diversity](toolshed.g2.bx.psu.edu/repos/iuc/krakentools_alpha_diversity/krakentools_alpha_diversity/1.2+galaxy1) %} with the following parameters:
>    - {% icon param-collection %} *"Abundance file"*: uploaded Bracken output files
>    - *"Specify alpha diversity type"*: `Shannon's alpha diversity`
>
> 2. Run {% tool [Krakentools: Calculate alpha diversity](toolshed.g2.bx.psu.edu/repos/iuc/krakentools_alpha_diversity/krakentools_alpha_diversity/1.2+galaxy1) %} with the 4 other indexes
{: .hands_on}

> <question-title></question-title>
>
> 1. What are the values of 5 different $$\alpha$$ indexes?
> 2. How should we interpret the different $$\alpha$$ indexes?
> 2. Are the results consistent among the different indexes?
> 3. What is the dominant species in the samples? What is the problem we can see?
>
> > <solution-title></solution-title>
> >
> > 1. The results of five $$\alpha$$ indexes for both the JC1A and JP4D sample as well as an explanation of the meaning of these values.
> >     
> >     |                 | JC1A      | JP4D      |
> >     | --------------- | --------- | --------- |
> >     | Shannon         | 2.06      | 3.74      |
> >     | Berger-Parker   | 0.46      | 0.08      |
> >     | Simpson         | 0.74      | 0.97      |
> >     | Inverse Simpson | 3.92      | 28.84     |
> >     | Fisher          | 209.73    | 456.61    |
> >      
> > 2. Intepretation of the different indexes:
> >    - **Shannon index**: the index ranges from 0 to a maximum value that depends on the number of species and their relative abundances. The higher the Shannon index value, the greater the diversity within the community. A value of ~4 indicates a **relatively high level of diversity** within the community. 
> >     - **Berger-Parker index**: In contrast to the Shannon index, which considers both species richness and evenness, the Berger-Parker index emphasizes the dominance of a particular species.
> >        - For JC1A, a value of 0.46 suggests that **a single species dominates the community**, as it represents **46%** of the total individuals in the community. This indicates a relatively low level of species evenness, meaning that the abundance of individuals is heavily skewed towards one dominant species. A value of 0.46 indicates that the community is heavily influenced by one species, while the other species in the community are less abundant.
> >        - In the case of JP4D, the dominant species accounts for only **8 %** of the total individuals, which implies a **more balanced distribution of individuals** among different species compared to a higher Berger-Parker index value.
> >     - **Simpson's index**: The index ranges from 0 to 1, with 1 representing maximum diversity. Therefore, value of 0.97 indicates a **high level of species diversity and evenness** within the community: community is highly diverse, with a relatively even distribution of individuals among different species. In other words, the value of 0.97 indicates that two randomly selected individuals from the community have 97% probability to belong to different species. This implies a rich and balanced community where multiple species coexist in relatively equal abundance.
> >     - **Inverse Simpson's index**: The Inverse Simpson's index is the reciprocal of the Simpson's index, which quantifies species diversity and evenness within a community. A higher Inverse Simpson's index value signifies a community with a greater number of species and a more even distribution of individuals among those species. The index ranges from 1 to the total number of species in the community, with higher values indicating higher diversity. A value of **3.92** suggests a **relatively low level of species diversity** within the community and a value of 28.84 suggests a relatively high level of species diversity within the community.
> > 
> >     Index | JC1A | JP4D |
> >     --- | --- | ---
> >     Shannon | Less diversity as in JP4D | Relatively high level of diversity
> >     Berger-Parker | A single species dominates the community | mMre balanced distribution of individuals among different species |
> >     Simpson | Lower level of species diversity and evenness| High level of species diversity and evenness |
> >     Inverse Simpson | Relatively low level of species diversity | Relatively high level of species diversity |
> >     Fisher | Lower species diversity | Greater species diversity |
> >
> >
> > 3. The results are consistent as all indexes show JP4D to be the more diverse sample compared to JC1A.                                                                   
> > 4. When looking at the Bracken files, we see that the dominant species in this samples is `Homo sapiens` - so us! This should not be part of the samples and is probably due to contamination. This finding should be used to reanalyze the samples and remove the human contamination. For the sake of simplicity we will continue with the samples as is and assume the `Homo sapiens` to be a species in out samples.                                                     
> {: .solution}
>
{: .question}


> <comment-title></comment-title>
>
> Apart from Krakentools, there are at least two more tools available in Galaxy that can be used to calculate diversity indexes, 
> 
> - **QIIME 2** ({% cite Bolyen.2019 %})
>
>    QIIME 2 (Quantitative Insights Into Microbial Ecology 2) is a powerful open-source bioinformatics software package that provides a comprehensive suite of tools and methods for processing, analyzing, and visualizing microbiome data. It offers a modular approach to microbiome analysis, allowing researchers to build flexible analysis pipelines tailored to their specific research goals. The software supports a wide range of data types, including 16S rRNA gene sequencing, metagenomics, metatranscriptomics, and others.
> 
>    Some of the key features and functionalities of QIIME 2 include:
>    1. Diversity Analysis: QIIME 2 allows users to explore and quantify microbial diversity within and between samples. It provides metrics for alpha diversity (within-sample diversity) and beta diversity (between-sample diversity).
>    2. Data Import and Preprocessing
>    3. Taxonomic Assignments
>    4. Community Analysis
>    5. Phylogenetic Analysis
>    6. Statistical Analysiss.
>    7. Visualization
>
> - **[Vegan](https://github.com/vegandevs/vegan)**
> 
>    The vegan package is a community ecology package in the R programming language. It provides a wide range of tools and methods for analyzing and interpreting ecological data, particularly in the context of community ecology. The package is designed to handle multivariate data and offers various statistical techniques for studying species composition, diversity, and community dynamics.
>
>    The vegan package encompasses several functionalities, including:
>
>    1. Diversity Analysis: vegan offers numerous diversity indices, such as species richness, Shannon diversity index, Simpson index, and many others. These indices allow researchers to quantify the diversity of species within a community and compare diversity between different samples or groups.
>    2. Ordination Techniques
>    3. Community Classification:
>    4. Ecological Network Analysis
>    5. Ecological Indices
>    6. Plotting and Visualization
>
{: .comment}

# Calculating $$\beta$$ diversity 

## Theory of $$\beta$$ diversity

**$$\beta$$ diversity** measures the distance between two or more separate entities. It therefore describes the difference between two communities or ecosystems. 

There are **multiple indexes** used to calculate $$\beta$$ diversity because different indexes emphasize different aspects of compositional dissimilarity between communities or sites.

These indexes have been developed to address specific research questions, accommodate different data types, or provide insights into different dimensions of $$\beta$$ diversity.

- **Jaccard Index** measures the proportion of shared species between two samples ({% cite Jaccard.1912 %}).
  
   $$ 
   J(X, Y) =  \frac{\| X ∩ Y\|}{\| X ∪ Y\|} 
   $$
  
   With:
   - $$X ∩ Y$$ the intersection of sets $$X$$ and $$Y$$, i.e. elements common to both sets
   - $$X ∪ Y$$ the union of sets $$X$$ and $$Y$$, i.e. all unique elements from both sets combined
  
- **Sørensen Index** is similar to Jaccard Index, but accounts for species abundance ({% cite Srensen.1948 %}).

   $$
   DSC = \frac{2\| X ∩ Y\|}{\| X\|} + \| Y\| 
   $$

   With:
   - $$X ∩ Y$$ the intersection of sets $$X$$ and $$Y$$, i.e.elements common to both sets
   - $$ \| X\|$$ and $$\| Y \|$$ the cardinalities of the two sets, i.e. the number of elements in each set.
  
- **Bray-Curtis Dissimilarity** measures the dissimilarity of species abundances between two samples ({% cite Bray.1957 %}).

   $$ 
   BC_{ij} = 1 - \frac{2C_{ij}}{S_{i} + S_{j}}
   $$

   With:
   - \\(C_{ij}\\) the sum of the absolute differences in abundances between corresponding species in samples $$i$$ and $$j$$
   - \\(S_{i}\\) the total abundance or sum of species abundances in sample $$i$$
   - \\(S_{j}\\) the total abundance or sum of species abundances in sample $$j$$

   The higher the dissimilarity value, the greater the difference in species composition or abundances.
  
- **Kulczynski Dissimilarity** masures the dissimilarity in the proportional abundances of shared species.

   $$ 
   D = 1 - \frac{S_{AB}}{S_{A} + S_{B} - 2S_{AB}} 
   $$

   With:
   - \\(S_{AB}\\) the number of shared species between communities $$A$$ and $$B$$
   - \\(S_{A}\\) the number of species in community $$A$$
   - \\(S_{B}\\) the number of species in community $$B$$
  
- **UniFrac** incorporates information on phylogenetic distances between observed species in the computation. Can be calculated either weighted (accounts for abundances) or unweighted (accounts only for richness).

   ![The image consists of three phylogenetic trees, each representing different levels of community relatedness based on a parameter denoted as DD. The first tree on the left illustrates "Identical communities" with D=0.0D=0.0, showing branches that are closely related and color-coded similarly, indicating high similarity or identical species composition among the communities. The middle tree represents "Related communities" with D=0.5D=0.5, displaying a mix of colors on the branches, which suggests some degree of divergence and shared ancestry, indicating that these communities are related but not identical. The tree on the right depicts "Unrelated communities" with D=1.0D=1.0, characterized by distinctly different colors on separate branches, signifying a high level of dissimilarity and no shared recent ancestry among the communities. This visual representation highlights how community composition and relatedness can vary from being identical to completely unrelated.](./images/unifrac.png)

## Computing $$\beta$$ diversity using Galaxy   

> <hands-on-title>Calculate β diversity with Krakentools</hands-on-title>
> 1. {% tool [Krakentools: Calculate beta diversity (Bray-Curtis dissimilarity)](toolshed.g2.bx.psu.edu/repos/iuc/krakentools_beta_diversity/krakentools_beta_diversity/1.2+galaxy1) %} with the following parameters:
>   - {% icon param-collection %} *"Taxonomy file"*: uploaded Bracken output files
>   - *"Specify type of input file"*: `Bracken species abundance file`
>    
{: .hands_on}


> <question-title></question-title>
>
> 1. What is the Bray-Curtis dissimilarity calculated for the two samples?
> 2. How can we interpret Bray-Curtis dissimilarity between the two samples?
>
> > <solution-title></solution-title>
> >
> > 1. The output file gives you a table comparing sample 0 and 1, respectively. Consequently, comparing 0 to 0 and 1 to 1 results in a dissimilarity of 0, as those are exactly the same. Comparing sample 0 to sample 1 shows a Bray-Curtis dissimilarity of 0.701.
> > 2. The Bray-Curtis dissimilarity measures the dissimiliraty of two samples. Consequently, an output of 0 represents two samples that are exactly the same, while an output of 1 means they are maximally divergent. In our case, a Bray-Curtis dissimilarity of 0.7 suggests that there is a substantial difference in the species composition or abundances between the two communities being compared. The higher the dissimilarity value, the greater the difference in species composition or abundances.
> {: .solution}
>
{: .question}

# Evaluation of different diversity metrics

Bonilla-Rosso et al. did **performance evaluation** of different diversity metrics using simulated data sets {% cite BonillaRosso.2012 %}. However, it is important to note that none of the estimated metrics showed statistical similarity to their corresponding parameters in the source communities. Moreover, the results obtained were inconsistent across the samples.

This **inconsistency** can be attributed to the fact that individual metrics only provide a **specific** perspective on diversity and are prone to bias in their estimation, leading to incorrect ranking of the samples.

In summary, **relying solely on single-diversity metrics may not be enough** to accurately compare the diversity between two communities. Instead, we recommend utilizing multi-dimensional metrics to capture diverse rankings across different scales of diversity, which can be affected differently in manipulative studies.

**Multidimensional diversity metrics**, also known as multivariate diversity metrics, are quantitative measures that capture multiple dimensions or aspects of diversity simultaneously. These metrics go beyond single-diversity metrics, such as species richness or Shannon entropy, which provide a one-dimensional representation of diversity.

In contrast, multidimensional diversity metrics **take into account various attributes or characteristics** of species or communities to provide a more comprehensive understanding of diversity. These attributes can include species abundances, functional traits, phylogenetic relationships, or spatial distributions.

The choice and composition of dimensions in multidimensional diversity metrics depend on the research context and the specific objectives of the study. Some aspects, multidimensional diversity metrics take into account, include:
1. **Functional Diversity**: This metric considers the range and variation of functional traits among species within a community. It assesses the diversity of ecological roles and functional strategies present, contributing to ecosystem functioning and resilience.
2. **Phylogenetic Diversity**: This metric incorporates the evolutionary relationships among species within a community. It quantifies the diversity based on the length and topology of the phylogenetic tree, highlighting the evolutionary history and relatedness of species.
3. **Spatial Diversity**: This metric incorporates spatial patterns and distributions of species within a landscape or ecosystem. It considers the heterogeneity of habitats, connectivity, and the arrangement of species populations across space.

Multidimensional diversity metrics offer a **more nuanced and holistic perspective** on biodiversity, capturing different facets and dimensions of ecological variation. They provide insights into the ecological processes shaping communities and ecosystems and can be valuable in conservation planning, ecosystem management, and understanding the functional implications of biodiversity patterns.

> <details-title>More details on Multidimensional diversity metrics and Parameter q</details-title>
>
> Expressing the compositional complexity of an assemblage cannot be accomplished with a single numerical value. Traditional measures like diversities (Hill numbers) and entropies (Rényi entropies) vary in their **order $$q$$**, which determines **the extent to which rare or common species are emphasized**. The ranking and comparison of assemblages rely on the chosen value of $$q$$. 
>
> Instead of selecting a few measures to describe an assemblage, it is preferable to **present a continuous profile** that depicts diversity or entropy as a function of $$q$$ (where $$q \sep 0$$). This approach enables a visual comparison of the compositional complexities among multiple assemblages and facilitates the assessment of the evenness in the relative abundance distributions of the assemblages. In practice, the profile is typically plotted for values of $$q$$ ranging from 0 to $$q =$$ 3 or 4, beyond which there is usually little change.
>
> 
> - **Hill numbers**
>
>    ![The image presents a graph and accompanying text that illustrate the diversity profiles of two ecological communities, labeled A and B, each containing the same number of species (100 species) and individuals (1,000 individuals), but differing in evenness. The graph plots Hill numbers, which represent the effective number of species, against a parameter qq, which determines the sensitivity of the measure to species relative abundance. Profile A, depicted by a blue curve, shows a steep decline as qq increases, indicating a few dominant species and many rare species, reflecting lower evenness. Profile B, represented by a red horizontal line, maintains a constant Hill number across all values of qq, indicating that all species have a similar number of individuals, reflecting higher evenness. The text explains that Hill numbers provide a way to compare diversity profiles graphically and quantitatively, with differences in the distribution of individuals among species affecting the diversity measures. Community A, with greater imbalance and less evenness, exhibits lower order 1 and order 2 diversity compared to the more even Community B.](./images/hill_numbers.png "<a href="https://www.redalyc.org/journal/5117/511766773011/html/">Source</a>")
> 
>    Hill numbers, also known as diversity indices or diversity measures, are mathematical metrics used to quantify the diversity or richness of a biological community. They were developed by ecologist Robert H. Whittaker and are widely used in ecology and biodiversity studies.
>
>    Hill numbers provide a way to summarize and compare the diversity of different communities based on the abundance or occurrence of different species within those communities. These numbers take into account both the number of species present and their relative abundances. The higher the Hill number, the greater the diversity or richness of the community.
>
>    Hill numbers are often represented by the symbol $$D$$, followed by a subscript that indicates the order of diversity. The order of diversity determines the weight given to rare versus common species. Commonly used Hill numbers include:
>
>    1. Species richness ($$D_0$$): This is the simplest Hill number and represents the total number of species in a community, without considering their abundances. It provides a basic measure of biodiversity based on species count.
>    2. Shannon diversity index ($$D_1$$): This index incorporates both species richness and evenness. It takes into account both the number of species and their relative abundances, providing a more comprehensive measure of diversity.
>    3. Simpson diversity index ($$D_2$$): This index focuses on the dominance or concentration of species within a community. It considers both species richness and the probability that two individuals randomly selected from the community belong to the same species.
>
> - **Rényi entropy**
>
>    Rényi entropy is a concept in information theory and statistical physics introduced by Alfréd Rényi, a Hungarian mathematician. It is a generalization of the Shannon entropy, which measures the uncertainty or information content of a random variable or probability distribution.
>    The Rényi entropy of a discrete probability distribution is defined by the parameter $$\alpha$$, which determines the order of the entropy. The formula for calculating Rényi entropy is:
>
>    $$ 
>    H \alpha (P) = \frac{1}{1 - α} \dot log_{2}(\sum{i=1}^{N} pi^{\alpha}) 
>    $$
>
>    where $$P = {p_1, p_2, ..., p_N}$$ is the probability distribution of $$N$$ discrete events or states, and $$p_i$$ represents the probability of the ith event.
>
>    The value of $$\alpha$$ determines the properties of Rényi entropy. When $$\alpha = 1$$, Rényi entropy reduces to Shannon entropy, providing a measure of average uncertainty or information content. As $$\alpha$$ approaches 0, Rényi entropy converges to the minimum value, representing the most certain or least diverse distribution. Conversely, as $$\alpha$$ approaches infinity, Rényi entropy approaches the maximum value, indicating a uniform or maximally diverse distribution.
>
>    Rényi entropy has applications in various fields, including information theory, statistical physics, and data analysis. It offers a way to quantify the diversity or randomness of a system beyond the traditional Shannon entropy, allowing for a more nuanced understanding of information content and structure.
>
{: .details}


For further information on how to choose the best diversity metric also check:

1. [Measuring the diversity of the human microbiota with targeted next-generation sequencing](https://academic.oup.com/bib/article/19/4/679/2871295) {% cite BonillaRosso.2012 %}
2. [Understanding microbial community diversity metrics derived from metagenomes: performance evaluation using simulated data sets](https://academic.oup.com/femsec/article/82/1/37/567182) {% cite Finotello.2018 %}

# Conclusion

In this tutorial, we look how to calculate $$\alpha$$ and $$\beta$$ diversity from microbiome data. We apply **Krakentools** to calculate the $$\alpha$$ and $$\beta$$ diversity of two microbiome sample datasets.


topic: microbiome
tutorial: nanopore-16S-metagenomics
title: 16S Microbial analysis with Nanopore data
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: ["metabarcoding", "16S", "nanopore", "microgalaxy", "plants"]
content_hash: 73914667de12a306d88ac460ef5fd109
last_modified: 2026-02-26T21:12:17.329978
zenodo_link: https://zenodo.org/record/4274812
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/nanopore-16S-metagenomics/tutorial.html

content:
__Healthy soils__ are an essential element in maintaining the planet’s ecological balance. That is why their protection must be considered a priority in order to guarantee the well-being of humanity. The alteration of microbial populations often precedes changes in the physical and chemical properties of soils, so monitoring their condition can serve to predict their future evolution, allowing to develop strategies to __mitigate ecosystem damage__.

Advances in sequencing technologies have opened up the possibility of using the study of taxa present in bacterial communities as indicators of soil condition. For such an approach to be possible, variations in microbial populations need to be less affected by spatial factors than by human-derived alterations, which has been confirmed by various investigations ({% cite Hermans2016 %}, {% cite Fierer2006 %}).

Among the environmental variables that have shown the greatest correlation with alterations in the composition of bacterial communities are soil pH, the carbon-nitrogen ratio (C:N), and the co-concentrations of Olsen P (a measure of plant available phosphorus), aluminium and copper (Fig. 1).


![Bacterial bioindicators](../../images/metagenomics-nanopore/bioindicators.jpg "Relationship between bacterial community composition and soil parameters. The radius of each circle represents the taxon abundance that was accounted for by each soil variable, based on adjusted R-squared values from distance-based multivariate multiple regression analyses. The univariate relationship between the abundance of each taxon and soil variables, calculated using Pearson’s correlation coefficient, is represented by the color of the circle. Phyla or classes are ordered according to overall abundance in all the samples, from most abundant (top) to least abundant (bottom). Image published originally in {% cite Hermans2016 %}.")


On the other hand, many species of microorganisms establish complex __symbiotic relationships__ with plant organisms. The fraction of the substrate that is directly influenced by root secretions and associated microorganism is known as __rhizosphere__. Thus, for example, bacteria of the genus _Bacillus_, _Pseudomonas_ or _Burkholderia_ appear associated with the plant roots, protecting them from pathogenic microorganisms.

![Bacterial communities](../../images/metagenomics-nanopore/bacteria_community.png "Representation of the relationships between root microbiomes and plant organisms.")

In this tutorial, we will use sequencing data obtained through the __MinION sequencer__ (Oxford Nanopore Technologies) with two objectives: 1) evaluate the health status of soil samples and 2) study how microbial populations are modified by their interaction with plant roots. This example was inspired by {% cite Brown2017 %}.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Background on data

Next, we will introduce some details about the datasets that we are going to use to perform the analysis.

## The datasets

In this example, we will use a dataset originally hosted in the __NCBI SRA database__, with the accession number [SRP194577](https://www.ncbi.nlm.nih.gov/sra?term=SRP194577). To obtain the samples, the DNA was extracted using the Zymo Research Kit, followed by __PCR amplification of 16S rRNA genes__. Sequencing was carried out on a Nanopore Minion device. We are going to use four datasets, corresponding to two experimental conditions:

- Soil:
	- Surface sample (0-5 cm): `bulk_top.fastq.gz`
	- Deep sample (10-15 cm): `bulk_bottom.fastq.gz`
- Rhizosphere:
	- Surface sample (0-5 cm): `rhizosphere_top.fastq.gz`
	- Deep sample (10-15 cm): `rhizosphere_bottom.fastq.gz`

> <question-title></question-title>
>
>  Why do we sequence the 16S rRNA genes for analyzing microbial communities?
>
> > <solution-title></solution-title>
> > There are several reasons to use these genes as taxonomic markers. First, they are constituent parts of ribosomes, a piece of highly-conserved biological machinery, which makes them universally distributed. On the other hand, these genes present regions with varying degrees of sequence variability, which allows them to be used as a species-specific signature.
> {: .solution}
{: .question}


# Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from Zenodo:
>
>    - Open the file {% icon galaxy-upload %} __upload__ menu
>    - Click on __Collection__ tab
>    - Click of the **Paste/Fetch** button
>    - Paste the Zenodo links and press <kbd>Start</kbd> and <kbd>Build</kbd>
>
>      ```
>      https://zenodo.org/record/4274812/files/bulk_bottom.fastq.gz
>      https://zenodo.org/record/4274812/files/bulk_top.fastq.gz
>      https://zenodo.org/record/4274812/files/rhizosphere_bottom.fastq.gz
>      https://zenodo.org/record/4274812/files/rhizosphere_top.fastq.gz
>      ```
>
>    - Assign a name to the new collection: `soil collection`
>
> {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
{: .hands_on}

# Assess datasets quality

Before starting to work on our datasets, it is necessary to assess their quality. This is an essential step if we aim to obtain a __meaningful downstream analysis__.


> <details-title>FASTQ sequence quality format</details-title>
>
> FASTQ format is a text-based format for storing both a biological sequence and its corresponding quality scores. Both the sequence letter and the quality score are encoded with a single ASCII character for brevity. You can find more info in the [Wikipedia article](https://en.wikipedia.org/wiki/FASTQ_format).
>
{: .details}

## Quality control using FastQC and MultiQC

__FastQC__ is one of the most widely used tools to __check the quality__ of the samples generated by High Throughput Sequencing (HTS) technologies.

> <hands-on-title>Quality check</hands-on-title>
>
> 1. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72+galaxy1) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: `soil collection`
>
> 2. Rename the outputs as `FastQC unprocessed: Raw` and `FastQC unprocessed: Web`
> 3. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy1) %} with the following parameters:
>    - In *"Results"*:
>      - *"Which tool was used generate logs?"*: `FastQC`
>      - {% icon param-collection %} *"Dataset collection"*:
>
>		- `FastQC unprocessed: Raw`
> 4. Click on the {% icon galaxy-eye %} (eye) icon and inspect the generated HTML file
>
>
{: .hands_on}

__FastQC__ provides information on various parameters, such as the range of quality values across all bases at each position. __MultiQC__ allows summarizing the output of different outputs from __FastQC__.

![FastQC plots length](../../images/metagenomics-nanopore/fastqc_sequence_length_distribution_plot.png "Sequence length distribution")

In the first place, we are going to analyse the sequence length distribution of the different datasets.

> <question-title></question-title>
>
> 1. Can you explain the sequence length distribution plot?
>
> > <solution-title></solution-title>
> >
> > The main peak, around 1700 bp, corresponds approximately to the length of the gene coding for 16S rRNA. There is also a secondary peak around 200 bp, which may be due to __truncated amplifications__, or as a result of __non-specific hybridization of primers__ used for PCR.
> >
> {: .solution}
{: .question}


![FastQC plots quality](../../images/metagenomics-nanopore/fastqc_per_base_sequence_quality_plot.png "Per base sequence quality")



If we examine figure 3, we can see that up to 3000 bp the quality of our sequencing data is around a __Phred score__ of 12, which is a relatively low value compared to other sequencing technologies. This is explained because Nanopore reads poses __high error rates in the basecalled reads__ (10% as compared to 1% for Illumina). However, Nanopore sequencing generates __very long reads__ (in theory only limited by the mechanisms of extraction of the genetic material), enabling the sequencing of the complete 16S rRNA gene, which makes it possible to identify bacterial taxa at higher resolution.

![FasTQC plots CG](../../images/metagenomics-nanopore/fastqc_per_sequence_gc_content_plot.png "Per sequence GC content")

Let's have a look at figure 4, which shows the content of the GC by sequence.

> <question-title></question-title>
>
> What can cause GC content to show a bimodal peak?
>
> > <solution-title></solution-title>
> >
> > There are several possible causes for the presence of twin peaks in the GC content. One possible explanation is the presence of adapters in the sequence. Another possible cause is some kind of contamination, such as chimeras.
> >
> {: .solution}
>
{: .question}

> <comment-title></comment-title>
> For more information on the topic, please see [the quality control tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).
{: .comment}


# Improve the dataset quality

In order to improve the quality of our data, we will use two tools presented above, __porechop__ and __fastp__.

## Adapter and chimera removal with porechop

Nanopore sequencing technology requires to ligate adapters to both ends of genomic material to facilitate the strand capture and loading of a processive enzyme at the 5'end, boosting the effectiveness of the sequencing process. Adapter sequences should be removed because they __can interfere with aligment of reads__ to 16S rRNA gene reference database, for which we will use the __porechop__ tool ({% cite rrwick2017 %}).

> <comment-title></comment-title>
> If you are interested in Nanopore sequecing technology, you can find more information in {% cite Jain2016 %}.
>
{: .comment}


On the other hand, __chimeric sequences__ are considered a contaminant and should be removed because they can result in artificial inflation of the microbial diversity. With __Porechop__ you can eliminate them.

> <hands-on-title>Remove adapters with porechop</hands-on-title>
>
> 1. {% tool [Porechop](toolshed.g2.bx.psu.edu/repos/iuc/porechop/porechop/0.2.3) %} with the following parameters:
>    - *"Input FASTA/FASTQ"*: {% icon param-collection %} `soil collection`
>    - *"Output format for the reads"*: `fastq`
> 2. Rename the output as `soil collection trimmed`
>
{: .hands_on}

## Filter sequences with fastp

To increase the specificity of the analysis, we will select the reads with lengths between 1000 bp and 2000 bp, which are more informative from a taxonomic point of view, because they include both preserved and hypervariable regions of the 16S rRNA gene. In addition, sequences will be filtered on a minimum average read quality score of 9, according to the recommendations from {% cite Nygaard2020 %}. This stage will be carried out through the use of __fastp__ ({% cite Chen_2018 %}), an open-source tool designed to process FASTQ files.

> <hands-on-title>Filter sequence with fastp</hands-on-title>
>
> 1. {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.20.1+galaxy0) %} with the following parameters:
>    - *"Single-end or paired reads"*: `Single-end`
>    - {% icon param-collection %} *"Dataset collection"*: `soil collection trimmed`
>        - In *"Adapter Trimming Options"*:
>            - *"Disable adapter trimming"*: `Yes`
>    - In *"Filter Options"*:
>        - In *"Quality filtering options"*:
>            - *"Qualified quality phred"*: `9`
>        - In *"Length filtering options"*:
>            - *"Length required"*: `1000`
>            - *"Maximum length"*: `2000`
>    - In *"Read Modification Options"*:
>        - *"PolyG tail trimming"*: `Disable polyG tail trimming`
>
> 2. Rename the output as `soil collection processed`
>
>
{: .hands_on}

# Re-evaluate datasets quality

After processing the sequences, we are going to analyze them again using __FastQC__ and __MultiQC__ to see if we have managed to correct the anomalies that we had detected.

> <hands-on-title>Quality check</hands-on-title>
>
> 1. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72+galaxy1) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: `soil collection processed`
>
> 4. Rename the outputs as `FastQC processed: Raw` and `FastQC processed: Web`
> 3. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy1) %} with the following parameters:
>    - In *"Results"*:
>      - *"Which tool was used generate logs?"*: `FastQC`
>      - {% icon param-collection %} *"Dataset collection"*:
>
>		- `FastQC processed: Raw`
> 4. Click on the {% icon galaxy-eye %} (eye) icon and inspect the generated HTML file
>
>
{: .hands_on}

We can verify that after processing the samples, the GC content presents a unimodal distribution, which indicates that the anomalies in the sequences have been successfully eliminated (Figure 6).

![FasTQC plots GC](../../images/metagenomics-nanopore/fasqc_per_sequence_content_plot_clean.png "Per sequence GC content in processed samples")

# Assign taxonomic classifications

One of the key steps in metagenomic data analysis is to identify the taxon to which the individual reads belong. Taxonomic classification tools are based on microbial genome databases to identify the origin of each sequence.

## Taxonomic classification with Kraken2

To perform the taxonomic classification we will use __Kraken2__ ({% cite Wood2019 %}). This tool uses the minimizer method to sample the k-mers (all the read's subsequences of length _k_) in a deterministic fashion in order to reduce memory constumption and processing time. In addition, it masks low-complexity sequences from reference sequences by using __dustmasker__.

{% snippet topics/microbiome/faqs/kraken.md %}

For this tutorial, we will use the __SILVA database__ ({% cite Quast2012 %}). It includes over 3.2 million 16S rRNA sequences from the _Bacteria_, _Archaea_ and _Eukaryota_ domains.

> <hands-on-title>Assign taxonomic labels with Kraken2</hands-on-title>
>
> 1. {% tool [Kraken2](toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.0.8_beta+galaxy0) %} with the following parameters:
>    - *"Single or paired reads"*: `Single`
>    - {% icon param-collection %} *"Dataset collection"*: `soil collection processed`
>    - *"Print scientific names instead of just taxids"*: `Yes`
>    - *"Confidence"*: `0.1`
>    - In *"Create Report"*:
>        - *"Print a report with aggregrate counts/clade to file"*: `Yes`
>        - *"Format report output like Kraken 1's kraken-mpa-report"*: `Yes`
>    - *"Select a Kraken2 database"*: `Silva (Created: 2020-06-24T164526Z, kmer-len=35, minimizer-len=31, minimizer-spaces=6)`
>
>    > <comment-title></comment-title>
>    >
>    > A confidence score of 0.1 means that at least 10% of the k-mers should match entries in the database. This value can be reduced if a less restrictive taxonomic assignation is desired.
>    {: .comment}
>
{: .hands_on}



# Analyze taxonomic assigment

Once we have assigned the corresponding taxa to each sequence, the next step is to properly visualize the data, for which we will use the __Krona pie chart__ tool ({% cite Ondov_2011 %}). But before that, we need to adjust the format of the data output from __Kraken2__.

> <hands-on-title>Adjust dataset format</hands-on-title>
>
> 1. {% tool [Reverse](toolshed.g2.bx.psu.edu/repos/iuc/datamash_reverse/datamash_reverse/1.1.0) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: `Report: Kraken2 on collection`
>
> 2. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: `Reverse on collection`
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"Find pattern"*: `\|`
>            - *"Replace with"*: `\t`
>
> 3. {% tool [Remove beginning](Remove beginning1) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: `Replace Text on collection`
>
{: .hands_on}

## Visualize the taxonomical classification with Krona

__Krona__ allows hierarchical data to be explored with zooming, multi-layered pie charts. With this tool, we can easily visualize the composition of the bacterial communities and compare how the populations of microorganisms are modified according to the conditions of the environment.

> <hands-on-title>Visualize metagenomics analysis results</hands-on-title>
>
> 1. {% tool [Krona pie chart](toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1) %} with the following parameters:
>    - *"What is the type of your input data"*: `Tabular`
>    - {% icon param-collection %} *"Dataset collection"*: `Remove beginning on collection`
>    - *"Provide a name for the basal rank"*: `Bacteria`
>
{: .hands_on}

Let's take a look at the result. Using the search bar we can check if certain taxa are present.

<iframe id="krona" src="krona_all.html" frameBorder="0" width="100%" height="900px"> ![Krona](../../images/metagenomics-nanopore/krona.png) </iframe>

> <question-title></question-title>
>
> 1. What can we say about the health status of the soil samples?
> 2. Are there significant differences in the bacterial population structure between soil samples and rhizosphere samples?
>
> > <solution-title></solution-title>
> >
> > 1. The low presence of _Alphaproteobacteria_ (including members of the order _Rhizobiales_), as well as the abundance of _Bacteroidetes_ and _Gammaproteobacteria_, suggest that the soil is highly exposed to phosphorus. This mineral is strongly associated with the anthropogenic activity as it is an important component of agrochemicals.
> > 2. We can observe that there are important differences in the structure of the bacterial populations between the soil and rhizosphere samples. Particularly significant is the increase in phylum _Planctomycetes_, which are usually abundant in the rhizosphere.
> >
> {: .solution}
>
{: .question}

# Conclusion

In this tutorial we used MinION Nanopore sequencing data to study the health status of soil samples and the structure of bacterial populations. The results indicate that the soil suffers some degree of erosion as a result of their exposure to agrochemicals. We have also been able to study how the composition of microbial communities is modified in presence of plant organisms.


topic: microbiome
tutorial: taxonomic-profiling
title: Taxonomic Profiling and Visualization of Metagenomic Data
difficulty: introductory
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: ["metagenomics", "taxonomic profiling", "microgalaxy"]
content_hash: 5e94b737c102d8c042e39e43005c2508
last_modified: 2026-02-26T21:12:17.367799
zenodo_link: https://zenodo.org/record/7871630
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/taxonomic-profiling/tutorial.html

content:
The term **"microbiome"** describes "a characteristic microbial community occupying a reasonably well-defined habitat which has distinct physio-chemical properties. The term thus not only refers to the microorganisms involved but also encompasses their theatre of activity" ({% cite whipps1988mycoparasitism %}).

Microbiome data can be gathered from different environments such as soil, water or the human gut. The biological interest lies in general in the question how the microbiome present at a specific site influences this environment. To study a microbiome, we need to use indirect methods like metagenomics or metatranscriptomics.

**Metagenomic samples** contain DNA from different organisms at a specific site, where the sample was collected. Metagenomic data can be used to find out which organisms coexist in that niche and which genes are present in the different organisms.
**Metatranscriptomic samples** include the transcribed gene products, thus RNA, that therefore allow to not only study the presence of genes but additionally their expression in the given environment. The following tutorial will focus on metagenomics data, but the principle is the same for metatranscriptomics data.

The investigation of microorganisms present at a specific site and their relative abundance is also called **"microbial community profiling"**.
The main objective is to identify the microorganisms that are present within the given sample. This can be achieved for all known microbes, where the DNA sequence specific for a certain species is known.

For that we try to **identify the taxon** to which each individual read belongs.

{% snippet topics/microbiome/faqs/taxon.md %}

For metagenomic data analysis we start with sequences derived from DNA fragments that are isolated from the sample of interest. Ideally, the sequences from all microbes in the sample are present. The underlying idea of taxonomic assignment is to compare the DNA sequences found in the sample (reads) to DNA sequences of a database. When a read matches a database DNA sequence of a known microbe, we can derive a list with microbes present in the sample.

When talking about taxonomic assignment or taxonomic classification, most of the time we actually talk about two methods, that in practice are often used interchangeably:
- **taxonomic binning**: the clustering of individual sequence reads based on similarities criteria and assignation of clusters to reference taxa
- **taxonomic profiling**: classification of individual reads to reference taxa to extract the relative abundances of the different taxa

The reads can be obtained from **amplicon sequencing**  (e.g. 16S, 18S, ITS), where only specific gene or gene fragments are targeted (using specific primers) or **shotgun metagenomic sequencing**, where all the accessible DNA of a mixed community is amplified (using random primers). 
This tutorial focuses on reads obtained from shotgun metagenomic sequencing. 

## Taxonomic profiling

Tools for taxonomic profiling can be divided into three groups. Nevertheless, all of them require a pre-computed database based on previously sequenced microbial DNA or protein sequences.
1. **DNA-to-DNA**: comparison of sequencing reads with genomic databases of DNA sequences with tools like Kraken ({% cite Wood2014 %})
2. **DNA-to-Protein** : comparison of sequencing reads with protein databases (more computationally intensive because all six frames of potential DNA-to amino acid translations need to be analyzed) with tools like DIAMOND)
3. **Marker based**: searching for marker genes (e.g. 16S rRNA sequence) in reads, which is quick, but introduces bias, with tools like MetaPhlAn ({% cite blanco2023extending %})

The comparison of reads to database sequences can be done in different ways, leading to three different types of taxonomic assignment:

- **Genome based** approach

   Reads are aligned to reference genomes. Considering the coverage and breadth, genomes are used to measure genome abundance. Furthermore, genes can be analyzed in genomic context. Advantages of this method are the high detection accuracy, that the unclassified percentage is known, that all SNVs can be detected and that high-resolution genomic comparisons are possible. This method takes medium compute cost.

- **Gene based** approach

   Reads are aligned to reference genes. Next, marker genes are used to estimate species abundance. Furthermore, genes can be analyzed in isolation for presence or absence in a specific condition. The major advantage is the detection of the pangenome (entire set of genes within a species). Major disadvantages are the high compute cost, low detection accuracy and that the unclassified percentage is unknown. At least intragenic SNVs can be detected and low-resolution genomic comparison is possible.

- **k-mer based** approach

   Databases as well as the samples DNA are broken into strings of length $$k$$ for comparison. From all the genomes in the database, where a specific k-mer is found, a lowest common ancestor (LCA) tree is derived and the abundance of k-mers within the tree is counted. This is the basis for a root-to-leaf path calculation, where the path with the highest score is used for classification of the sample. By counting the abundance of k-mers, also an estimation of relative abundance of taxa is possible. The major advantage of k-mer based analysis is the low compute cost. Major disadvantages are the low detection accuracy, that the unclassified percentage is unknown and that there is no gene detection, no SNVs detection and no genomic comparison possible. An example for a k-mer based analysis tool is Kraken, which will be used in this tutorial.

After this theoretical introduction, let's now get hands on analyzing an actual dataset!

## Background on data

The dataset we will use for this tutorial comes from an oasis in the Mexican desert called Cuatro Ciénegas ({% cite Okie.2020 %}). The researchers were interested in genomic traits that affect the rates and costs of biochemical information processing within cells. They performed a whole-ecosystem experiment, thus fertilizing the pond to achieve nutrient enriched conditions.

Here we will use 2 datasets:
- `JP4D`: a microbiome sample collected from the Lagunita Fertilized Pond
- `JC1A`: a **control** samples from a control mesocosm.

The datasets differ in size, but according to the authors this doesn't matter for their analysis of genomic traits. Also, they underline that differences between the two samples reflect trait-mediated ecological dynamics instead of microevolutionary changes as the duration of the experiment was only 32 days. This means that depending on available nutrients, specific lineages within the pond grow more successfully than others because of their genomic traits.

The datafiles are named according to the first four characters of the filenames.
It is a collection of paired-end data with R1 being the forward reads and R2 being the reverse reads. Additionally, the reads have been trimmed using __cutadapt__ as explained in the [Quality control tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare Galaxy and data

Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

Now, we need to import the data

> <hands-on-title>Import datasets</hands-on-title>
>
> 1. Import the following samples via link from [Zenodo]({{ page.zenodo_link }}) or Galaxy shared data libraries:
>
>    ```text
>    {{ page.zenodo_link }}/files/JC1A_R1.fastqsanger.gz
>    {{ page.zenodo_link }}/files/JC1A_R2.fastqsanger.gz
>    {{ page.zenodo_link }}/files/JP4D_R1.fastqsanger.gz
>    {{ page.zenodo_link }}/files/JP4D_R2.fastqsanger.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Create a paired collection.
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
{: .hands_on}

# k-mer based taxonomic assignment with Kraken2

Our input data is the DNA reads of microbes present at Cuatro Ciénegas.

To find out which microorganisms are present, we will compare the reads of the sample to a reference database, i.e. sequences of known microorganisms stored in a database, using **Kraken2** ({% cite wood2019improved %}).

{% snippet topics/microbiome/faqs/kraken.md %}

For this tutorial, we will use the PlusPF database which contains the Standard (archaea, bacteria, viral, plasmid, human, UniVec_Core), protozoa and fungi data.

> <details-title>What is in the PlusPF database</details-title>
> 
> Database | Origin
> --- | ---
> Archaea | RefSeq complete archaeal genomes/proteins
> Bacteria | RefSeq complete bacterial genomes/proteins
> Plasmid | RefSeq plasmid nucleotide/protein sequences
> Viral | RefSeq complete viral genomes/proteins
> Human | GRCh38 human genome/proteins
> Fungi | RefSeq complete fungal genomes/proteins
> Plant | RefSeq complete plant genomes/proteins
> Protozoa | RefSeq complete protozoan genomes/proteins
> UniVec_Core | A subset of UniVec, NCBI-supplied database of vector, adapter, linker, and primer sequences that may be contaminating sequencing projects and/or assemblies, chosen to minimize false positive hits to the vector database
>
> The databases have been prepared for Kraken2 by Ben Leagmead and [details can found on his page](https://benlangmead.github.io/aws-indexes/k2).
{: .details}

> <hands-on-title>Assign taxonomic labels with Kraken2</hands-on-title>
>
> 1. {% tool [Kraken2](toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.1.3+galaxy1) %} with the following parameters:
>    - *"Single or paired reads"*: `Paired`
>      - {% icon param-collection %} *"Collection of paired reads"*: Input paired collection
>    - *"Confidence"*: `0.1`
>
>      A confidence score of 0.1 means that at least 10% of the k-mers should match entries in the database. This value can be reduced if a less restrictive taxonomic assignation is desired.
>
>    - In *"Create Report"*:
>        - *"Print a report with aggregrate counts/clade to file"*: `Yes`
>    - *"Select a Kraken2 database"*: most recent `Prebuilt Refseq indexes: PlusPF`
>
{: .hands_on}

**Kraken2** will create two outputs for each dataset

- **Classification**: tabular files with one line for each sequence classified by Kraken and 5 columns:

   1. `C`/`U`: a one letter indicating if the sequence classified or unclassified
   2. Sequence ID as in the input file
   3. NCBI taxonomy ID assigned to the sequence, or 0 if unclassified
   4. Length of sequence in bp (`read1|read2` for paired reads)
   5. A space-delimited list indicating the lowest common ancestor (LCA) mapping of each k-mer in the sequence     
      
      For example, `562:13 561:4 A:31 0:1 562:3` would indicate that:
      1. The first 13 k-mers mapped to taxonomy ID #562
      2. The next 4 k-mers mapped to taxonomy ID #561
      3. The next 31 k-mers contained an ambiguous nucleotide
      4. The next k-mer was not in the database
      5. The last 3 k-mers mapped to taxonomy ID #562     
      
      `|:|` indicates end of first read, start of second read for paired reads

   For JC1A:

   ```
   Column 1	Column 2	Column 3	Column 4	Column 5
   U 	MISEQ-LAB244-W7:91:000000000-A5C7L:1:1101:13417:1998 	0 	151|190 	A:17 0:15 2055:5 0:1 2220095:5 0:74 |:| 0:3 A:53 2:2 0:32 204455:1 2823043:5 0:60
   U 	MISEQ-LAB244-W7:91:000000000-A5C7L:1:1101:15782:2187 	0 	169|173 	0:101 37329:1 0:33 |:| 0:10 2751189:5 0:30 1883:2 0:39 2609255:5 0:48
   U 	MISEQ-LAB244-W7:91:000000000-A5C7L:1:1101:11745:2196 	0 	235|214 	0:173 2282523:5 2746321:2 0:21 |:| 0:65 2746321:2 2282523:5 0:108
   U 	MISEQ-LAB244-W7:91:000000000-A5C7L:1:1101:18358:2213 	0 	251|251 	0:35 281093:5 0:3 651822:5 0:145 106591:3 0:21 |:| 0:64 106591:3 0:145 651822:5
   U 	MISEQ-LAB244-W7:91:000000000-A5C7L:1:1101:14892:2226 	0 	68|59 	0:34 |:| 0:25
   U 	MISEQ-LAB244-W7:91:000000000-A5C7L:1:1101:18764:2247 	0 	146|146 	0:112 |:| 0:112
   C 	MISEQ-LAB244-W7:91:000000000-A5C7L:1:1101:12147:2252 	9606 	220|220 	9606:148 0:19 9606:19 |:| 9606:19 0:19 9606:148
   ```

   > <question-title></question-title>
   >
   > For JC1A sample
   > 1. Is the first sequence in the file classified or unclassified?
   > 2. What is the taxonomy ID assigned to the first classified sequence?
   > 3. What is the corresponding taxon?
   >
   > > <solution-title></solution-title>
   > > 1. unclassified
   > > 2. 9606, for the line 7
   > > 3. 9606 corresponds to Homo sapiens when looking at NCBI.
   > {: .solution}
   {: .question}

- **Report**: tabular files with one line per taxon and 6 columns or fields

   1. Percentage of fragments covered by the clade rooted at this taxon
   2. Number of fragments covered by the clade rooted at this taxon
   3. Number of fragments assigned directly to this taxon
   4. A rank code, indicating
      - (U)nclassified
      - (R)oot
      - (D)omain
      - (K)ingdom
      - (P)hylum
      - (C)lass
      - (O)rder
      - (F)amily
      - (G)enus, or
      - (S)pecies

      Taxa that are not at any of these 10 ranks have a rank code that is formed by using the rank code of the closest ancestor rank with a number indicating the distance from that rank. E.g., `G2` is a rank code indicating a taxon is between genus and species and the grandparent taxon is at the genus rank.

   5. NCBI taxonomic ID number
   6. Indented scientific name

   ```
   Column 1	Column 2	Column 3	Column 4	Column 5	Column 6
   76.85 	105397 	105397 	U 	0 	unclassified
   23.15 	31742 	1197 	R 	1 	root
   22.20 	30450 	312 	R1 	131567 	cellular organisms
   12.58 	17256 	3768 	D 	2 	Bacteria
   8.77 	12028 	2868 	P 	1224 	Proteobacteria
   4.94 	6779 	3494 	C 	28211 	Alphaproteobacteria
   1.30 	1782 	1085 	O 	204455 	Rhodobacterales
   0.43 	593 	461 	F 	31989 	Rhodobacteraceae
   0.05 	74 	53 	G 	265 	Paracoccus
   ```

   > <question-title></question-title>
   >
   > 1. What are the percentage on unclassified for JC1A and JP4D?
   > 2. What are the kindgoms found for JC1A and JP4D?
   > 3. Where might the eukaryotic DNA come from?
   > 4. How is the diversity of Proteobacteria in JC1A and JP4D?
   >
   > > <solution-title></solution-title>
   > >
   > > 1. 77% for JC1A and 90% for JP4D
   > > 2. Kindgoms:
   > >    - JC1A: 13% Bacteria, 9% Eukaryota, 0.03% Virus
   > >    - JP4D: 9% Bacteria, 0.7% Eukaryota
   > > 3. It seems to be human contamination
   > > 4. JC1A seems to have a big diversity of classes and species of Proteobacteria. JP4D seems more dominated by Aphaproteobacteria.
   > {: .solution}
   >
   {: .question}

Getting an overview of the assignation is not straightforward with the **Kraken2** outputs directly. We can use visualisation tools for that.

A "simple and worthwile addition to Kraken for better abundance estimates" ({% cite Ye.2019 %}) is called __Bracken__ (Bayesian Reestimation of Abundance after Classification with Kraken). Instead of only using proportions of classified reads, it takes a probabilistic approach to generate final abundance profiles. It works by re-distributing reads in the taxonomic tree: "Reads assigned to nodes above the species level are distributed down to the species nodes, while reads assigned at the strain level are re-distributed upward to their parent species" ({% cite Lu.2017 %}).

> <hands-on-title>Estimate species abundance with Bracken</hands-on-title>
>
> 1. {% tool [Braken](toolshed.g2.bx.psu.edu/repos/iuc/bracken/est_abundance/3.1+galaxy0) %} with the following parameters:
>     - {% icon param-collection %} *"Kraken report file"*: **Report** output of **Kraken**
>     - *"Select a kmer distribution"*: `PlusPF`, same as for Kraken
>
>        It is important to choose the same database that you also chose for Kraken2
>
>     - *"Level"*: `Species`
>     - *"Produce Kraken-Style Bracken report"*: `yes`
>
{: .hands_on}

# Visualization of taxonomic assignment

Once we have assigned the corresponding taxa to each sequence, the next step is to properly visualize the data. There are several tools for that:
- __Krona__ ({% cite Ondov.2011 %})
- __Phinch__ ({% cite Bik.2014 %})
- __Pavian__ ({% cite Breitwieser.2020 %})

## Visualisation using Krona

__Krona__ creates an interactive HTML file allowing hierarchical data to be explored with zooming, multi-layered pie charts. With this tool, we can easily visualize the composition of the bacterial communities and compare how the populations of microorganisms are modified according to the conditions of the environment.

Kraken outputs can not be given directly to **Krona**, they first need to be converted.

__Krakentools__ ({% cite Lu.2017 %}) is a suite of tools to work on Kraken outputs. It include a tool designed to translate results of the Kraken metagenomic classifier to the full representation of NCBI taxonomy. The output of this tool can be directly visualized by the Krona tool.

> <hands-on-title>Convert Kraken report file</hands-on-title>
>
> 1. {% tool [Krakentools: Convert kraken report file](toolshed.g2.bx.psu.edu/repos/iuc/krakentools_kreport2krona/krakentools_kreport2krona/1.2+galaxy2) %} with the following parameters:
>    - {% icon param-collection %} *"Kraken report file"*: **Report** collection of **Kraken**
>
> 2. Inspect the generated output for JC1A
{: .hands_on}


> <question-title></question-title>
>
> ```
> 3869 	k__Bacteria
> 2868 	k__Bacteria 	p__Proteobacteria
> 3494 	k__Bacteria 	p__Proteobacteria 	c__Alphaproteobacteria
> 1085 	k__Bacteria 	p__Proteobacteria 	c__Alphaproteobacteria 	o__Rhodobacterales
> 461 	k__Bacteria 	p__Proteobacteria 	c__Alphaproteobacteria 	o__Rhodobacterales 	f__Rhodobacteraceae
> 53 	k__Bacteria 	p__Proteobacteria 	c__Alphaproteobacteria 	o__Rhodobacterales 	f__Rhodobacteraceae 	g__Paracoccus
> 10 	k__Bacteria 	p__Proteobacteria 	c__Alphaproteobacteria 	o__Rhodobacterales 	f__Rhodobacteraceae 	g__Paracoccus 	s__Paracoccus_pantotrophus
> 6 	k__Bacteria 	p__Proteobacteria 	c__Alphaproteobacteria 	o__Rhodobacterales 	f__Rhodobacteraceae 	g__Paracoccus 	s__Paracoccus_sanguinis
> 4 	k__Bacteria 	p__Proteobacteria 	c__Alphaproteobacteria 	o__Rhodobacterales 	f__Rhodobacteraceae 	g__Paracoccus 	s__Paracoccus_sp._AK26
> 1 	k__Bacteria 	p__Proteobacteria 	c__Alphaproteobacteria 	o__Rhodobacterales 	f__Rhodobacteraceae 	g__Paracoccus 	s__Paracoccus_sp._MA
> ```
>
> 1. What are the different columns?
> 2. What are the lines?
>
> > <solution-title></solution-title>
> >
> > 1. Column 1 seems to correspond to the number of fragments covered by a taxon, the columns after represent the different taxonomic level (from kingdom to species)
> > 2. A line is a taxon with its hierarchy and the number of reads assigned to it
> {: .solution}
>
{: .question}

Let's now run **Krona**

> <hands-on-title>Generate Krona visualisation</hands-on-title>
> 1. {% tool [Krona pie chart](toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1+galaxy0) %} with the following parameters:
>    - *"Type of input data"*: `Tabular`
>    - {% icon param-collection %} *"Input file"*: output of **Krakentools**
>
> 2. Inspect the generated file
{: .hands_on}

<iframe id="krona" src="krona-kraken.html" frameBorder="0" width="100%" height="900px"> ![Krona chart with multi-layered pie chart representing the community profile with in the center the higher taxonomy levels (i.e. domain) and on the exterior the more detailed ones (i.e. species)](./images/krona-kraken.png) </iframe>

> <question-title></question-title>
>
> 1. What are the percentage on unclassified for JC1A and JP4D?
> 2. What are the kindgoms found for JC1A and JP4D?
> 3. Where might the eukaryotic DNA come from?
> 4. How is the diversity of Proteobacteria in JC1A and JP4D?
>
> > <solution-title></solution-title>
> >
> > 1. 78% for JC1A and 90% for JP4D
> > 2. Kindgoms:
> >    - JC1A: 13% Bacteria, 10% Eukaryota, 0.03% Virus
> >    - JP4D: 10% Bacteria, 0.7% Eukaryota
> > 3. It seems to be human contamination
> > 4. JC1A seems to have a big diversity of classes and species of Proteobacteria. JP4D seems more dominated by Aphaproteobacteria.
> {: .solution}
>
{: .question}


## Visualization using Pavian

__Pavian__ (pathogen visualization and more) ({% cite Breitwieser.2020 %}) is an interactive visualization tool for metagenomic data. It was developed for the clinical metagenomic problem to find a disease-causing pathogen in a patient sample, but it is useful to analyze and visualize any kind of metagenomics data.

> <hands-on-title>Launch Pavian</hands-on-title>
> 1. {% tool [Pavian](interactive_tool_pavian) %} with the following paramters:
>     - {% icon param-collection %} *"Kraken and MetaPhlAn-style reports"*: **Report** collection of **Kraken**
>
{: .hands_on}

**Pavian** runs a Galaxy Interactive tool. You can access it when it become orange.

> <hands-on-title>Interact with Pavian</hands-on-title>
>
> 1. Open **Pavian**
>
>    {% snippet faqs/galaxy/interactive_tools_open.md %}
>
> 2. Import data
>    1. Click on `Use data on server`
>    2. Select both samples
>    3. Click on `Read selected directories`
>    4. Check you have a table in `Available sample sets` looks like
>
>       X | FormatOK | Include | Name | ReportFile | ReportFilePath
>       --- | --- | --- | --- | --- | ---
>       1 | X | X | JP4AD | JP4AD | /home/shiny//JP4AD
>       2 | X | X | JC1A | JC1A | /home/shiny//JC1A
>
>    5. Click on `Save table`
>
> 3. Click on `Results Overview` in the left panel
>
{: .hands_on}

This page shows the summary of the classifications in the selected sample set:

![Screenshot of a table with 2 rows (JP4A and JC1A) and columns: Name, Number of raw reads, Classified reads, Chordate reads, Artificial reads, Unclassified reads, Microbial reads, Bacterial reads, Viral reads, Fungal reads, Protozoan reads. The cells have a barchart that shows the relation of the cell value to other cell values in the same category](./images/pavian-kraken-results-overview.png)

> <question-title></question-title>
>
> 1. Does both sample have same size?
> 1. What are the percentage of classified reads for JC1A and JP4D?
> 2. Are the percentage of bacterial reads similar?
>
> > <solution-title></solution-title>
> >
> > 1. JP4D has much more reads than JC1A
> > 2. 10.2% for JP4D and 23.1% for JC1A
> > 3. 12.6% for JC1A and 9.22% for JP4D. So similar magnitude orders
> {: .solution}
>
{: .question}

Let's now inspect assignements to reads per sample.

> <hands-on-title>Inspect samples with Pavian</hands-on-title>
>
> 1. Click on `Sample` in the left panel
> 2. Select `JC1A` in the `Select sample` drop-down on the top
>
>    The first view gives a Sankey diagram for one sample:
>
>    ![Sankey plot with taxonomy hierarchy from domain on the left to species on the right](./images/pavian-kraken-sankey-JC1A.png)
>
>    > <question-title></question-title>
>    >
>    > 1. What is a Sankey diagram?
>    > 2. What are the different set of values represented as the horizontal axis?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. A sankey diagram is a visualization used to depict a flow from one set of values to another
>    > > 2. The taxonomy hierarchy from domain on the left to species on the right
>    > {: .solution}
>    >
>    {: .question}
>
> 3. Click on `Proteobacteria` in the Sankey plot
> 4. Inspect the created graph on the right
>
>    ![Bar plot with the number of reads across all samples for Proteobacteria. JP4A bar is much higher than JC1A one. The turqoise bar shows the number of reads that are identified at the specific taxon; the orange bar shows the number of reads identified at children of the specified taxon.](./images/pavian-kraken-proteobacteria.png)
>
>    > <question-title></question-title>
>    >
>    > 1. Are the number of reads assigned to Proteobacteria similar for both samples?
>    > 2. Why?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. JP4A has many more reads assigned to Proteobacteria than JC1A
>    > > 2. JP4A has many more reads initialls
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}

We would like now to compare both samples.

> <hands-on-title>Inspect samples with Pavian</hands-on-title>
>
> 1. Click on `Comparison` in the left panel
> 2. Select `%`  and unclick `Reads` in the blue area drop-down on the top
> 3. Click on `Domain` green button
>
>    ![Screenshot of a table with 4 rows (Bacteria, Eukaryota, Archaea, Viruses) and / columns (Name, Rank, TID, Max, JP4A, JC1A, Lineage)](./images/pavian-kraken-comparison-domain.png)
>
>    > <question-title></question-title>
>    >
>    > Is there similar proportion of Bacteria in both samples?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > JP4D has much higher proportion of Bacteria (> 93%>) than JC1A (57%), which contains quite a lot of Eukaryote
>    > {: .solution}
>    >
>    {: .question}
>
> 4. Select `Homo sapiens` in the `Filter taxa` box below the green buttons
>
>    > <question-title></question-title>
>    >
>    > Is there similar proportion of Bacteria in both samples?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > After human filtering, both samples have similar proportion of Bacteria
>    > {: .solution}
>    >
>    {: .question}
>
> 3. Click on `Class` green button
>
>    > <question-title></question-title>
>    >
>    > 1. How are the diversities of classes in both samples?
>    > 2. What could it biologically mean given that JC1A is a control and JP4D a sample from fertilized pond?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. JP4D seems highly dominated by a Alphaproteobacteria class. JC1A has also a majority of Alphaproteobacteria, but also significant proportions of Betaproteobacteria, Gammaproteobacteria, Flavobacteria, Actinomycetia
>    > > 2. Alphaproteobacteria seems to have a survival advantage in the new environment. According to the authors this correlates with specific genomic traits that enable them to cope better with high nutrient availability.
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}

Once you are done with Pavian, you should delete it in your history so the corresponding job is killed.

> <details-title>Visualize the taxonomical classification with Phinch</details-title>
>
> __Phinch__ ({% cite Bik.2014 %}) is another tools to visualize large biological datasets like our taxonomic classification. Taxonomy Bar Charts, Bubble Charts, Sankey Diagrams, Donut Partitions and Attributes Column Chart can be generated using this tool.
>
> As a first step, we need to convert the Kraken output file into a kraken-biom file to make it accessible for Phinch. For this, we need to add a metadata file, provided on [Zenodo]({{ page.zenodo_link }}/files/metadata.tabular).
> When generating a metadata file for your own data, you can take this as an example and apply the [general guidelines](http://qiime.org/documentation/file_formats.html).
>
> > <hands-on-title>Phinch</hands-on-title>
> > 1. Import the metadata tabular from [Zenodo]({{ page.zenodo_link }}) or Galaxy shared data libraries:
> >
> >    ```text
> >    {{ page.zenodo_link }}/files/metadata.tabular
> >    ```
> >
> > 1. Use {% tool [Kraken-biom](toolshed.g2.bx.psu.edu/repos/iuc/kraken_biom/kraken_biom/1.2.0+galaxy1) %} to convert Kraken2 report into the correct format for phinch with the following parameters.
> >    - {% icon param-collection %} *"Input"*: **Report** output of **Kraken2**
> >    - {% icon param-file %} *"Sample Metadata file"*: Metadata tabular
> >    - *"Output format"*: `JSON`
> >
> > 2. {% tool [Phinch Visualisation](interactive_tool_phinch) %} with the following paramters:
> >    - *"Input"*: `Kraken-biom output file`
> {: .hands_on}
>
> **Phinch** runs a Galaxy Interactive tool. You can access it when it become orange.
>
> > <hands-on-title>Interact with Phinch</hands-on-title>
> >
> > 1. Open **Phinch**
> >
> >    {% snippet faqs/galaxy/interactive_tools_open.md %}
> >
> >    The first pages shows an overview of your samples. Here, you have the possibility to further filter your data, for example by date or location, depending on which information you provided in your metadata file.
> > 
> >    > <question-title></question-title>
> >    >
> >    > 1. How many sequence reads do the samples contain?
> >    >
> >    > > <solution-title></solution-title>
> >    > >
> >    > > 1. JC1A (Phinch name: 0) contains 56008 reads, while JP4D (Phinch name: 1) contains 242438 reads
> >    > {: .solution}
> >    {: .question}
> >
> > 2. Click on `Proceed to gallery` to see an overview of all visualization options.
> {: .hands_on}
>
> Let's have a look at the **taxonomy bar chart**. Here, you can see the abundance of different taxa depicted in different colors in your samples. On top of the chart you can select which rank is supposed to be shown in the chart.  You can also change the display options to for example switch between value und percentage. 
>
> > <question-title></question-title>
> >
> > 1. What information can you get from hovering over a sample?
> > 2. How many percent of the sample reads are bacteria and how many are eukaryota?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. the taxon’s name and the taxonomy occurrence in the sample
> > > 2. choose kingdom and hover over the bars to find "taxonomy occurence in this sample":
> > >    - Sample 0: 75,65 % bacteria; 24,51 % eukaryota
> > >    - Sample 1: 92,70 % bacteria; 6,87 % eukaryota
> > {: .solution}
> {: .question}
>
> Let's go back to the gallery and choose the **bubble chart**. Here, you can find the distribution of taxa across the whole dataset at the rank that you can choose above the chart. When hovering over the bubbles, you get additional information concerning the taxon. 
> 
> > <question-title></question-title>
> >
> > 1. Which is the most abundant Class?
> > 2. How many reads are found in both samples?
> >
> > > <solution-title></solution-title>
> > >
> > > To order the bubbles according to their size you can choose the `list` option shown right next to the taxonomy level. Clicking on one bubble gives you the direct comparison of the distribution of this taxon in the different samples. 
> > > 1. The most abundant Class is Alphaproteobacteria
> > > 2. With 18.114 reads in Sample 0 and 153.230 reads in sample 1
> > {: .solution}
> {: .question}
>
>
> Another displaying option is the **Sankey diagram**, that is depicting the abundance of taxonomies as a flow chart. Again, you can choose the taxonomy level that you want to show in your diagram. When clicking on one bar of the diagram, this part is enlarged for better view.
>
> The **donut partition** summarizes the microbial community according to non-numerical attributes. In the drop-down menu at the top right corner, you can switch between the different attributes provided in the metadata file. In our case, you can for example choose the 'environmental medium' to see the difference between sediment and water (It doesn't really make a lot of sense in our very simple dataset, as this will show the same result as sorting them by sample 0 and 1, but if attributes change across different samples this might be an interesting visualization option). When clicking on one part of the donut you will also find the distribution of the taxon across the samples. On the right hand side you can additionally choose if you’d like to have dynamic y axis or prefer standard y axis to compare different donuts with each other.
>
> The **attributes column chart** summarizes the microbial community according to numerical attributes. In the drop-down menu at the top right corner, you can switch between the different attributes provided in the metadata file. In our case, you can for example choose the 'geographic location' to (again, it doesn't really make a lot of sense in our very simple dataset, as this will show the same result as sorting them by sample 0 and 1, but if attributes change across different samples this might be an interesting visualization option).
> 
> Once you are done with Phinch, you should delete it in your history so the corresponding job is killed.
{: .details}

# Choosing the right tool

When it comes to taxonomic assignment while analyzing metagenomic data, **Kraken2** is not the only tool available. Several papers do benchmarking of different tools ({% cite Meyer.2022 %},{% cite Sczyrba.2017 %},{% cite Ye.2019 %}).

> <details-title>Benchmarking taxonomic classification tools</details-title>
>
> The benchmarking papers present different methods for comparing the available tools:
> - The **CAMI challenge** is based on results of different labs that each used the CAMI dataset to perform their analysis on and send it back to the authors.
> - {% cite Ye.2019 %} performed all the analysis themselves.
>
> Additionally, the datasets used for both benchmarking approaches differ:
> - **CAMI**: only ~30%-40% of reads are simulated from known taxa while the rest of the reads are from novel taxa, plasmids or simulated evolved strains.
> - {% cite Ye.2019 %} used International Metagenomics and Microbiome Standards Alliance (IMMSA) datasets, wherein the taxa are described better.
>
> When benchmarking different classification tools, several metrics are used to compare their performance:
> 1. **Precision**: proportion of true positive species identified in the sample divided by number of total species identified by the method.
> 2. **Recall**: proportion of true positive species divided by the number of distinct species actually in the sample.
> 3. Precision-recall curve: each point represents the precision and recall scores at a specific abundance threshold, the **area under the precision-recall curve (AUPR)**
> 4. **L2 distance**: representation of abundance profiles → how accurately the abundance of each species or genera in the resulting classification reflects the abundance of each species in the original biological sample (“ground truth”)
>
> When it comes to taxonomic profiling, thus investigating the abundance of specific taxa, the biggest problem is the abundance bias. It is introduced during isolation of DNA (which might work for some organisms better then for others) and by PCR duplicates during PCR amplification.
{: .details}

> <details-title>Profiling tools</details-title>
>
> Profilers, which are tools that investigate relative abundances of taxa within a dataset, fall into three groups depending on their performance:
> 1. Profilers, that correctly predict relative abundances
> 2. Precise profilers (suitable, when many false positives would increase cost and effort in downstream analysis)
> 3. Profilers with high recall (suitable for pathogen detection, when the failure of detecting an organism can have severe negative consequences)
>
> However, some characteristics are common to all profilers:
> - Most profilers only perform well until the family level
> - Drastic decrease in performance between family and genus level, while little change between order and family level
> - Poorer performance of all profilers on CAMI datasets compared to International Metagenomics and Microbiome Standards Alliance (IMMSA)
> - Fidelity of abundance estimates decreases notably when viruses and plasmids were present
> - High numbers of false positive calls at low abundance
> - Taxonomic profilers vs profiles from taxonomic binning:
> Precision and recall of the taxonomic binners were comparable to that of the profilers;
> abundance estimation at higher ranks was more problematic for the binners
{: .details}

Tool | Approach | Available in Galaxy | CAMI challenge | {% cite Ye.2019 %}
--- | --- | --- | --- | ---
mOTUs |  | No | Most memory efficient |
MetaPhlAn | Marker genes | Yes | | Recommended for low computational requirements (< 2 Gb of memory)
DUDes | | No | |
FOCUS |  | No | Fast, most memory efficient |
Bracken | k-mer | Yes | Fast |
Kraken | k-mer | Yes | Fastest; most memory efficient | Good performance metrics; very fast on large numbers of samples; allow custom databases when high amounts of memory (>100 Gb) are available

# Marker gene based approach using MetaPhlAn 

In this tutorial, we follow second approach using **MetaPhlAn** ({% cite truong2015metaphlan2 %}). MetaPhlAn is a computational tool for profiling the composition of microbial communities (Bacteria, Archaea and Eukaryotes) from metagenomic shotgun sequencing data (i.e. not 16S) at species-level. MetaPhlAn 4 relies on ~5.1M unique clade-specific marker genes identified from ~1M microbial genomes (~236,600 references (bacterial, archeal, viral and eukaryotic) and 771,500 metagenomic assembled genomes) spanning 26,970 species-level genome bins.

It allows:
- unambiguous taxonomic assignments;
- accurate estimation of organismal relative abundance;
- species-level resolution for bacteria, archaea, eukaryotes and viruses;
- strain identification and tracking
- orders of magnitude speedups compared to existing methods.
- microbiota strain-level population genomics
- MetaPhlAn clade-abundance estimation

The basic usage of MetaPhlAn consists in the identification of the clades (from phyla to species and strains in particular cases) present in the microbiota obtained from a microbiome sample and their relative abundance.

**MetaPhlAn** in Galaxy can not directly take as input a paired collection but expect 2 collections: 1 with forward data and 1 with reverse. Before launching **MetaPhlAn**, we need to split our input paired collection.

> <hands-on-title>Assign taxonomic labels with MetaPhlAn</hands-on-title>
>
> 1. {% tool [Unzip collection](__UNZIP_COLLECTION__) %} with the following parameters:
>    - {% icon param-collection %} *"Paired input to unzip"*: Input paired collection
>
> 2. {% tool [MetaPhlAn](toolshed.g2.bx.psu.edu/repos/iuc/metaphlan/metaphlan/4.1.1+galaxy4) %} with the following parameters:
>    - In *"Inputs"*
>      - *"Input(s)"*: `Fasta/FastQ file(s) with microbiota reads`
>        - *"Fasta/FastQ file(s) with microbiota reads"*: `Paired-end files`
>          - {% icon param-collection %} *"Forward paired-end Fasta/FastQ file with microbiota reads"*: output of **Unzip collection** with **forward** in the name
>          - {% icon param-collection %} *"Reverse paired-end Fasta/FastQ file with microbiota reads"*: output of **Unzip collection** with **reverse** in the name
>    - In *"Outputs"*:
>        - *"Output for Krona?"*: `Yes`
{: .hands_on}


5 files and a collection are generated by **MetaPhlAn** {% icon tool %}:

- `Predicted taxon relative abundances`: tabular files with the **community profile**

   ```
   #SampleID 	Metaphlan_Analysis 		
   #clade_name 	NCBI_tax_id 	relative_abundance 	additional_species
   k__Bacteria 	2 	100.0 	
   k__Bacteria|p__Bacteroidetes 	2|976 	94.38814 	
   k__Bacteria|p__Proteobacteria 	2|1224 	5.61186 	
   k__Bacteria|p__Bacteroidetes|c__CFGB45935 	2|976| 	94.38814 	
   k__Bacteria|p__Proteobacteria|c__Alphaproteobacteria 	2|1224|28211 	5.61186 	
   k__Bacteria|p__Bacteroidetes|c__CFGB45935|o__OFGB45935 	2|976|| 	94.38814 	
   k__Bacteria|p__Proteobacteria|c__Alphaproteobacteria|o__Rhodobacterales 	2|1224|28211|204455 	5.61186 	
   k__Bacteria|p__Bacteroidetes|c__CFGB45935|o__OFGB45935|f__FGB45935 	2|976||| 	94.38814 	
   k__Bacteria|p__Proteobacteria|c__Alphaproteobacteria|o__Rhodobacterales|f__Rhodobacteraceae 	2|1224|28211|> 204455|    31989 	5.61186 	
   k__Bacteria|p__Bacteroidetes|c__CFGB45935|o__OFGB45935|f__FGB45935|g__GGB56609 	2|976|||| 	94.38814 	
   k__Bacteria|p__Proteobacteria|c__Alphaproteobacteria|o__Rhodobacterales|f__Rhodobacteraceae|g__Phycocomes 	2| 1224|    28211|204455|31989|2873978 	5.61186 	
   k__Bacteria|p__Bacteroidetes|c__CFGB45935|o__OFGB45935|f__FGB45935|g__GGB56609|s__GGB56609_SGB78025 	2| 976||||| 	94.    38814 	
   k__Bacteria|p__Proteobacteria|c__Alphaproteobacteria|o__Rhodobacterales|f__Rhodobacteraceae|g__Phycocomes|     s__Phycocomes_zhengii 	2|1224|28211|204455|31989|2873978|2056810 	5.61186 	
   k__Bacteria|p__Bacteroidetes|c__CFGB45935|o__OFGB45935|f__FGB45935|g__GGB56609|s__GGB56609_SGB78025| t__SGB78025 	2|    976|||||| 	94.38814 	
   k__Bacteria|p__Proteobacteria|c__Alphaproteobacteria|o__Rhodobacterales|f__Rhodobacteraceae|g__Phycocomes|     s__Phycocomes_zhengii|t__SGB31485 	2|1224|28211|204455|31989|2873978|2056810| 	5.61186 	
   ```

   Each line contains 4 columns:
   1. the lineage with different taxonomic levels, from high level taxa (kingdom: `k__`) to more precise taxa
   2. the NCBI taxon ids of the lineage taxonomic level
   3. the relative abundance found for our sample for the lineage
   4. any additional species

   > <question-title></question-title>
   >
   > 1. Which kindgoms have been identified for JC1A?
   > 2. For JP4D?
   > 3. How is the diversity for JP4D compared to **Kraken** results?
   >
   > > <solution-title></solution-title>
   > > 1. All reads have been unclassified so no kindgom identified
   > > 2. Bacteria, no eukaryotes
   > > 3. Diversity is really reduced for JP4D using **MetaPhlAn**, compared to the one identified with **Kraken**
   > {: .solution}
   {: .question}

- `Predicted taxon relative abundances for Krona`: same information as the previous files but formatted for > visualization using **Krona**

   ```
   Column 1	Column 2	Column 3	Column 4	Column 5	Column 6	Column 7	Column 8	Column 9	Column 10
   94.38814 		Bacteria 	Bacteroidetes 	CFGB45935 	OFGB45935 	FGB45935 	GGB56609 	GGB56609 SGB78025 	
   5.61186 		Bacteria 	Proteobacteria 	Alphaproteobacteria 	Rhodobacterales 	Rhodobacteraceae 	Phycocomes 	Phycocomes zhengii 	
   94.38814 		Bacteria 	Bacteroidetes 	CFGB45935 	OFGB45935 	FGB45935 	GGB56609 	GGB56609 SGB78025 	SGB78025
   5.61186 		Bacteria 	Proteobacteria 	Alphaproteobacteria 	Rhodobacterales 	Rhodobacteraceae 	Phycocomes 	Phycocomes zhengii 	SGB31485 
   ```

   Each line represent an identified taxons with 9 columns:
   - Column 1: The percentage of reads assigned the taxon
   - Column 2-9: The taxon description at the different taxonomic levels from Kindgom to more precise taxa

- A **collection** with the same information as in the tabular file but splitted into different files, one per taxonomic level
- `BIOM file` with **community profile** in BIOM format, a common and standard format in microbiomics and used as the input for tools like mothur or Qiime.

- `Bowtie2 output` and `SAM file` with the results of the sequence mapping on the reference database

Let's now run **Krona** to visualize the communities

> <hands-on-title>Generate Krona visualisation</hands-on-title>
> 1. {% tool [Krona pie chart](toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1) %} with the following parameters:
>    - *"Type of input data"*: `tabular`
>    - {% icon param-collection %} *"Input file"*: Krona output of **Metaphlan**
> 
> 2. Inspect the generated file
{: .hands_on}

As pointed before, the community looks a lot less diverse than with **Kraken**. This is probably due to the reference database, which is potentially not complete enough yet to identify all taxons. Or there are too few reads in the input data to cover enough marker genes. Indeed, no taxon has been identified for JC1A, which contains much less reads than JP4D. However, **Kraken** is also known to have high number of false positive. A benchmark dataset or mock community where the community DNA content is known would be required to correctly judge which tool provides the better taxonomic classification.

# Conclusion

In this tutorial, we look how to get the community profile from microbiome data. We apply **Kraken2** or **MetaPhlAn** to assign taxonomic labels to two microbiome sample datasets. We then visualize the results using **Krona**, **Pavian** and **Phinch** to analyze and compare the datasets. Finally, we discuss important facts when it comes to choosing the right tool for taxonomic assignment.å


topic: microbiome
tutorial: dada-16S
title: Building an amplicon sequence variant (ASV) table from 16S data using DADA2
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["metabarcoding", "16S", "microgalaxy"]
content_hash: 8b733ab61a3d8e437432cd2fe97a2aa3
last_modified: 2026-02-26T21:12:17.271537
zenodo_link: https://doi.org/10.5281/zenodo.800651
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/dada-16S/tutorial.html

content:
The investigation of environmental microbial communities and microbiomes has been revolutionized by the 
development of high-throughput amplicon sequencing. In amplicon sequencing a particular genetic locus, 
for example the 16S rRNA gene (or a part of it) in bacteria, is amplified from DNA extracted from the community of interest, 
and then sequenced on a next-generation sequencing platform. This technique removes the need to culture 
microbes in order to detect their presence, and cost-effectively provides a deep census of a microbial community.

> <comment-title>Sources</comment-title>
>
> This tutorial is significantly based on [DADA2 Pipeline Tutorial](https://benjjneb.github.io/dada2/tutorial.html) and ["Introduction to dada2"](https://bioconductor.org/packages/devel/bioc/vignettes/dada2/inst/doc/dada2-intro.html)
>
{: .comment}

{% include topics/microbiome/tutorials/mothur-miseq-sop/background_16S.md %}

To process 16S amplicon sequencing data, several tools exist: **QIIME2** ({% cite Bolyen_2019 %}), **Mothur** ({% cite Schloss_2009 %}), **Lotus2** ({% cite _zkurt_2022 %}), and **DADA2** ({% cite Callahan_2016 %}). All these tools can be used in Galaxy. In this tutorial we will use **DADA2**.

> <comment-title>Processing 16S amplicon data in Galaxy using other tools than DADA2</comment-title>
>
> Tutorials are available for the other tools in Galaxy:
> - [QIIME 2 Cancer Microbiome Intervention](https://docs.qiime2.org/jupyterbooks/cancer-microbiome-intervention-tutorial/index.html#)
> - [QIIME 2 Moving Pictures](https://docs.qiime2.org/2024.2/tutorials/moving-pictures-usage)
> - [16S Microbial Analysis with mothur]({% link topics/microbiome/tutorials/mothur-miseq-sop/tutorial.md %})
>
{: .comment}

The output of the dada2 pipeline is a feature table of amplicon sequence variants (an ASV table): A matrix with columns corresponding to samples and rows to ASVs, in which the value of each entry is the number of times that ASV was observed in that sample. This table is analogous to the traditional OTU table, except that one gets the actual sequences of the "true" sequence variants instead of rather anonymous and abstract entities called OTU1, OTU2, etc. 

The process of amplicon sequencing introduces errors into the sequencing data, and these errors 
severely complicate the interpretation of the results. **DADA2** implements a novel algorithm that models the errors introduced during amplicon sequencing, and uses that error model to infer the true sample composition. 

**DADA2** replaces the traditional "OTU-picking" step in amplicon sequencing workflows, producing instead higher-resolution tables of amplicon sequence variants (ASVs).

{% include topics/microbiome/tutorials/mothur-miseq-sop/background_otus.md %}

As seen in the paper introducing DADA2 ({% cite Callahan_2016 %}) and in [further benchmarking](http://benjjneb.github.io/dada2/SotA.html), the **DADA2** method is more sensitive and specific than traditional OTU methods: DADA2 detects real biological variation missed by OTU methods while outputting fewer spurious sequences.

To illustrate the 16S amplicon data processing using **DADA2**, we will use dataset generated by the Schloss lab in their
[their mothur MiSeq SOP](https://www.mothur.org/wiki/MiSeq_SOP): FASTQ files generated by 2x250 Illumina Miseq amplicon sequencing of the V4 region of the 16S rRNA gene from gut samples collected longitudinally from a mouse post-weaning. They [describe the experiment](https://www.mothur.org/wiki/MiSeq_SOP#Logistics) as follows:

> *The Schloss lab is interested in understanding the effect of normal variation in the gut microbiome on host health. To that end,
> we collected fresh feces from mice on a daily basis for 365 days post weaning. During the first 150 days post weaning (dpw),
> nothing was done to our mice except allow them to eat, get fat, and be merry. We were curious whether the rapid change in
> weight observed during the first 10 dpw affected the stability microbiome compared to the microbiome observed between days
> 140 and 150.*
> 
{: .quote}

To speed up analysis for this tutorial, we will use only a subset of this data. We will look at a single mouse at 20 different time points (10 early, 10 late).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare Galaxy and data

Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>History creation</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

The starting point for the **DADA2** pipeline is a set of demultiplexed FASTQ files corresponding to the samples in your amplicon sequencing study. That is, **DADA2** expects there to be an individual FASTQ file for each sample (or two FASTQ files, one forward and one reverse, for each sample). Demultiplexing is often performed at the sequencing center, but if that has not been done there are a variety of tools do accomplish this.

In addition, **DADA2** expects that there are no non-biological bases in the sequencing data. This may require pre-processing with outside software if, as is relatively common, your PCR primers were included in the sequenced amplicon region.

For this tutorial, we have 20 pairs of files. For example, the following pair of files:

- `F3D0_S188_L001_R1_001.fastq`
- `F3D0_S188_L001_R2_001.fastq`

The first part of the file name indicates the sample: `F3D0` here signifies that this sample was obtained from Female 3 on Day 0.

The rest of the file name is identical, except for `_R1` and `_R2`, this is used to indicate the forward and reverse reads respectively.

> <hands-on-title>Import datasets</hands-on-title>
>
> 1. Import the following samples via link from [Zenodo]({{ page.zenodo_link }}) or Galaxy shared data libraries:
>
>    > <solution-title>List of Zenodo URLs</solution-title>
>    > ```
>    > https://zenodo.org/record/800651/files/F3D0_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D0_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D141_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D141_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D142_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D142_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D143_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D143_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D144_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D144_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D145_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D145_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D146_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D146_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D147_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D147_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D148_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D148_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D149_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D149_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D150_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D150_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D1_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D1_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D2_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D2_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D3_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D3_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D5_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D5_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D6_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D6_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D7_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D7_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D8_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D8_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D9_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D9_R2.fastq
>    > https://zenodo.org/record/800651/files/Mock_R1.fastq
>    > https://zenodo.org/record/800651/files/Mock_R2.fastq
>    > ```
>    {: .solution }
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

40 files are a lot of files to manage. Luckily Galaxy can make life a bit easier by allowing us to create **dataset collections**. This enables us to easily run tools on multiple datasets at once.

Since we have **paired-end** data, each sample consist of two separate FASTQ files:
- one containing the forward reads, and 
- one containing the reverse reads.

We can recognize the pairing from the file names, which will differ only by `_R1` or `_R2` in the filename. We can tell Galaxy about this paired naming convention, so that our tools will know which files belong together. We do this by building a **List of Dataset Pairs**

> <hands-on-title>Organizing our data into a sorted paired collection</hands-on-title>
>
> 1. Create a paired collection named `Raw Reads`, rename your pairs with the sample name
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
> 2. {% tool [Sort collection](__SORTLIST__) %} with the following parameters:
>    - {% icon param-collection %} *"Input Collection"*: `Raw Reads`
>
> 3. Rename output collection to `Sorted Raw Reads`
>
>    {% snippet faqs/galaxy/collections_rename.md %}
>
{: .hands_on}

We can now start processing the reads using **DADA2**.

{% include _includes/cyoa-choices.html option1="Step-by-step" option2="Workflow" default="Step-by-step" text="Do you want to go step-by-step or using a workflow?" disambiguation="workflow"%}

# Processing data using DADA2

Once demultiplexed FASTQ files without non-biological nucleotides are in hand, the **DADA2** pipeline proceeds as follows:

1. Prepare reads
   1. Inspect read quality profiles
   2. Filter and trim
   3. Learn the Error Rates
   4. Infer sample composition
   5. Merge forward/reverse reads

3. Construct sequence table
   1. Make sequence table
   2. Remove chimeras
  
4. Assign taxonomy
5. Evaluate accuracy

<div class="Workflow" markdown="1">

{% snippet faqs/galaxy/workflows_run_wfh.md title="dada2/main" wfhub_id="790" version="2"%}

> <hands-on-title>Define DADA2 workflow parameters</hands-on-title>
>
> 1. Select following parameters:
>    - {% icon param-collection %} *"Paired short read data"*: `Raw Reads`
>    - *"Read length forward read"*: `240`
>    - *"Read length reverse read"*: `160`
>    - *"Pool samples"*: `process samples individually`
>    - *"Cached reference database"*: `Silva version 132`
>
> 2. Run the workflow
{: .hands_on}

The workflow generates several outputs that we will inspect them given the step they correspond to.

</div>

# Prepare reads

The first step is to check and improve the quality of data.

## Inspect read quality profiles

We start by visualizing the quality profiles of the forward reads:

<div class="Step-by-step" markdown="1">

> <hands-on-title>Inspect read quality profiles</hands-on-title>
>
> 1. {% tool [dada2: plotQualityProfile](toolshed.g2.bx.psu.edu/repos/iuc/dada2_plotqualityprofile/dada2_plotQualityProfile/1.34.0+galaxy0) %} with the following parameters:
>    - *"Processing mode"*: `Joint`
>        - *"Paired reads"*: `paired - in a data set pair`
>            - {% icon param-collection %} *"Paired short read data"*: `Sorted Raw Reads`
>        - *"Aggregate data"*: `No`
>    - *"sample number"*: `10000000`
{: .hands_on}
</div>

**dada2: plotQualityProfile** generates 2 files with quality profiles per sample: 1 PDF for forward reads and 1 PDF for reverse reads.

> <hands-on-title>Inspect read quality profiles</hands-on-title>
>
> 1. Inspect **plotQualityProfile** forward read output
{: .hands_on}

![Quality Profiles for the forward reads of all samples with read length on the X-axis and Quality score on the Y-axis](./images/plotQualityProfile_forward.png)

In gray-scale is a heatmap of the frequency of each quality score at each base position. The mean quality score at each position is shown by the green line, and the quartiles of the quality score distribution by the orange lines. The red line shows the scaled proportion of reads that extend to at least that position (this is more useful for other sequencing technologies, as Illumina reads are typically all the same length, hence the flat red line).

> <question-title></question-title>
>
> 1. Are the forward reads good quality?
> 2. We generally advise trimming the last few nucleotides to avoid less well-controlled errors that can arise there. What would you recommend as threshold?
> 3. Are the reverse reads good quality?
> 4. What would you recommend as threshold for trimming?
>
> > <solution-title></solution-title>
> >
> > 1. The forward reads are good quality
> > 2. These quality profiles do not suggest that any additional trimming is needed. We will truncate the forward reads at position 240 (trimming the last 10 nucleotides).
> > 3. The reverse reads are of significantly worse quality, especially at the end, which is common in Illumina sequencing. This isn’t too worrisome, as DADA2 incorporates quality information into its error model which makes the algorithm robust to lower quality sequence, but trimming as the average qualities crash will improve the algorithm’s sensitivity to rare sequence variants.
> > 
> >    ![Quality Profiles for the reverse reads of all samples with read length on the X-axis and Quality score on the Y-axis](./images/plotQualityProfile_reverse.png)
> >    
> > 4. Based on the profiles, we will truncate the reverse reads at position 160 where the quality distribution crashes.
> > 
> {: .solution}
>
{: .question}

> <comment-title>Considerations for your own data</comment-title>
>
> Your reads must still overlap after truncation in order to merge them later! The tutorial is using 2x250 V4 (expected size, 270 bp–387 bp) sequence data, so the forward and reverse reads almost completely overlap and our trimming can be completely guided by the quality scores. If you are using a less-overlapping primer set, like V1-V2 or V3-V4, your truncated length must be large enough to maintain $$ 20 + biological.length.variation $$ nucleotides of overlap between them.
>
{: .comment}

## Filter and trim

<div class="Step-by-step" markdown="1">

Let's now trim and filter the reads:

> <hands-on-title>Filter and trim</hands-on-title>
>
> 1. {% tool [dada2: filterAndTrim](toolshed.g2.bx.psu.edu/repos/iuc/dada2_filterandtrim/dada2_filterAndTrim/1.34.0+galaxy0) %} with the following parameters:
>    - *"Paired reads"*: `paired - in a data set pair`
>        - {% icon param-collection %} *"Paired short read data"*: `Sorted Raw Reads`
>    - In *"Trimming parameters"*:
>        - *"Truncate read length"*: `240`
>    - In *"Filtering parameters"*:
>        - *"Remove reads by number expected errors"*: `2`
>
>          This parameter sets the maximum number of "expected errors" allowed in a read, which is [a better filter than simply averaging quality scores](https://academic.oup.com/bioinformatics/article/31/21/3476/194979).
>
>    - *"Separate filters and trimmers for reverse reads"*: `Yes`
>        - In *"Trimming parameters"*:
>            - *"Truncate read length"*: `160`
>        - In *"Filtering parameters"*:
>            - *"Remove reads by number expected errors"*: `2`
>
{: .hands_on}

> <comment-title>Speeding up downstream computation</comment-title>
> 
>  The standard filtering parameters are starting points, not set in stone. If you want to speed up downstream computation, consider tightening the *"Remove reads by number expected errors"* value. If too few reads are passing the filter, consider relaxing the value, perhaps especially on the reverse reads (e.g. *"Remove reads by number expected errors"* to `2` for forward and `5` for reverse), and reducing the *"Truncate read length"* to remove low quality tails. Remember though, when choosing *"Truncate read length"* for paired-end reads you must maintain overlap after truncation in order to merge them later. 
> 
{: .comment}

> <comment-title>ITS data</comment-title>
> 
> For ITS sequencing, it is usually undesirable to truncate reads to a fixed length due to the large length variation at that locus. That is OK, just leave out  *"Truncate read length"*. See the [DADA2 ITS workflow](https://benjjneb.github.io/dada2/ITS_workflow.html) for more information 
> 
{: .comment}

</div>

**dada2: filterAndTrim** generates 2 outputs:
- 1 paired-collection named `Paired reads` with the trimmed and filtered reads
- 1 collection with statistics for each sample

   > <question-title></question-title>
   >
   > 1. How many reads were in F3D0 sample before Filter and Trim?
   > 2. How many reads are in F3D0 sample after Filter and Trim?
   >
   > > <solution-title></solution-title>
   > >
   > > In F3D0 element in the collection, we see:
   > > 1. `reads.in`: 7793
   > > 2. `reads.out`: 7113
   > >
   > {: .solution}
   >
   {: .question}

To better see the impact of filtering and trimming, we can inspect the read quality profiles and compare to the profiles before.

<div class="Step-by-step" markdown="1">

> <hands-on-title>Inspect read quality profiles after Filter And Trim</hands-on-title>
>
> 1. {% tool [dada2: plotQualityProfile](toolshed.g2.bx.psu.edu/repos/iuc/dada2_plotqualityprofile/dada2_plotQualityProfile/1.34.0+galaxy0) %} with the following parameters:
>    - *"Processing mode"*: `Joint`
>        - *"Paired reads"*: `paired - in a data set pair`
>            - {% icon param-collection %} *"Paired short read data"*: `Paired reads` output of **dada2: filterAndTrim**
>        - *"Aggregate data"*: `No`
>    - *"sample number"*: `10000000`
>
{: .hands_on}

</div>

> <question-title></question-title>
>
> Looking at outputs of **plotQualityProfile** for reverse reads before and after **dada2: filterAndTrim**
> 
> {% snippet faqs/galaxy/features_scratchbook.md %}
>
> ![Quality Profiles for forward reads of all samples with read length on the X-axis and Quality score on the Y-axis](./images/plotQualityProfile_reverse.png "Quality profiles for reverse reads before trimming and filtering")
>
> ![Quality Profiles for reverse reads of all samples with read length on the X-axis and Quality score on the Y-axis](./images/plotQualityProfile_reverse_after_filter_trim.png "Quality profiles for reverse reads after trimming and filtering")
>
> How did trimming and filtering change the quality profiles for reverse reads?
>
> > <solution-title></solution-title>
> >
> > The reads have been trimmed making the profiles looking better. 
> >
> {: .solution}
>
{: .question}

## Learn the Error Rates

The DADA2 algorithm makes use of a parametric error model (`err`) and every amplicon dataset has a different set of error rates. The **dada2: learnErrors** method learns this error model from the data, by alternating estimation of the error rates and inference of sample composition until they converge on a jointly consistent solution. As in many machine-learning problems, the algorithm must begin with an initial guess, for which the maximum possible error rates in this data are used (the error rates if only the most abundant sequence is correct and all the rest are errors).

<div class="Step-by-step" markdown="1">

**dada2: learnErrors** tool expects as input one collection and not a paired collection. We then need to split to paired collection into 2 collections: one with forward reads and one with reverse reads.

> <hands-on-title> Unzip paired collection </hands-on-title>
>
> 1. {% tool [Unzip collection](__UNZIP_COLLECTION__) %} with the following parameters:
>    - {% icon param-file %} *"Paired input to unzip"*: `Paired reads` output of **dada2: filterAndTrim**
>
> 2. Tag the `(forward)` collection with `#forward`
>
>    {% snippet faqs/galaxy/collections_add_tag.md tag="#forward" %}
>
> 3. Tag the `(reverse)` collection with `#reverse`
{: .hands_on}

Let's now run **dada2: learnErrors** on both collections.

> <hands-on-title> Learn the Error Rate </hands-on-title>
>
> 1. {% tool [dada2: learnErrors](toolshed.g2.bx.psu.edu/repos/iuc/dada2_learnerrors/dada2_learnErrors/1.34.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Short read data"*: `forward` output of **Unzip collection**
>   
> 3. {% tool [dada2: learnErrors](toolshed.g2.bx.psu.edu/repos/iuc/dada2_learnerrors/dada2_learnErrors/1.34.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Short read data"*: `reverse` output of **Unzip collection**
>   
{: .hands_on}

</div>

![Error rates for each possible transition as a function of the associated quality score, the final estimated error rates, the initial input rates, and the expected error rates under the nominal definition of quality scores](./images/learnErrors_forward.png "Error rates for each possible transition for the forward reads")

The error rates for each possible transition (A->C, A->G, ...) are shown as a function of the associated quality score, the final estimated error rates (if they exist), the initial input rates, and the expected error rates under the nominal definition of quality scores. Points are the observed error rates for each consensus quality score. The black line shows the estimated error rates after convergence of the machine-learning algorithm. The red line shows the error rates expected under the nominal definition of the Q-score.

> <question-title></question-title>
>
> 1. Are the estimated error rates (black line) in a good fit to the observed rates (points) for the forward reads?
> 2. What about the reverse reads?
>
>    ![Error rates for each possible transition as a function of the associated quality score, the final estimated error rates, the initial input rates, and the expected error rates under the nominal definition of quality scores](./images/learnErrors_reverse.png "Error rates for each possible transition for the reverse reads")
>
> > <solution-title></solution-title>
> >
> > 1. The estimated error rates (black line) are a good fit to the observed rates (points), and the error rates drop with increased quality as expected. Everything looks reasonable and we proceed with confidence.
> > 2. Same
> >
> {: .solution}
>
{: .question}
 

## Infer Sample Composition 

In this step, the core sample inference algorithm ({% cite Callahan_2016 %}) is applied to the filtered and trimmed sequence data.

<div class="Step-by-step" markdown="1">

> <hands-on-title> Sample Inference </hands-on-title>
>
> 1. {% tool [dada2: dada](toolshed.g2.bx.psu.edu/repos/iuc/dada2_dada/dada2_dada/1.34.0+galaxy0) %} with the following parameters:
>    - *"Process samples in batches"*: `no`
>
>      > <comment-title>Process samples in batches or not</comment-title>
>      >
>      > Choosing `yes` for *"Process samples in batches"* gives identical results as `no` if samples are not pooled.
>      >
>      > - `no`: a single Galaxy job is started where the samples are processed sequentially
>      > - `yes`: a separate Galaxy job is started for each sample (given compute resources, this can be much faster)
>      >
>      > If `yes`, the sorting of the collection is not needed.
>      >
>      {: .comment}
>
>        - {% icon param-file %} *"Reads"*: output of **Unzip collection** with `#forward` tag
>        - *"Pool samples"*: `process samples individually`
>
>           > <comment-title>Sample pooling</comment-title>
>           >
>           > By default, the **dada2: dada** function processes each sample independently. However, pooling information across samples can increase sensitivity to sequence variants that may be present at very low frequencies in multiple samples. Two types of pooling are possible:
>           >
>           > - `pool sample`: standard pooled processing, in which all samples are pooled together for sample inference
>           > - `pseudo pooling between individually processed samples`: pseudo-pooling, in which samples are processed independently after sharing information between samples, approximating pooled sample inference in linear time.
>           >
>           {: .comment}
>
>    - {% icon param-file %} *"Error rates"*: output of **dada2: learnErrors** with `#forward` tag
>
> 3. {% tool [dada2: dada](toolshed.g2.bx.psu.edu/repos/iuc/dada2_dada/dada2_dada/1.34.0+galaxy0) %} with the following parameters:
>    - *"Process samples in batches"*: `no`
>        - {% icon param-file %} *"Reads"*: output of **Unzip collection** with `#reverse` tag
>        - *"Pool samples"*: `process samples individually`
>    - {% icon param-file %} *"Error rates"*: output of **dada2: learnErrors** with `#reverse` tag
{: .hands_on}

</div>

We can look at the number of true sequence variants from unique sequences have been inferred by **DADA2**.

> <hands-on-title> Inspect number of true sequence variants from unique sequences </hands-on-title>
>
> 1. Click one output collection of **dada2: dada**
> 2. Click on {% icon details %} *Show Details*
> 2. Expand *Tool Standard Output*
{: .hands_on}

> <question-title></question-title>
>
> 1. How many unique sequences are in forward reads for sample F3D0?
> 2. How many true sequence variants from unique sequences have been inferred by DADA2 for this dataset?
>
> > <solution-title></solution-title>
> >
> > 1. 7113 reads in 1979 unique sequences
> > 2. 128 sequence variants were inferred from 1979 input unique sequences.
> >
> {: .solution}
>
{: .question}

## Merge paired reads

We now merge the forward and reverse reads together to obtain the full denoised sequences. Merging is performed by aligning the denoised forward reads with the reverse-complement of the corresponding denoised reverse reads, and then constructing the merged "contig" sequences. By default, merged sequences are only output if the forward and reverse reads overlap by at least 12 bases, and are identical to each other in the overlap region (but these conditions can be changed via tool parameters).

<div class="Step-by-step" markdown="1">

> <hands-on-title> Merge paired reads </hands-on-title>
>
> 1. {% tool [dada2: mergePairs](toolshed.g2.bx.psu.edu/repos/iuc/dada2_mergepairs/dada2_mergePairs/1.34.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Dada results for forward reads"*: output of **dada2: dada** with `#forward` tag
>    - {% icon param-file %} *"Forward reads"*: output of **Unzip collection** with `#forward` tag
>    - {% icon param-file %} *"Dada results for reverse reads"*:  output of **dada2: dada** with `#reverse` tag
>    - {% icon param-file %} *"Reverse reads"*: output of **Unzip collection** with `#reverse` tag
>    - *"Concatenated rather than merge"*: `No`
>
>      > <comment-title>Non-overlapping reads</comment-title>
>      > 
>      > Non-overlapping reads are supported, but not recommended. In this case, *"Concatenated rather than merge"* should be `Yes`.
>      > 
>      {: .comment}
>
>    - *"Output detailed table"*: `Yes`
{: .hands_on}

</div>

Let's inspect the merged data, more specifically the table in `details` collection

> <question-title></question-title>
>
> 1. How many reads have been merged?
> 2. What are the different columns in the detail table for F3D0?
>
> > <solution-title></solution-title>
> >
> > 1. The detail table for F3D0 has 108 lines including a header. So 107 reads (over 128) have been merged
> > 2. The table contains:
> >    1. merged sequence,
> >    2. its abundance,
> >    3. the indices of the forward sequence variants that were merged
> >    4. the indices of the reverse sequence variants that were merged
> >    5. number of match in the alignment between forward and reverse
> >    6. number of mismatch in the alignment between forward and reverse
> >    7. number of indels in the alignment between forward and reverse
> >    8. prefer
> >    9. status to accept
> >   
> >    Paired reads that did not exactly overlap were removed by mergePairs, further reducing spurious output.
> >
> {: .solution}
>
{: .question}

> <comment-title>Not enough reads merged</comment-title>
> 
> Most of the reads should successfully merge. If that is not the case upstream parameters may need to be revisited: Did you trim away the overlap between your reads? 
> 
{: .comment}

# Make sequence table

In this step, we construct an amplicon sequence variant table (ASV) table, a higher-resolution version of the OTU table produced by traditional methods.

<div class="Step-by-step" markdown="1">

> <hands-on-title> Make sequence table </hands-on-title>
>
> 1. {% tool [dada2: makeSequenceTable](toolshed.g2.bx.psu.edu/repos/iuc/dada2_makesequencetable/dada2_makeSequenceTable/1.34.0+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"samples"*: output of **dada2: mergePairs**
>    - *"Length filter method"*: `No filter`
>
{: .hands_on}

</div>

**dada2: makeSequenceTable** generates 2 outputs:
1. A sequence table: a matrix with rows corresponding to (and named by) the samples, and columns corresponding to (and named by) the sequence variants. 

   > <question-title></question-title>
   >
   > How many ASVs have been identified?
   >
   > > <solution-title></solution-title>
   > >
   > > The file contains 294 lines, including an header. So 293 ASVs have been identified.
   > >
   > {: .solution}
   >
   {: .question}

2. An image with sequence length distribution

   ![Histogram representing sequence length distribution](./images/sequence_length_distribution.png)

   > <question-title></question-title>
   >
   > 1. What is the range of merged sequences?
   > 2. Does the range make sense given the type of amplicon?
   >
   > > <solution-title></solution-title>
   > >
   > > 1. The lengths of merged sequences are between 251 and 255bp
   > > 2. The range falls within the expected range for this V4 amplicon.
   > >
   > {: .solution}
   >
   {: .question}

   > <comment-title>Sequences too long or too short</comment-title>
   > 
   > Sequences that are much longer or shorter than expected may be the result of non-specific priming. Non-target-length sequences can be removed directly from the sequence table. This is analogous to "cutting a band" in-silico to get amplicons of the targeted length.
   > 
   {: .comment}

## Remove chimeras

The core DADA method corrects substitution and indel errors, but chimeras remain. Fortunately, the accuracy of sequence variants after denoising makes identifying chimeric ASVs simpler than when dealing with fuzzy OTUs. Chimeric sequences are identified if they can be exactly reconstructed by combining a left-segment and a right-segment from two more abundant "parent" sequences.

<div class="Step-by-step" markdown="1">

> <hands-on-title> Remove chimeras </hands-on-title>
>
> 1. {% tool [dada2: removeBimeraDenovo](toolshed.g2.bx.psu.edu/repos/iuc/dada2_removebimeradenovo/dada2_removeBimeraDenovo/1.34.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"sequence table"*: output of **dada2: makeSequenceTable**
>
{: .hands_on}

</div>

> <question-title></question-title>
>
> How many non chimeric reads have been found?
>
> > <solution-title></solution-title>
> >
> > remaining nonchimeric:  96.40374 % (visible when expanding the dataset in history)
> >
> {: .solution}
>
{: .question}

The frequency of chimeric sequences varies substantially from dataset to dataset, and depends on on factors including experimental procedures and sample complexity. Here chimeras make up about 21% of the merged sequence variants, but when we account for the abundances of those variants we see they account for only about 4% of the merged sequence reads.

> <comment-title>Sequences too long or too short</comment-title>
> 
> Most of the **reads** should remain after chimera removal (it is not uncommon for a majority of **sequence variants** to be removed though). If most of the reads were removed as chimeric, upstream processing may need to be revisited. In almost all cases this is caused by primer sequences with ambiguous nucleotides that were not removed prior to beginning the DADA2 pipeline.
> 
{: .comment}

## Track reads through the pipeline

As a final check of our progress, we’ll look at the number of reads that made it through each step in the pipeline.

<div class="Step-by-step" markdown="1">

> <hands-on-title> Track reads through the pipeline </hands-on-title>
>
> 1. {% tool [dada2: sequence counts](toolshed.g2.bx.psu.edu/repos/iuc/dada2_seqcounts/dada2_seqCounts/1.34.0+galaxy0) %} with the following parameters:
>    - In *"data sets"*:
>        - {% icon param-repeat %} *"Insert data sets"*
>            - {% icon param-collection %} *"Dataset(s)"*: output of **dada2: filterAndTrim**
>            - *"name"*: `FiltTrim`
>        - {% icon param-repeat %} *"Insert data sets"*
>            - {% icon param-collection %} *"Dataset(s)"*: output of **dada2: dada** with `#forward` tag
>            - *"name"*: `Dada Forward`
>        - {% icon param-repeat %} *"Insert data sets"*
>            - {% icon param-collection %} *"Dataset(s)"*: output of **dada2: dada** with `#reverse` tag
>            - *"name"*: `Dada Reverse`
>        - {% icon param-repeat %} *"Insert data sets"*
>            - {% icon param-collection %} *"Dataset(s)"*: output of **dada2: mergePairs**
>            - *"name"*: `MakePairs`
>        - {% icon param-repeat %} *"Insert data sets"*
>            - {% icon param-file %} *"Dataset(s)"*: output of **dada2: makeSequenceTable**
>            - *"name"*: `SeqTab`
>        - {% icon param-repeat %} *"Insert data sets"*
>            - {% icon param-file %} *"Dataset(s)"*: output of **dada2: removeBimeraDenovo**
>            - *"name"*: `Bimera`
>
{: .hands_on}

</div>

> <question-title></question-title>
>
> 1. Have we kept the majority of the raw reads for F3D143?
> 2. Is there any exceptionally large drop associated with any single step for any sample?
>
> > <solution-title></solution-title>
> >
> > 1. Yes
> > 2. No
> >
> {: .solution}
>
{: .question}

> <comment-title>Sequences too long or too short</comment-title>
> 
> This is a great place to do a last sanity check. Outside of filtering, there should no step in which a majority of reads are lost. If a majority of reads failed to merge, you may need to revisit the truncLen parameter used in the filtering step and make sure that the truncated reads span your amplicon. If a majority of reads were removed as chimeric, you may need to revisit the removal of primers, as the ambiguous nucleotides in unremoved primers interfere with chimera identification.
> 
{: .comment}

# Assign taxonomy

It is common at this point, especially in 16S/18S/ITS amplicon sequencing, to assign taxonomy to the sequence variants. The DADA2 package provides a native implementation of the naive Bayesian classifier method ({% cite Wang_2007 %}) for this purpose. The **dada2: assignTaxonomy** function takes as input a set of sequences to be classified and a training set of reference sequences with known taxonomy, and outputs taxonomic assignments with at least `minBoot` bootstrap confidence.

DADA2 maintains [formatted training fastas for the RDP training set, GreenGenes clustered at 97% identity, and the Silva reference database](https://benjjneb.github.io/dada2/training.html), and additional trainings fastas suitable for protists and certain specific environments have been contributed. For fungal taxonomy, the General Fasta release files from the [UNITE ITS database](https://unite.ut.ee/repository.php) can be used as is. 
<div class="Step-by-step" markdown="1">

> <hands-on-title> Assign taxonomy </hands-on-title>
>
> 1. {% tool [dada2: assignTaxonomy and addSpecies](toolshed.g2.bx.psu.edu/repos/iuc/dada2_assigntaxonomyaddspecies/dada2_assignTaxonomyAddspecies/1.34.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"sequences to be assigned"*: output of **dada2: removeBimeraDenovo**
>    - *"Select a reference dataset your history or use a built-in?"*: `Use a built-in reference`
>        - *"Select reference data set"*: `Silva version 132`
>    - *"Add genus-species binomials to the taxonomic table"*: `Yes`
>        - *"Select a reference dataset your history or use a built-in?"*: `Use a built-in reference`
>           - *"Select reference data set"*: `Silva version 132`
>       - *"reporting options"*: `only unambiguous identifications`
>
>       > <comment-title>Species level assignments based on **exact matching**</comment-title>
>       > 
>       > The method of DADA2 make [species level assignments is based on **exact matching**](https://benjjneb.github.io/dada2/assign.html#species-assignment) between ASVs and sequenced reference strains. Recent analysis suggests that exact matching (or 100% identity) is the only appropriate way to assign species to 16S gene fragments ({% cite edgar2018updating %}). Currently, [species-assignment training fastas are available for the Silva and RDP 16S databases](https://benjjneb.github.io/dada2/training.html).
>       > 
>       {: .comment}
>
{: .hands_on}

</div>

**dada2: assignTaxonomy and addSpecies** generates a table with rows being the ASVs and columns the taxonomic levels. Let's inspect the taxonomic assignments.

> <question-title></question-title>
>
> 1. Which kingdom have been identified?
> 2. Which phyla have been identified? And how many ASVs per phylum?
> 3. How many ASVs have species assignments?
>
> > <solution-title></solution-title>
> >
> > 1. Only bacteria
> >    
> >    > <comment-title>Wrong kingdom</comment-title>
> >    > 
> >    > If your reads do not seem to be appropriately assigned, for example lots of your bacterial 16S sequences are being assigned as `Eukaryota`, your reads may be in the opposite orientation as the reference database. Tell DADA2 to try the reverse-complement orientation (*"Try reverse complement"*) with **dada2: assignTaxonomy and addSpecies** and see if this fixes the assignments.
> >    > 
> >    {: .comment}
> > 
> > 2. To get the identified phyla and the number of associated ASVs, we run {% tool [ Group data by a column](Grouping1) %} with the following parameters:
> >    - {% icon param-file %} *"Select data"*: output of **dada2: assignTaxonomy and addSpecies**
> >    - *"Group by column"*:`Column 3`
> >    - {% icon param-repeat %} *Insert Operation*
> >         - *"Type"*: `Count`
> >         - *"On column"*: `Column 3`
> >
> >     Phyla | Number of ASVs
> >     --- | ---
> >     Actinobacteria | 6
> >     Bacteroidetes | 20
> >     Cyanobacteria | 3
> >     Deinococcus-Thermus | 1
> >     Epsilonbacteraeota | 1
> >     Firmicutes | 185
> >     Patescibacteria | 2
> >     Phylum | 1
> >     Proteobacteria | 7
> >     Tenericutes | 6
> >     Verrucomicrobia | 1 
> >
> >     Unsurprisingly, the Bacteroidetes and Firmicutes are well represented among the most abundant taxa in these fecal samples.
> >
> > 3. 16/232 species assignments were made, both because it is often not possible to make unambiguous species assignments from subsegments of the 16S gene, and because there is surprisingly little coverage of the indigenous mouse gut microbiota in reference databases.
> >
> {: .solution}
{: .question}

# Exploration of ASVs with phyloseq

**phyloseq** ({% cite McMurdie_2013 %}) is a powerful framework for further analysis of microbiome data. It imports, stores, analyzes, and graphically displays complex phylogenetic sequencing data that has already been clustered into Operational Taxonomic Units (OTUs) or ASV, especially when there is associated sample data, phylogenetic tree, and/or taxonomic assignment of the OTUs/ASVs. 

## Prepare metadata table and launch phyloseq

To use phyloseq properly, the best is to provide it a metadata table. Here, we will also uses the small amount of metadata we have: the samples are named by the gender (G), mouse subject number (X) and the day post-weaning (Y) it was sampled (eg. GXDY).

We will construct a simple sample table from the information encoded in the filenames. Usually this step would instead involve reading the sample data in from a file. Here we will:

1. Extract element identifiers of a collection
2. Add information about gender, mouse subject number and the day post-weaning using the extracted element identifiers
4. Remove last line
5. Add extra column `When` to indicate if `Early` (days < 100) or `Late` (day > 100)
3. Add a header

> <hands-on-title>Prepare metadata table</hands-on-title>
> 1. {% tool [Extract element identifiers](toolshed.g2.bx.psu.edu/repos/iuc/collection_element_identifiers/collection_element_identifiers/0.0.2) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: output of **dada2: mergePairs**
>
>    We will now extract from the names the factors:
>
> 2. {% tool [Replace Text in entire line](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.5+galaxy2) %}
>      - {% icon param-file %} *"File to process"*: output of **Extract element identifiers** {% icon tool %}
>      - In *"Replacement"*:
>         - In *"1: Replacement"*
>            - *"Find pattern"*: `(.)(.)D(.*)`
>            - *"Replace with"*: `\1\2D\3\t\1\t\2\t\3`
>
>    The given pattern `..D.` will be applied to the identifiers, where `.` will match any single character and `.*` any number of any character. The additional parentheses will record the matched character such that it can be used in the replacement (e.g. `\1` will refer to the content that matched the pattern in the 1st pair of parentheses)
>     This step creates 3 additional columns with the gender, the mouse subject number and the day post-weaning
>
> 3. Change the datatype to `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 5. {% tool [Select first lines from a dataset (head)](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_head_tool/9.5+galaxy2) %} to remove the last line
>    - {% icon param-file %} *"File to select"*: output of **Replace Text**
>    - *"Operation"*: `Remove last lines`
>    - *"Number of lines"*: `1`
>
> 6. {% tool [Compute](toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1) %} on rows with the following parameters:
>    - {% icon param-file %} *"Input file"*: output of **Select first lines**
>    - *"Input has a header line with column names?"*: `No`
>    - In *"Expressions"*:
>      - *"Add expression"*: `bool(c4>100)`
>      - *"Mode of the operation?"*: `Append`
> 
> 7. {% tool [Replace Text in a specific column](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1) %} on rows with the following parameters:
>    - {% icon param-file %} *"File to process"*: output of **Compute**
>    - In *"Replacement"*:
>      - In *"1: Replacement"*
>        - *"in column"*: `Column: 5`
>        - *"Find pattern"*: `True`
>        - *"Replace with"*: `Late`
>      - {% icon param-repeat %} *"Insert Replacement"*
>      - In *"2: Replacement"*
>        - *"in column"*: `Column: 5`
>        - *"Find pattern"*: `False`
>        - *"Replace with"*: `Early`
>      - {% icon param-repeat %} *"Insert Replacement"*
>      - In *"3: Replacement"*
>        - *"in column"*: `Column: 2`
>        - *"Find pattern"*: `F`
>        - *"Replace with"*: `Female`
>      - {% icon param-repeat %} *"Insert Replacement"*
>      - In *"4: Replacement"*
>        - *"in column"*: `Column: 2`
>        - *"Find pattern"*: `M`
>        - *"Replace with"*: `Male`
>
> 4. Create a new file (`header`) from the following (header line of the metadata table)
>
>    ```text
>    Sample	Gender	Mouse	Day	When
>    ```
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md name="header" format="tabular" %}
>
> 5. {% tool [Concatenate datasets](cat1) %} to add this header line to the **Replace Text** output:
>    - {% icon param-file %} *"Concatenate Dataset"*: the `header` dataset
>    - *"Dataset"*
>       - {% icon param-repeat %} *"Insert Dataset"*
>         - {% icon param-file %} *"select"*: output of **Replace Text**
>
> 6. Rename the output to `Metadata table`
{: .hands_on}

We have now a metadata table with five columns:
1. Sample
2. Gender
3. Mouse subject
4. Day post-weaning 
5. When

We now construct a phyloseq object directly with it and the DADA2 outputs and launch Shiny-phyloseq ({% cite mcmurdie2015shiny %}) to explore the phyloseq object.

> <hands-on-title> Create phyloseq object and launch phyloseq exploration </hands-on-title>
>
> 1. {% tool [Create phyloseq object from dada2](toolshed.g2.bx.psu.edu/repos/iuc/phyloseq_from_dada2/phyloseq_from_dada2/1.50.0+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Sequence table"*: output of **dada2: removeBimeraDenovo**
>    - {% icon param-file %} *"Taxonomy table"*: output of **dada2: assignTaxonomy and addSpecies**
>    - {% icon param-file %} *"Sample table"*: `Metadata table`
>
> 2. {% tool [Phyloseq](interactive_tool_phyloseq) %} with the following parameters:
>    - {% icon param-file %} *"Phyloseq R object"*: output of **Create phyloseq object from dada2**
>
>    {% snippet faqs/galaxy/interactive_tools_open.md %}
>
{: .hands_on}

A new page will be launched with Shiny-phyloseq ready to explore the phyloseq object.

## Visualize alpha-diversity

A **diversity index** is a quantitative measure that is used to assess the level of diversity or variety within a particular system, such as a biological community, a population, or a workplace. It provides a way to capture and quantify the distribution of different types or categories within a system.

Related to ecology, the term **diversity** describes the number of different species present in one particular area and their relative abundance. More specifically, several different metrics of diversity can be calculated. The most common ones are α, β and γ diversity.

The **α diversity** describes the diversity within a community. It considers the number of different species in an environment (also referred to as species **richness**). Additionally, it can take the abundance of each species into account to measure how evenly individuals are distributed across the sample (also referred to as species **evenness**).

There are several different indices used to calculate α diversity because different indexes capture different aspects of diversity and have varying sensitivities to different factors. These indexes have been developed to address specific research questions, account for different ecological or population characteristics, or highlight certain aspects of diversity. 

![Explanation of the α diversity](./images/alphadiversity_metrics.png "<a href="https://medium.com/pjtorres-high-gut-alpha-diversity-and-health/high-alpha-diversity-and-health-65e5eca7fa36">Source</a>")

Here we will plot the **α diversity** for the samples given the Days using 2 measures:

- The **Shannons** index, which calculates the uncertainty in predicting the species identity of an individual that is selected from a community ({% cite Shannon.1948 %}).

- The **Simpsons** index, which calculates the probability that two individuals selected from a community will be of the same species. Obtains small values in datasets of high diversity and large values in datasets of low diversity ({% cite SIMPSON.1949 %}).

> <hands-on-title> Plot α diversity </hands-on-title>
>
> 1. Click on **Alpha Diversity** tab
> 2. Select the following parameters:
>    - *"X"*: `Day`
>    - *"Color"*: `When`
>    - *"alpha Measures"*: `Shannon`, `Simpson`
{: .hands_on}

![2 panels for 2 α diversity metrics (Shannon, Simpson) with α diversity value on Y axis and days on X axis. The points are colored given When](./images/alpha_diversity.png)

> <question-title></question-title>
>
> Is there any obvious systematic difference in alpha-diversity between early and late samples?
>
> > <solution-title></solution-title>
> >
> > No
> >
> {: .solution}
>
{: .question}

## Plot Ordination

Ordination refers to a set of multivariate techniques used to analyze complex ecological data. It involves summarizing large datasets, such as species abundance across different samples, by projecting them onto a low-dimensional space. This visualization makes it easier to identify patterns and relationships within the data, such as how species distributions correspond to environmental gradients. Ordination helps understand the underlying factors that structure ecological communities and the relative importance of different environmental variables.

Phyloseq offers several ordination methods:

1. **MDS (Multidimensional Scaling)**: MDS is a technique that visualizes the level of similarity or dissimilarity of individual cases of a dataset by positioning them in a low-dimensional space.

2. **PCoA (Principal Coordinates Analysis)**: PCoA is a distance-based ordination method that transforms a distance matrix into a set of coordinates, allowing the visualization of the relationships between samples in a reduced dimensional space.

3. **DCA (Detrended Correspondence Analysis)**: DCA is an ordination method that removes the arch effect in Correspondence Analysis, providing a more accurate representation of ecological gradients.

2. **CCA (Canonical Correspondence Analysis)**: CCA is a direct gradient analysis technique that relates species composition data to measured environmental variables, highlighting the influence of these variables on community structure.

3. **RDA (Redundancy Analysis)**: RDA is a linear ordination method that co


topic: microbiome
tutorial: beer-data-analysis
title: Identification of the micro-organisms in a beer using Nanopore sequencing
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["nanopore", "beer", "citizen science", "metagenomics", "microgalaxy"]
content_hash: e54c7e58446a450d1cd5a056aa630d89
last_modified: 2026-02-26T21:12:17.262598
zenodo_link: https://doi.org/10.5281/zenodo.7093173
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/beer-data-analysis/tutorial.html

content:
What is a microbiome? It is a collection of small living creatures.
These small creatures are called **micro-organisms** and they are **everywhere**. In our gut,
in the soil, on vending machines, and even inside the beer. Most of these micro-organisms are
actually very good for us, but some can make us very ill.

Micro-organisms come in different shapes and sizes, but they have the same components.
One crucial component is the **DNA**, the blueprint of life. The DNA encodes the
shape and size and many other characteristics unique to a species. Because DNA is so species-specific,
reading the DNA can be used to identify what kind of micro-organism
the it is from. Therefore, within a metagenomic specimen, *e.g.* a sample form soil, gut,
or beer, one can identify what kind of species are inside the sample.

In this tutorial, we will use data of beer microbiome generated via the
[BeerDEcoded project](https://streetscience.community/projects/beerdecoded/).

> <details-title>The BeerDEcoded project</details-title>
>
> The BeerDEcoded Project is a series of workshops organized with and for schools as well as the general
> audience, aiming to introduce biology and genomic science. People learn in an interactive way
> about DNA, sequencing technologies, bioinformatics, open science, how these technologies
> and concepts are applied and how they are impacting their daily life.
>
> Beer is alive and contains many microorganisms. It can be found in many places
> and there are many of them. It is a fun media to bring the people to the
> contact of molecular biology, data-analysis, and open science.
>
> A BeerDEcoded workshop includes the following steps:
> 1. Extract yeasts and their DNA from beer bottle,
> 2. Sequence the extracted DNA using a MinION sequencer to obtain the sequence of bases/nucleotides (A, T, C and G) for each DNA fragment in the sample,
> 3. Analyze the sequenced data in order to know which organisms this DNA is from
>
> ![The image represents a BeerDEcoded workshop. On the left, there is a beer glass. An arrow goes from the bottle to DNA with "Extraction" written on the below. An arrow goes from DNA to DNA sequences with "Sequencing" written on the below. An arrow goes from the DNA sequences to Yeasts with "Data analysis" written on the below](./images/beerprocess.png)
>
{: .details}

> <comment-title>Beer microbiome</comment-title>
>
> Beer is alive! It contains microorganisms, in particular **yeasts**.
>
> Indeed, grain and water create a sugary liquid (called wort). The beer brewer
> adds yeasts to it. By eating the sugar, yeast creates alcohol,
> and other compounds (esters, phenols, etc.) that give beer its particular
> flavor.
>
> Yeasts are microorganisms, more precisely **unicellular fungi**. The majority
> of beers use a yeast genus called ***Saccharomyces***, which in Greek means
> "sugar fungus". Within that genus, two specific species of *Saccharomyces*
> are the most commonly used:
>
> - ***Saccharomyces cerevisiae***: a top-fermenting (*i.e.* yeast which rise
> up to the top of the beer as it metabolizes sugars, delivering alcohol as a by-product),
> ale yeast responsible for a huge range of beer styles like witbiers, stouts, ambers,
> tripels, saisons, IPAs, and many more. It is most likely the yeast that the early brewers were
> inadvertently brewing with over 3,000 years ago.
> - ***Saccharomyces pastorianus***: a bottom-fermenting (*i.e.* it sits on the
> bottom of the tank as it ferments) lager yeast, responsible for beer styles
> like Pilsners, lagers, märzens, bocks, and more. This yeast was originally
> found, and cultivated, by Bavarian brewers a little over 200 years ago. It is
> the most commonly used yeast in terms of the raw amount of beer produced around the world.
>
> Since yeast is all around us, we can actually brew spontaneously
> fermented beer by using wild yeasts and souring microbiota floating through the air.
>
{: .comment}

During one BeerDEcoded workshop, we extracted yeasts out of a bottle of
[Chimay](https://en.wikipedia.org/wiki/Chimay_Brewery). We then
extracted the DNA of these yeasts and sequenced it using a MinION to obtain the DNA
sequences. Now, we would like to **identify the yeast species** sequenced there, and
thereby **outline the diversity of microorganisms** (the microbiome community) in the beer
sample.

To get this information, we need to process the sequenced data in a few steps:
1. Check the quality of the data
2. Assign a taxonomic label, *i.e.* assign 'species' to each sequence
3. Visualize the distribution of the different species

This type of data analysis requires running several bioinformatics tools and
usually requires a computer science background. [Galaxy](https://galaxyproject.org/) is
an open-source platform for data analysis that enables anyone to use bioinformatics
tools through its graphical web interface, accessible via any Web browser.

So, in this tutorial, we will use Galaxy to extract and visualize the community
of yeasts from a bottle of beer.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Prepare Galaxy and data

First of all, this tutorial will get you hands on with some basic Galaxy tasks, including creating a history and importing data.

## Get familiar with Galaxy

> <hands-on-title>Open Galaxy</hands-on-title>
>
> 1. Open your favorite browser (Works on Chrome, Firefox, Safari but not Internet Explorer!)
> 2. Create a Galaxy account if you do not have one
>
>    {% snippet faqs/galaxy/account_create.md %}
>
{: .hands_on}

The Galaxy homepage is divided into three panels:
* Tools on the left
* Viewing panel in the middle
* History of analysis and files on the right

![Galaxy interface screenshot showing history panel on the right, tools panel on the left, and main panel at the center]({% link topics/introduction/images/galaxy_interface.png %} "The Galaxy interface")

The first time you use Galaxy, there will be no files in your history panel.

Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Prepare the Galaxy history</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

## Get data

Before we can begin any Galaxy analysis, we need to upload the input data: FASTQ files

> <hands-on-title>Upload your dataset</hands-on-title>
>
> 1. Import the sequenced data including fastq in the name
>
>    - Option 1 [{% icon video %}](https://youtu.be/FFCDx1rMGAQ): Your own local data using **Upload Data** (recommended for 1-10 datasets).
>
>      {% snippet faqs/galaxy/datasets_upload.md %}
>
>    - Option 2: From Zenodo, an external server, via URL
>
>      ```text
>      https://zenodo.org/record/7093173/files/ABJ044_c38189e89895cdde6770a18635db438c8a00641b.fastq
>      ```
>
>      {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    Your uploaded file is now in your current history. When the file is fully uploaded to Galaxy, it will turn green. But, what is this file?
>
> 2. Click on the {% icon galaxy-eye %} (eye) icon next to the dataset name to look at the file contents.
>
{: .hands_on}

The contents of the file will be displayed in the central Galaxy panel.

This file contains the sequences, also called **reads**, of DNA, *i.e.* succession of nucleotides, for all fragments from the yeasts in the beer, in FASTQ format.

{% snippet topics/sequence-analysis/faqs/fastq.md %}

# Data quality

## Assess data quality

Before starting to work on our data, it is necessary to assess its quality. This is an essential step if we aim to obtain a **meaningful downstream analysis**.

**FastQC** is one of the most widely used tools to **check the quality** of data generated by High Throughput Sequencing (HTS) technologies.

> <hands-on-title>Quality check</hands-on-title>
>
> 1. {% tool [FASTQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0) %} with the following parameters
>    - {% icon param-file %} *"Raw read data from your current history"*: `Reads`
>
> 2. Inspect the generated HTML file
>
{: .hands_on}

> <question-title></question-title>
>
> Given the Basic Statistics table on the top of the page:
>
> ![Screenshot of the FastQC Basic Statistics with filename, file type (Conventional base calls), encoding (Sanger / Illumina 1.9), total Sequences (1876), sequences flagged as poor quality (0), sequence length (130-2327) and %GC (29)](./images/fastqc_top_table.png)
>
> 1. How many sequences are in the FASTQ file?
> 2. How long are the sequences?
>
> > <solution-title></solution-title>
> >
> > 1. There are 1876 sequences.
> > 2. The sequences range from 130 nucleotides to 2327 nucleotides. Not all sequences have then the same length.
> >
> {: .solution}
>
{: .question}

**FastQC** provides information on various parameters, such as the range of quality values across all bases at each position:

![FastQC Per base sequence quality with scores below 20](./images/fastqc_1.png "Per base sequence quality. X-axis: position in the reads (in base pair). Y-axis: quality score, between 0 and 40 - the higher the score, the better the base call. For each position, a boxplot is drawn with: the median value, represented by the central red line;the inter-quartile range (25-75%), represented by the yellow box; the 10% and 90% values in the upper and lower whiskers; and the mean quality, represented by the blue line. The background of the graph divides the y-axis into very good quality scores (green), scores of reasonable quality (orange), and reads of poor quality (red).")

We can see that the quality of our sequencing data grows after the first few bases, stays around a score of 18 and then decreases again at the end of the sequences. MinION and Oxford Nanopore Technologies (ONT) are known to have a higher error rate compared to other sequencing techniques and platforms ({% cite delahaye2021sequencing %}).

For more detailed information about the other plots in the FASTQC report, check out our [dedicated tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).

## Improve the dataset quality

In order to improve the quality of our data, we will use two tools:
- **porechop** ({% cite rrwick2017 %}) to remove adapters that were added for sequencing and chimera (contaminant)
- **fastp** ({% cite Chen_2018 %}) to filter sequences with low quality scores (below 10)

> <hands-on-title>Improve the dataset quality</hands-on-title>
>
> 1. {% tool [Porechop](toolshed.g2.bx.psu.edu/repos/iuc/porechop/porechop/0.2.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input FASTA/FASTQ"*:  `Reads`
>    - *"Output format for the reads"*: `fastq`
>
> 2. {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.23.2+galaxy0) %} with the following parameters:
>    - *"Single-end or paired reads"*: `Single-end`
>      - {% icon param-file %} *"Input 1"*: output of **Porechop**
>      - In *"Adapter Trimming Options"*:
>        - *"Disable adapter trimming"*: `Yes`
>    - In *"Filter Options"*:
>        - In *"Quality filtering options"*:
>            - *"Qualified quality phred"*: `10`
>    - In *"Read Modification Options"*:
>        - *"PolyG tail trimming"*: `Disable polyG tail trimming`
>
> 3. Inspect the HTML report of **fastp** to see how the quality has been improved
{: .hands_on}

> <question-title></question-title>
>
> 1. How many sequences are there before filtering? Is it the same number as in FASTQC report?
> 2. How many sequences are there after filtering? How many sequences have then been removed by filtering?
> 3. What is the mean length before filtering? And after filtering?
>
> > <solution-title></solution-title>
> >
> > 1. There are 1,869 reads before filtering. The number is lower than in the FASTQC report. Some reads may have been discarded via Porechop
> > 2. There are 1,350 reads after filtering. So the filtering step has removed $$1869-1350 = 519$$ sequences.
> > 3. The mean length is 314 nucleotide before filtering and 316bp after filtering.
> {: .solution}
>
{: .question}

# Assign taxonomic classification

One of the main aims in microbiome data analysis is to identify the organisms sequenced. For that we try to **identify the taxon** to which each individual read belong.

{% snippet topics/microbiome/faqs/taxon.md %}

> <question-title></question-title>
>
> 1. Which microorganisms do we expect to identify in our data?
> 2. What is the taxonomy of the main expected microorganism?
>
> > <solution-title></solution-title>
> >
> > 1. The sequences are supposed to be yeasts extracted from a bottle of beer. The majority of beers contain a yeast genus called ***Saccharomyces*** and 2 species in that genus: *Saccharomyces cerevisiae* (ale yeast) and *Saccharomyces pastorianus* (lager yeast). The used beer is an ale beer, so we expect to find ***Saccharomyces cerevisiae***. But other yeasts can also have been used and then found. We could also have some DNA left from other beer components, but also contaminations by other microorganisms and even human DNA from people who manipulated the beer or did the extraction.
> >
> >
> > 2. The main expected microorganism is ***Saccharomyces cerevisiae*** with its taxonomy:
> >
> >    Level | Classification
> >    --- | ---
> >    Domain | Eukaryota
> >    Kingdom | Fungi
> >    Phylum | Ascomycota
> >    Class | Saccharomycetes
> >    Order | Saccharomycetales
> >    Family | Saccharomycetaceae
> >    Genus | *Saccharomyces*
> >    Species | *S. cerevisiae*
> >
> {: .solution}
>
{: .question}

Taxonomic assignment or classification is the process of assigning an **Operational Taxonomic Unit** (OTUs, that is, groups of related individuals / taxon) to sequences. To assign an OTU to a sequence it is compared against a database, but this comparison can be done in different ways, with different bioinformatics tools. Here we will use **Kraken2** ({% cite wood2019improved %}).

{% snippet topics/microbiome/faqs/kraken.md %}

> <hands-on-title>Kraken2</hands-on-title>
>
> 1. {% tool [Kraken2](toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.1.1+galaxy1) %} with the following parameters:
>    - *"Single or paired reads"*: `Single`
>        - {% icon param-file %} *"Input sequences"*: Output of **fastp**
>    - *"Print scientific names instead of just taxids"*: `Yes`
>    - In *"Create Report"*:
>        - *"Print a report with aggregrate counts/clade to file"*: `Yes`
>    - *"Select a Kraken2 database"*: `Prebuilt Refseq indexes: PlusPF`
>
>      The database here contains reference sequences and taxonomies. We need to
>      be sure it contains yeasts, i.e. fungi.
>
> 2. Inspect the report file
{: .hands_on}

The Kraken report is a tabular files with one line per taxon and 6 columns or fields:

1. Percentage of fragments covered by the clade rooted at this taxon
2. Number of fragments covered by the clade rooted at this taxon
3. Number of fragments assigned directly to this taxon
4. A rank code, indicating
    - (U)nclassified
    - (R)oot
    - (D)omain
    - (K)ingdom
    - (P)hylum
    - (C)lass
    - (O)rder
    - (F)amily
    - (G)enus, or
    - (S)pecies

    Taxa that are not at any of these 10 ranks have a rank code that is formed by using the rank code of the closest ancestor rank with a number indicating the distance from that rank. E.g., `G2` is a rank code indicating a taxon is between genus and species and the grandparent taxon is at the genus rank.

5. NCBI taxonomic ID number
6. Indented scientific name


```
Column 1	Column 2	Column 3	Column 4	Column 5	Column 6
38.00 	513 	513 	U 	0 	unclassified
62.00 	837 	1 	R 	1 	root
61.93 	836 	27 	R1 	131567 	cellular organisms
56.00 	756 	3 	D 	2759 	Eukaryota
55.33 	747 	3 	D1 	33154 	Opisthokonta
29.78 	402 	0 	K 	33208 	Metazoa
29.78 	402 	0 	K1 	6072 	Eumetazoa
29.78 	402 	0 	K2 	33213 	Bilateria
29.78 	402 	0 	K3 	33511 	Deuterostomia
29.78 	402 	0 	P 	7711 	Chordata
29.78 	402 	0 	P1 	89593 	Craniata
29.78 	402 	0 	P2 	7742 	Vertebrata
29.78 	402 	0 	P3 	7776 	Gnathostomata
29.78 	402 	0 	P4 	117570 	Teleostomi
29.78 	402 	0 	P5 	117571 	Euteleostomi
29.78 	402 	0 	P6 	8287 	Sarcopterygii
29.78 	402 	0 	P7 	1338369 	Dipnotetrapodomorpha
29.78 	402 	0 	P8 	32523 	Tetrapoda
29.78 	402 	0 	P9 	32524 	Amniota
29.78 	402 	0 	C 	40674 	Mammalia 
```

> <question-title></question-title>
>
> 1. How many taxons have been identified?
> 2. How much reads have been classified?
> 3. Which domains were found and with how many reads?
> 4. How much reads have been assigned by fungi Kingdom?
>
> > <solution-title></solution-title>
> >
> > 1. The file contains 300 lines (*information visible when expanding the report dataset in the history panel*). So 300-2 = 298 taxons have been identified.
> >
> > 2. On the 1350 sequences in the input, 837 (62%) were classified (or identified as a taxon) and 513 unclassified (38%). *Information visible when expanding the report dataset in the history panel, and scrolling in the small box starting with "Loading database information" below the format information, but also on the top of the report.*
> >
> > 3. The domains are identified by a `D` in colum 4. 
> >
> >    To get the domains (or other taxonomic level), we can use {% tool [Filter](Filter1) %} with the following parameters:
> >    - {% icon param-file %} *"Filter"*: report outpout of **Kraken2**
> >    - *"With following condition"*: `c4=='D'` 
> >
> >    The 3 domains were found:
> >    - Eukaryota with 756 (56%) reads assigned to it
> >    - Bacteria with 51 (3.78%) reads
> >    - Archaea with 2 (0.15%) reads
> >
> > 4. 342 (25.33%) reads are assigned to fungi.
> {: .solution}
>
{: .question}

Other taxons than yeast have been identified. They could be contamination or misidentification of reads. Indeed, many taxons have less than 5 reads assigned. We will filter these reads out to get a better view of the possible contaminations.

> <hands-on-title>Filter taxons with low assignements</hands-on-title>
>
> 1. {% tool [Filter](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: report outpout of **Kraken2**
>    - *"With following condition"*: `c2>5`
>
>      We want to keep only taxons with more than 5 reads assigned, *i.e.* the value in the 2nd column, is higher than 5.
>
> 2. Inspect the output
{: .hands_on}

> <question-title></question-title>
>
> 1. How many taxons have been removed? How many were kept?
> 2. What are the possible contaminations?
>
> > <solution-title></solution-title>
> > 1. 59 lines are now in the file so $$300 - 59 = 241$$ taxons have been removed because low assignment rates.
> >
> > 2. Most of the reads (402) were assigned to humans (*Homo sapiens*). This is likely a contamination either during the beer production or more likely during DNA extraction.
> >
> >    Bacteria were also found: Firmicutes, Proteobacteria and Bacteroidetes. But the identified taxons are not really precise (not below order level). So difficult to identify the possible source of contamination.
> {: .solution}
{: .question}

# Visualize the community

Once we have assigned the corresponding taxa to the sequences, the next step is to properly visualize the data: visualize the diversity of taxons at different levels.

To do that, we will use the tool **Krona** ({% cite Ondov_2011 %}). But before that, we need to adjust the output from Kraken2 to the requirements of **Krona**. Indeed, **Krona** expects as input a table with the first column containing a count and the remaining columns describing the hierarchy. Currently, we have a report tabular file with the first column containing the taxonomy and the second column the number of reads. We will now use another tool, which also provides taxonomic classification, but it produces the exact formatting Krona needs.

> <hands-on-title>Prepare dataset for Krona</hands-on-title>
>
> 1. {% tool [Krakentools: Convert kraken report file](toolshed.g2.bx.psu.edu/repos/iuc/krakentools_kreport2krona/krakentools_kreport2krona/1.2+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Kraken report file"*: **Report** output of **Kraken**
>
> 2. Inspect the output file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > Column 1	Column 2	Column 3	Column 4	Column 5	Column 6	Column 7	Column 8
>    > 513 	Unclassified 						
>    > 7 	k__Eukaryota 						
>    > 0 	k__Eukaryota 	p__Chordata 					
>    > 0 	k__Eukaryota 	p__Chordata 	c__Mammalia 				
>    > 0 	k__Eukaryota 	p__Chordata 	c__Mammalia 	o__Primates 			
>    > 0 	k__Eukaryota 	p__Chordata 	c__Mammalia 	o__Primates 	f__Hominidae 		
>    > 0 	k__Eukaryota 	p__Chordata 	c__Mammalia 	o__Primates 	f__Hominidae 	g__Homo 	
>    > 402 	k__Eukaryota 	p__Chordata 	c__Mammalia 	o__Primates 	f__Hominidae 	g__Homo 	s__Homo_sapiens					
>    > ```
>    >
>    > 1. What are the columns in the file?
>    >
>    > > <solution-title></solution-title>
>    > > 1. 8 columns: one with the number of reads and one for each of the 7 levels of taxonomy.
>    > {: .solution}
>    {: .question}
{: .hands_on}

We can now run **Krona**. This tool creates an interactive report that allows hierarchical data (like taxonomy) to be explored with zooming, as multi-layered pie charts. With this tool, we can easily visualize the composition of a microbiome community.

> <hands-on-title>Krona pie chart</hands-on-title>
>
> 1. {% tool [Krona pie chart](toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1+galaxy0) %} with the following parameters:
>    - *"What is the type of your input data"*: `Tabular`
>        - {% icon param-file %} *"Input file"*:  output of **Krakentools** {% icon tool %}
>
> 2. Inspect the generated file
{: .hands_on}

Let's take a look at the result.

<iframe id="krona" src="krona.html" frameBorder="0" width="100%" height="900px"> ![Krona chart with multi-layered pie chart representing the community profile with in the center the higher taxonomy levels (i.e. domain) and on the exterior the more detailed ones (i.e. species)](./images/krona.png) </iframe>

> <question-title></question-title>
>
> 1. What is the percentage of reads assigned to *Homo sapiens*? 
> 2. To Archaea?
>
> > <solution-title></solution-title>
> >
> > 1. 30% of reads are assigned to *Homo sapiens*
> > 2. 0.08% of Archaea
> {: .solution}
>
{: .question}

# Investigate the beer microbiome

Let's come back to our original question: characterization of the beer microbiome, specially looking at the yeasts.

Yeasts do not form a single taxonomic group ({% cite kurtzman1994molecular %}). They are parts of the fungi kingdom but belong two separate phyla: the Ascomycota and the Basidiomycota. [But the "true yeasts" are classified in the order Saccharomycetales](https://web.archive.org/web/20090226151906/http://www.yeastgenome.org/VL-what_are_yeast.html).

> <question-title></question-title>
> 1. Click on `o__Saccharomycetales` in the graph (Krona pie chart). Which yeast species have been identified? Are they the expected in beer?
> 2. Click on *Saccharomyces* in the graph. What are the percentages of identified reads assigned to *Saccharomyces* for different levels?
> 3. Click a second time on *Saccharomyces* in the graph. What is the repartition between the different *Saccharomyces* species?
>
> > <solution-title></solution-title>
> >
> > 1. 6 species from the Saccharomycetales order have been identified:
> >    - Saccharomycetaceae family
> >      - *Saccharomyces* genus
> >        - *Saccharomyces cerevisiae* species, the most abundant identified yeast species with 293 reads and the one expected given the type of beers
> >        - *Saccharomyces paradoxus* species, a wild yeast and the closest known species to *Saccharomyces cerevisiae*
> >
> >          These reads might have been misidentified to *Saccharomyces paradoxus* instead of *Saccharomyces cerevisiae* because of some errors in the sequences, as *Saccharomyces cerevisiae* and *Saccharomyces paradoxus* are close species and should share then a lot of similarity in their sequences.
> >
> >        - *Saccharomyces eubayanus* species, most likely the parent of the lager brewing yeast, *Saccharomyces pastorianus* ({% cite sampaio2018microbe %})
> >
> >          Similar to *Saccharomyces paradoxus*, these reads might have been misassigned.
> >
> >      - *Kluyveromyces* genus - *Kluyveromyces marxianus* species: only 1 read
> >
> >    - Trichomonascaceae family - *Sugiyamaella* genus - *Sugiyamaella lignohabitans* species: only 1 read
> >    - Debaryomycetaceae family - *Candida* genus - *Candida dubliniensis* species: only 1 read
> >
> >    Everything except *Saccharomyces cerevisiae* are probably misindentified reads.
> >
> > 2. Reads are assigned to *Saccharomyces*
> >    - 25% out of total reads (root)
> >    - 44% out of identified reads for Eukaryota domain
> >    - 96% out of identified reads for Ascomycota phylum
> >    - 98% out of identified reads for Saccharomycetales order
> >    - 99% out of identified reads for Saccharomycetaceae family
> >
> >    ![Krona chart with multi-layered pie chart representing the community profile with in the center the higher taxonomy levels (i.e. domain) and on the exterior the more detailed ones (i.e. species). *Saccharomyces* is highligthed with on the right the percentages of identified reads assigned to *Saccharomyces* for the different taxonomic levels](./images/krona_saccharomyces_1.png)
> >
> > 3. 92% of *Saccharomyces* reads are assigned to *Saccharomyces cerevisiae*, 5% to *Saccharomyces paradoxus* and 3% to *Saccharomyces eubayanus*,.
> >
> >    ![Krona chart with the community profile for *Saccharomyces* with the different species](./images/krona_saccharomyces_2.png)
> {: .solution}
>
{: .question}

Microbiome of several beers, including Chimay beers, have been previously investigated by targeting specifically the fungi, in which we can find yeasts ({% cite sobel2017beerdecoded %}):

![Heatmap with rows being type of beers and columns fungi species. For the rows, the name of beers are on the right and clustering tree on the left side of the heatmap. For the columns, the name of the species are on the bottom and the clustering tree on the top. The fungi species are sorted from the least abundant on the left to the most abundant on the right.](./images/beerdecoded.gif "Heatmap of the number of reads per internal transcribed spacer (ITS) sequencing of fungal species per beer. Source {% cite sobel2017beerdecoded %}")

The species identified for Chimay beers are (from the most abundant to the least one):
- *Saccharomyces cerevisiae*
- *Saccharomyces mikatea*: a species generally used in winemaking ({% cite bellon2013introducing %})
- *Kazachstania martiniae*: *Kazachstania* is a genus from the family Saccharomycetaceaethe.
- *Saccharomyces kudriavzevii*
- *Brettanomyces bruxellensis*

    *Brettanomyces* is a non-spore forming genus of yeast in the family Saccharomycetaceae, and is important to both the brewing and wine industries due to the sensory compounds it produces. 

    *Brettanomyces bruxellensis* is typically used for the production of the Belgian beers.

- *Saccharomyces paradoxus*: a wild yeast species closely related to *Saccharomyces cerevisiae*
- *Kazachstania kunashirensis*
- *Saccharomyces cariocanus*: a wild yeast species closely related to *Saccharomyces cerevisiae*
- *Filobasidium magnum*
- *Malasseria restricta*
- *Pichia kudriavzevii*
- *Aureobasidium pullulans*
- *Sporidiobolus metaroseus*

In a structured way:

Phylum | Class | Order | Family | Genus | Species
--- | --- | --- | --- | --- | ---
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces cerevisiae*
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces mikatea*
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces kudriavzevii*
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces paradoxus*
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces cariocanus*
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Kazachstania | *Kazachstania martiniae*
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Kazachstania | *Kazachstania kunashirensis*
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Brettanomyces | *Brettanomyces bruxellensis*
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae |  | *Pichia kudriavzevii*
Ascomycota | Dothideomycetes | Dothideales | Dothioraceae | Aureobasidium | *Aureobasidium pullulans*
Basidiomycota | Tremellomycetes | Filobasidiales | Filobasidiaceae | Filobasidium | *Filobasidium magnum*
Basidiomycota | Malasseziomycetes | Malasseziales | Malasseziaceae | Malassezia | *Malasseria restricta*
Basidiomycota | Sporidiobolales | Sporidiobolales | Sporidiobolaceae | Sporidiobolus | *Sporidiobolus metaroseus*

> <question-title></question-title>
>
> By looking at the output of **Krakentools**, which fungi species identified for the Chimay beers in {% cite sobel2017beerdecoded %} are also identified in our data? And vice versa?
>
> > <solution-title></solution-title>
> >
> > Phylum | Class | Order | Family | Genus | Species | {% cite sobel2017beerdecoded %} | Our data 
> > --- | --- | --- | --- | --- | --- | --- | ---
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces cerevisiae* | X | X
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces mikatea* | X | 
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces kudriavzevii* | X | 
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces paradoxus* | X | X
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces cariocanus* | X | 
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces eubayanus* | | X
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Kazachstania | *Kazachstania martiniae* | X | 
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Kazachstania | *Kazachstania kunashirensis* | X | 
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Kluyveromyces | *Kluyveromyces marxianus* | | X
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Brettanomyces | *Brettanomyces bruxellensis* | X | 
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae |  | *Pichia kudriavzevii* | X | 
> > Ascomycota | Saccharomycetes | Saccharomycetales | Debaryomycetaceae  | Candida | *Candida dubliniensis* | | X 
> > Ascomycota | Dothideomycetes | Dothideales | Dothioraceae | Aureobasidium | *Aureobasidium pullulans* | X | 
> > Ascomycota | Sordariomycetes | Sordariales | Sordariaceae | Neurospora | *Neurospora crassa* | | X 
> > Basidiomycota | Tremellomycetes | Filobasidiales | Filobasidiaceae | Filobasidium | *Filobasidium magnum* | X | 
> > Basidiomycota | Malasseziomycetes | Malasseziales | Malasseziaceae | Malassezia | *Malasseria restricta* | X | 
> > Basidiomycota | Sporidiobolales | Sporidiobolales | Sporidiobolaceae | Sporidiobolus | *Sporidiobolus metaroseus* | X | 
> {: .solution}
{: .question}

Some interesting yeast have been found in  {% cite sobel2017beerdecoded %} and not in our data (e.g. *Brettanomyces bruxellensis*), and vice versa.

# (Optional) Sharing your history

One of the most important features of Galaxy comes at the end of an analysis: sharing your histories with others so they can review them.

{% snippet faqs/galaxy/histories_sharing.md %}

# Conclusion


topic: microbiome
tutorial: lotus2-identifying-fungi
title: Identifying Mycorrhizal Fungi from ITS2 sequencing using LotuS2
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["fungi", "ecology"]
content_hash: daba4cb90030d92e65a6735f17ab4cc0
last_modified: 2026-02-26T21:12:17.291984
zenodo_link: https://zenodo.org/records/13710591
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/lotus2-identifying-fungi/tutorial.html

content:
tutorial-example-run
    date: 2024-09-20
---

# Introduction

This tutorial is for you if you are a mycologist using metabarcoding data to understand the fungal composition of soil samples. In particular, this tutorial and its downstream applications will be of most interest to mycorrhizal scientists. [{SPUN}](https://www.spun.earth) is a research and advocacy organisation that uses Illumina sequencing of environmental {eDNA} from soil samples to identify mycorrhizal fungi. At {SPUN}, we use both {ITS2} and {SSU} amplicon sequencing to identify ectomycorrhizal and arbuscular mycorrhizal fungi, respectively. This tutorial focuses on the {ITS2} sequencing used to identify {MF}.

We do this by using a specific primer pair that amplifies just the {ITS2} region of the ribosomal DNA present in a soil sample. This tutorial covers data for an {ITS2} amplicon produced using the ITS3F and ITS4R primer set and sequenced using Illumina sequencing, which generates paired-end data. In this case, the example data has been generated using a NextSeq 2000, using 2x300bp chemistry.

This tutorial concentrates on the bioinformatics part of the process, i.e. the steps we need to do **after** we get data back from the sequencer. Starting from some example FASTQ files, we demonstrate how to upload data, run LotuS2, and examine the output files. We end with a brief description of what we can do with these output files.

We use LotuS2 at {SPUN} because we have found that it performs best out of all the tools we tried for identifying {MF} from sequencing data. The LotuS2 publication also includes a benchmark where it outperformed most amplicon tools using default parameters (see {% cite ozkurt2022 %}).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Pre-requisites for this tutorial

If you have never used Galaxy before, we highly recommend doing their [interactive tour](https://usegalaxy.eu/tours/core.galaxy_ui) first (takes a few minutes). Or, if you have more time (1h 30 min recommended), you can do the [Galaxy Basics for Everyone]({% link topics/introduction/tutorials/galaxy-intro-101-everyone/tutorial.md %}) tutorial.

We recommend signing up for a user account at [https://usegalaxy.eu](https://usegalaxy.eu). Although other UseGalaxy servers are available (see the [first part of the Galaxy Basics tutorial]({% link topics/introduction/tutorials/galaxy-intro-101-everyone/tutorial.md %})), LotuS2 is currently fully functional only on this Galaxy server.

> <hands-on-title>Launch galaxy</hands-on-title>
>
> 1. Open your favorite browser (preferably Chrome/Chromium or Firefox. Internet Explorer/Edge and Safari sometimes have problems!)
> 2. Browse to [https://usegalaxy.eu](https://usegalaxy.eu)
> 3. We recommend keeping this tutorial open in a separate browser window side-by-side if you have space on your desktop. If you only have one screen, you can also launch tutorial mode on the UseGalaxy server.
>
> {% snippet faqs/galaxy/tutorial_mode.md %}
>
{: .hands_on}

# Understanding the files needed for running LotuS2

At {SPUN}, we run LotuS2 to identify mycorrhizal fungi in a set of samples, using the following input files:

1. DNA sequence files for each sample in **gzipped FASTQ** format from the Illumina MiSeq sequencer
2. A mapping file in a **tab-separated-values** format which specifies which FASTQ files correspond to which samples. The file can also provide additional metadata for each sample, such as the primers used for each sample, the location where the sample was collected, and so on.
3. A {sdm} file in **text** format which specifies how the sequence FASTQ files should be quality filtered and demultiplexed.

For this tutorial, we have already provided a few example files at {{ page.zenodo_link }}. You can click on this link to see which files are available (there is no need to download them).

These files correspond to the three types of input files above, as shown:

- DNA sequence files:
    1. C_ITS2_S160_R1_001.fastq.gz
    1. C_ITS2_S160_R2_001.fastq.gz
    1. N5_ITS2_S140_R1_001.fastq.gz
    1. N5_ITS2_S140_R2_001.fastq.gz
    1. Pcov3_ITS2_S151_R1_001.fastq.gz
    1. Pcov3_ITS2_S151_R2_001.fastq.gz
- A mapping file:
    1. Colombia_ITS2_Mapping.tsv
- An {sdm} file
    1. sdm_miSeq_ITS.txt

For this tutorial, you do not need to download these files. Galaxy allows you to fetch data from a remote location directly into a tool or workflow without first downloading the files.

In the next section, we will get the data into Galaxy, and after that, we will look at the files to see what they look like.

# Get Data

This section describes two options that will allow you to download and use {SPUN}'s example data files. Select one option and follow the instructions.

> <hands-on-title>Option 1: Data upload - Add to history</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the 6 sequencing read files, 1 mapping file, and 1 sdm parameters file from [Zenodo]({{ page.zenodo_link }}) by pasting the following URLs into the upload window under "Paste/Fetch Data"
>
>    ```
>    {{ page.zenodo_link }}/files/C_ITS2_S160_R1_001.fastq.gz
>    {{ page.zenodo_link }}/files/C_ITS2_S160_R2_001.fastq.gz
>    {{ page.zenodo_link }}/files/N5_ITS2_S140_R1_001.fastq.gz
>    {{ page.zenodo_link }}/files/N5_ITS2_S140_R2_001.fastq.gz
>    {{ page.zenodo_link }}/files/Pcov3_ITS2_S151_R1_001.fastq.gz
>    {{ page.zenodo_link }}/files/Pcov3_ITS2_S151_R2_001.fastq.gz
>    {{ page.zenodo_link }}/files/Colombia_ITS2_Mapping.tsv
>    {{ page.zenodo_link }}/files/sdm_miSeq_ITS.txt
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. The history should be visible in a pane on the right of your window called ‘History’. Here, you can see the files and rename them.
> 4. **Rename** {% icon galaxy-pencil %} the history to your name of choice. This should be something that helps you to remember the project, such as "SPUN Colombia 24"
>
{: .hands_on}

> <hands-on-title>Option 2: Data upload - Upload from your own computer</hands-on-title>
>
> 1. This option will take more time, and should not be used if you have a slow internet connection.
> 2. However, this option will most closely mimic how you will use this tool in your real analysis, i.e. showing you how to upload data from your own computer
> 3. First, download all 8 files from [Zenodo]({{ page.zenodo_link }}) to your local computer. You can do this by scrolling down to the "Files" list section and clicking the "Download All" link at the top of this section. This will download a 55 {MB} zip file to your local computer
> 4. Unzip the file - the folder should have 8 files in it.
> 5. Create a new history in Galaxy
> 6. Upload these 8 files to the new history
>
>    {% snippet faqs/galaxy/datasets_upload.md %}
>
{: .hands_on}

## Examine the input files

We will briefly look at each type of file to see that it has been uploaded correctly. This doesn’t need to be done for every file each time you use LotuS2 if you have a large number of files, but it is good practice to check some files to see that they have been uploaded in the correct format.

> <hands-on-title>Inspect the FASTQ file</hands-on-title>
>
> * Click on the file name of one of the fastq.gz files in the history
>    - The filename should expand to show you some information such as the size of the file, and the type of the file
> * Click on the {% icon galaxy-eye %} (eye) icon next to a fastq.gz file in the history
> * You should see 4 lines for each sequence:
>     1. A header beginning with `>` followed by sequence identifiers
>     2. The DNA sequence of the read, made up of ATGCN
>     3. A line with just `+` on it (indicates that the next line has sequence quality values)
>     4. A line with sequence quality values for each nucleotide, in ASCII format
>
> You do not need to know more about FASTQ files for this tutorial, or about sequence quality in ASCII format, but if you want to learn more, you can do the [Sequence Analysis: Quality Control tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).
>
{: .hands_on}


> <hands-on-title>Inspect the mapping {TSV} file</hands-on-title>
>
> * Click on the {% icon galaxy-eye %} (eye) icon next to *Colombia_ITS2_Mapping.tsv* in your history
> * Each row is a sample
> * The columns tell you information about each sample:
>    - #SampleID is a short name/ID for the sample. Here, "C" stands for "Control" sample, and the other two are called samples "N5" and "Pcov3"
>    - fastqFile lists the sequence file names that correspond to each sample. We have two FASTQ files for paired-end sequencing: first, the forward file is listed, followed by a "," (comma), and then the reverse file name is listed, with no spaces between these.
>    - ForwardPrimer: Lists the forward primer sequence used during PCR to produce the sequenced amplicon (ITS3 forward primer here).
>    - ReversePrimer: Lists the forward primer sequence used during PCR to produce the sequenced amplicon (ITS4 reverse primer here).
> * The remaining columns (Latitude, Longitude, Country, Vegetation, etc) are metadata for each sample. LotuS2 does not **need** these columns, but it will copy them to the final R Phyloseq object that it creates. These columns will be needed for doing ecological analyses later. You can add metadata to the phyloseq object that is created through LotuS2 later in R, as long as the sample ID in your metadata sheet matches the sample ID in your mapping file.
>
{: .hands_on}

> <comment-title>Note</comment-title>
> 
> - In this case, the ForwardPrimer and ReversePrimer are the same across all samples. This is typical as we usually sequence the same region in all samples using the same primer pair.
> - LotuS2 allows you to specify the primer set as a parameter, and if this is done, the primer sequence does not need to be written in the mapping file. In this case, we are including primer sequences in the mapping file. When using different primer sets, if you are following these instructions for your data, you may need to change the primer sequences to reflect the primer set used for your sequencing.
> 
{: .comment}

> <hands-on-title>Inspect the {sdm} options file</hands-on-title>
>
> * Click on the {% icon galaxy-eye %} (eye) icon next to *sdm_miSeq_ITS.txt* in your history
> * This file specifies some of the parameters needed for processing ITS data with Illumina MiSeq paired-end sequencing using the software 'SDM' used by LotuS2. It can be also used for our files that were generated using an Illlumina NextSeq 2000.
> * The sdm options file is not needed if you are using default sdm settings, because LotuS2 provides an internal default version of the file. But in this case, specific parameters are required for ITS sequencing, so we provide this file.
>
{: .hands_on}

> <question-title>Check your understanding of the inputs</question-title>
>
> 1. What is the reverse primer for sample "N5"?
> 2. What is the DNA concentration in ng per µl for sample "Pcov3"?
> 3. When LotuS2 processes ITS paired-end sequences using these inputs, what is the minimum sequence length (minSeqLength) it will consider, below which it will discard the sequence?
> 4. Which of the 6 FASTQ files has the most data? (i.e., is the biggest in size?)
>
> > <solution-title></solution-title>
> >
> >1. In *Colombia_ITS2_Mapping.tsv*: **TCCTCCGCTTATTGATATGC**
> >2. In *Colombia_ITS2_Mapping.tsv*, under column *DNA_concentration_ng_ul*: **20.4**
> >3. In *sdm_miSeq_ITS.txt*, next to *minSeqLength*: **110**
> >4. Click on the names of each of the fastq.gz files in the history. The biggest is N5_ITS2_S140_R2_001.fastq.gz: **19.0 MB**
> >
> {: .solution}
>
{: .question}


## Create a "list of pairs" from the fastq.gz files

- This is a very important step when you first upload the data, because the LotuS2 tool needs our fastq.gz files in pairs.
- Galaxy has a very handy feature that automatically detects pairs of files from the filenames

> <hands-on-title>Create pairs of fastq files</hands-on-title>
>
> 1. Select ONLY the 6 fastq.gz files in your history (do not select `sdm_miSeq_ITS.txt` and `Colombia_ITS2_Mapping.tsv`)
> 2. Choose the "Build List of Dataset Pairs" option for these 6 files! [Screenshot showing how to select 6 out of 8 items, and then click the top right dropdown to choose the "Build List of Dataset Pairs" option](images/history-build-list-of-dataset-pairs.png)
> 3. Galaxy will automatically pair the files for you on the next screen
> 4. Rename the new collection as "Colombia ITS2 FASTQ pairs"
> 5. By default these two options are checked: "Hide original elements" and "Remove file extensions". You can leave them checked or unchecked. If you leave them checked, the 6 individual fastq.gz files will disappear from your history and be replaced by one collection with 3 pairs of FASTQ files with the new name "Colombia ITS2 FASTQ pairs"
>
> {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
{: .hands_on}

# Run LotuS2

When we run the LotuS2 tool on our data, it runs many steps in the background:

1. demultiplexing and filtering raw FASTQ sequences
2. denoising, removing chimeric sequences and clustering sequences into very high quality {OTU}s/{ASV}s
3. determining the taxonomic origin of each OTU using specialized and general-purpose databases and statistical algorithms
4. constructing OTU, genus, family, class, order and phylum abundance tables in .txt or .biom format
5. reconstructing the OTU phylogenetic tree
6. generating phyloseq objects for downstream analysis

The figure below with an overview of LotuS2 is taken from {% cite ozkurt2022 %}.

![Flowchart image showing the different steps done by LotuS2](https://media.springernature.com/full/springer-static/image/art%3A10.1186%2Fs40168-022-01365-1/MediaObjects/40168_2022_1365_Fig1_HTML.png "Workflow of the LotuS2 pipeline")

As LotuS2 is a very powerful, general-purpose tool used in many metabarcoding projects for bacteria, fungi, and eukaryotes, it provides many different parameters (options for running the software) specified for each special use case.

In the next subsection, we show how to run LotuS2 in Galaxy and how to set the parameters needed for a fungal dataset.

## Run LotuS2 with the example fungal dataset

> <hands-on-title>Run LotuS2</hands-on-title>
>
> In the panel on the left, select "Tools", search for "LotuS2" and then run the tool using the parameters below. Leave the rest of the parameters at their default settings. 
>
> Make sure you have the right version (2.32+galaxy0). You can check the version by clicking the {% icon tool-versions %} (blocks) icon.
>
> 1. {% tool [LotuS2](toolshed.g2.bx.psu.edu/repos/earlhaminst/lotus2/lotus2/2.32+galaxy0) %} with the following parameters:
>    - *"Single- or Paired-end data?"*: `Paired-end list`
>        - In *"List of paired reads"*: choose the paired-list you created in the previous section: `Colombia ITS2 FASTQ pairs` (or whatever name you gave to the collection)
>    - In *"Mapping file (optional)"*: `Colombia_ITS2_mapping.tsv`
>    - Forward (and Reverse) Primer: Leave blank
>    - *"Clustering algorithm"*: `VSEARCH`
>    - *"Taxonomy aligner for taxonomic profiling of OTUs"*: `Lambda, LCA against custom reference database`
>        - *"Taxonomy reference database"*: `Use a built-in taxonomy database`
>            - *"Using reference database"*: `ITS fungi specific (UNITE)`
>    - In *"Other Clustering Options"*:
>        - *"Minimum size of dereplicated raw reads (optional)"*: `10:1,5:2,3:3`
>    - In *"Other Taxonomy Options"*:
>        - *"Amplicon type"*: `ITS2`
>        - *"Tax group"*: `fungal 18S/23S/ITS annotation`
>    - In *"Other options"*:
>        - *"SDM option file"*: `sdm_miSeq_ITS.txt`
>
> 2. Once all the parameters above are entered, click the **"Run Tool"** button at the end.
> 3. You should see a green box on the next page saying "Started tool LotuS2 and successfully added 1 job to the queue". The box lists the 6 outputs that the tool produces.
> 4. This step can take 5-10 minutes to run or longer depending on how many other jobs are running on the Galaxy server. While we are waiting for it to finish, you can do the next step on "Creating your own mapping.tsv" file
> 5. You will know when the tool has finished because all the outputs in the history will turn green.
> 
> 
>  > <comment-title>Notes</comment-title>
>  >
>  > - Remember to choose _Paired-end list_ in the sequencing read data section. Galaxy will pick up the Paired-end list available in the History, which will have the name you gave it in the _Create a list of pairs_ step
>  > - In *"Forward (and Reverse) Primer"*: Leave blank, as we have already provided them in the mapping.tsv file. If you are carrying out an analysis using your own data, you may add here the primer sequences used in your analyses instead of in the mapping file, if you prefer.
>  > - In *"Other Clustering Options"*: *"Minimum size of dereplicated raw reads (optional)"*: we put `10:1,5:2,3:3`. Each "X:Y" pair means "A unique dereplicated read must be seen at least X times in at least Y samples". So, if a sequence read is only found in 1 sample, it must be present in 10 copies. If a sequence read is found in only 2 samples, it must be found in 5 copies in each, etc. This is a strict parameter, so that sequence errors are not taken as novel biological sequences, reducing the occurrences of false positive OTUs or ASVs. You may want to use more lenient values if you know your data has a high number of real singletons, i.e., sequences that occur only once.
>  >
>  {: .comment}
>
{: .hands_on}

> <question-title>Examine the parameter options</question-title>
>
> 1. What other clustering algorithms can you use inside LotuS2?
> 2. What other amplicon types can LotuS2 classify?
>
> > <solution-title></solution-title>
> >
> >1. In the "LotuS2" tool's parameter page, under *"Clustering Algorithms"*, click on the drop down. You should see these tools:
> >    - DADA2
> >    - swarm
> >    - CD-HIT
> >    - VSEARCH
> >
> >    SPUN uses VSEARCH for our {ITS2} amplicon for the identification of fungi from soils.
> >2. In *"Other Taxonomy Options"*, click on the drop-down under *"Amplicon type"*:
> >    - Default
> >    - LSU
> >    - SSU
> >    - ITS
> >    - ITS1
> >    - ITS2
> >
> >    Here, we were using general fungal primers that amplify the {ITS2} region. In another tutorial, we will see how to use the {SSU} region for specific amplification of {AMF}
> >
> {: .solution}
>
{: .question}

# Examine the outputs

The LotuS2 Galaxy tool creates 6 output files that you should see in your history, with names like this (the exact number after each data might be different):
1. LotuS2 on data 8, data 6, and others: main log file
2. LotuS2 on data 8, data 6, and others: mapping file
3. LotuS2 on data 8, data 6, and others: Newick-formatted phylogenetic trees between sequences
4. LotuS2 on data 8, data 6, and others: FASTA-formatted extended OTU seed sequences
5. LotuS2 on data 8, data 6, and others: OTU abundance matrix
6. LotuS2 on data 8, data 6, and others: Complete LotuS2 output

> <hands-on-title>Examine outputs</hands-on-title>
> Click on the little {% icon galaxy-eye %} (eye) icon next to each output in the history to examine it. An explanation of each file is given below.
>
> 1. **main log file**: If we had run LotuS on the command line, the contents of the `output/LotuSLogS/LotuS_run.log` would be this main LotuS2 run log file.
> 
>     It has information on all the parameters passed to the tool, and information on the start time of each step in the program, plus the output of each step.
>   
>     You should always check this file first to see if LotuS2 completed correctly, and approximately how many reads were used/classified. If this number is much lower than what you expected, that might indicate a problem with the run and the parameters.
> 
> 2. **mapping file**: You can ignore this file as it is a repeat of the mapping TSV file (tab-separated-values) that we used as an input.
> 
> 3. **Newick-formatted phylogenetic trees between sequences**: A phylogenetic tree created from the ITS2 sequences. If you want to see what the tree looks like, you can copy the contents of this file and paste it at this [online tree viewer](http://etetoolkit.org/)treeview/
> 
> 4. **FASTA-formatted extended OTU seed sequences**: OTU sequences created by the LotuS2 program after clustering near-identical reads
> 
> 5. **OTU abundance matrix**: A tabular file with rows and columns with tab-separated-values. Each row is an OTU. The first column has the OTU name, and the remaining columns have the OTU abundance (i.e. how many reads were seen for that OTU) in each sample
> 
> 6. **Complete LotuS2 output**: If you try to view this file you will see some unreadable binary characters on the screen. That's because this is a zip file with the complete LotuS2 output folder in one zip folder. You should download this zip file and unzip it on your local computer if you want to see everything inside this folder. ![screenshot of how to download the zip file](images/history-download-zip.png)
> 
>     One of the most useful files in this output folder is the `phyloseq.Rdata` file which you can load in R to do further ecological analysis. This is beyond the scope of the current tutorial, but we will cover this topic in future tutorials. If you want to get started on your own, you can look at the last part "Exploration with phyloseq" in the [Building an amplicon sequence variant (ASV) table from 16S data using DADA2]({% link topics/microbiome/tutorials/dada-16S/tutorial.md %}) tutorial.
{: .hands_on}

> <question-title>Test your understanding of the outputs</question-title>
>
> If for some reason, your LotuS2 run did not complete, you can look at this [example LotuS2 run](https://usegalaxy.eu/u/sujai_spun_earth/h/identifying-mf-from-its2-sequencing-using-lotus2---tutorial-example-run) that we generated using the steps above.
>
> 1. How many total reads were in the OTU abundance matrix? (hint: main log file)
> 2. What percentage of reads are assigned at the phylum level and at the genus level? (hint: main log file)
> 3. How long did the entire LotuS2 pipeline take to run? (hint: the main log has time stamps at the left of each step in the format hh:mm:ss)
> 4. Which OTU was most abundant in all samples? (hint: OTU abundance matrix). How many reads were present in each sample?
> 5. What was the sequence of the least abundant OTU? (hint: FASTA-formatted extended OTU seed sequences)  
>
> > <solution-title>Answers</solution-title>
> >
> > For the first 3 questions, scroll to near the bottom of the *"main log file"*. Under "Calculating Taxonomic Abundance Tables" the answers in our [example LotuS2 run](https://usegalaxy.eu/u/sujai_spun_earth/h/identifying-mf-from-its2-sequencing-using-lotus2---tutorial-example-run) were:
> >
> > 1. Total reads in matrix: 132123
> > 2. Phylum	95%, Genus	65%
> > 3. 00:02:34 (i.e. 2 minutes 34 seconds)
> > 4. In the *"OTU abundance matrix"*, the OTUs are arranged in order of greatest-to-least abundance so OTU1 was the most abundant with 0 reads in sample **C** (Control), 25,659 reads in sample **N5**, and 0 reads in sample **Pcov3**
> > 5. In the *"OTU abundance matrix"*, the two least abundant sequences with 6 reads total each were OTU256 and OTU257. We can look up the sequences of these files in the *"FASTA-formatted extended OTU seed sequences"* at the bottom of the file:
> >
> > ```
> > >OTU256
> > >OTU257
> > ```
> >
> {: .solution}
>
{: .question}


# Run LotuS2 on your own data

To run the LotuS2 workflow on your own fungal data, you only need the sequencing FASTQ files and some information about what primers were used.

In the example dataset, we used the ITS3/ITS4 primer pair to sequence the ITS2 region, so our Forward primer was `GCATCGATGAAGAACGCAGC` and the Reverse primer was `TCCTCCGCTTATTGATATGC`

When you want to process your own sequencing files, you can specify the Forward and Reverse primer in the Galaxy LotuS2 Parameters directly in the options if all your samples have the same forward and reverse primers:
  - *"Forward primer used to amplify DNA region (optional) - optional"*: `GCATCGATGAAGAACGCAGC`
  - *"Reverse primer used to amplify DNA region (optional) - optional"*: `TCCTCCGCTTATTGATATGC`

We advise adding metadata to the phyloseq object after running LotuS2, so that a large mapping file with all metadata does not have to be created. We will have future tutorials on adding metadata from a .csv file to a phyloseq object in R.

## Create your own mapping tsv file

Mapping files for LotuS2 have to be in tab-separated-values format. You can create this file in any spreadsheet software, eg: Microsoft Excel or Google Sheets. The important thing to note is that when you save or export the file, you should select the TSV format (called "Tab separated values" in Google Sheets or "Tab-delimited text .txt" in Excel)

LotuS2 allows many columns in the mapping tsv file according to the [specification](https://lotus2.earlham.ac.uk//main.php?site=documentation#mappingfile). However, for simplicity, we recommend this format for sequencing data using the SPUN protocol:

- Only 1 header row that begins with `#SampleID`
- One row per sample
- The only essential columns are `#SampleID`, `fastqFile`
- If you have the same ForwardPrimer and ReversePrimer in all your samples, then you can provide them in the LotuS2 Galaxy tool and delete the ForwardPrimer and ReversePrimer columns
- If you have any additional metadata variables that you want to analyse downstream using R, then you can provide them as columns
- We recommend using "SequencingRun" and labelling with the name of the sequencing run, which is arbitrary. Something like "Scripps run 1". This is because if samples were run across multiple different sequencing runs it is important for LotuS2 to factor this into analyses. If samples were all run on the same run, write the same for every sample.
- In the data files for this tutorial, and in most cases, the sequencing provider will separate the data from separate samples into separate pairs of files. This is known as *demultiplexing*. If all your samples are combined in one pair of files, then you will need to provide the demultiplexing barcodes in the mapping tsv file.

For more information, see the [LotuS2 Mapping File Format Documentation](https://lotus2.earlham.ac.uk/main.php?site=documentation#mappingfile)

In the exercise below you will create your own mapping tsv file for a new run.

> <hands-on-title>Create and use your own mapping tsv file</hands-on-title>
>
> Steps:
>
> 1. Import history from the [mapping example history](https://usegalaxy.eu/u/sujai_spun_earth/h/identifying-mf-from-its2-sequencing-using-lotus2--create-mapping)
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
> 2. Create a table in your favourite spreadsheet software (eg *Google Sheets* or *Microsoft Excel*) with the following columns (you can copy and paste from the example below. We have filled in the first two columns for you #SampleID and fastqFile):
>
>    | #SampleID | fastqFile | ForwardPrimer | ReversePrimer | Vegetation |
>    | N11 | N11_ITS2_S134_R1_001.fastq.gz,N11_ITS2_S134_R2_001.fastq.gz | | | |
>    | N16 | N16_ITS2_S138_R1_001.fastq.gz,N16_ITS2_S138_R2_001.fastq.gz | | | |
>
> 3. Complete the table:
>   - Add the correct ForwardPrimer `GCATCGATGAAGAACGCAGC` and the correct ReversePrimer `TCCTCCGCTTATTGATATGC` to each row
>   - Add the Vegetation `Trigonobalanus excelsa forest` to each row
>
> 4. Export or Save the spreadsheet as a new file in tab-separated-values format (Google Sheets: _File_: _Download_: _Tab separated values (.tsv)_; Excel: _File_: _Save as_: choose Tab delimited text in the format)
>
> 5. Name the file something like "mapping.tsv" or "mapping.txt"
>
> 6. Upload this file to your history as a local file
>
>    {% snippet faqs/galaxy/datasets_upload.md %}
>
> 7. Now you should have all the files you need in this new history to run LotuS2:
>    - 4 paired-end FASTQ files for 2 samples
>    - 1 mapping tsv file specifying sample IDs, fastqFiles, primers, and metadata for these 2 samples
>    - 1 sdm_miSeq_ITS.txt
>
> 8. Rerun LotuS2 following the [steps above](#run-lotus2)
>
> 9. If your LotuS2 run finishes - then CONGRATULATIONS! You now know how to create your own mapping tsv file and run LotuS2 on a new set of fastq.gz files
>
{: .hands_on}

# Conclusion

- LotuS2 is a powerful tool that can be used to identify species present in a DNA sequencing sample
- You have learnt how to run LotuS2 on example data
- You have learnt how to create a mapping tsv file for a new set of data


topic: microbiome
tutorial: mgnify-amplicon
title: MGnify v5.0 Amplicon Pipeline
difficulty: introductory
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 3db490aeb379e2ea398da257c5fb294d
last_modified: 2026-02-26T21:12:17.321796
zenodo_link: https://zenodo.org/records/13710235
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/mgnify-amplicon/tutorial.html

content:
Amplicon sequencing is a powerful research method designed to explore and reveal the taxonomic composition of diverse microbial communities. By focusing on specific genetic regions, researchers can gain insights into the structure and function of these communities. Typically, ribosomal ribonucleic acid (rRNA) or ribosomal deoxyribonucleic acid (rDNA) is sequenced, with particular attention given to key marker genes. For bacteria and archaea, the 16S rRNA gene is commonly targeted; for eukaryotes, the 18S rRNA gene is used, while fungi are often studied through internal transcribed spacers (ITS) regions. These marker genes provide the genetic signatures necessary for identifying and categorizing the various organisms present within a sample.<br>

In this tutorial, we will dive into the MGnify amplicon pipeline v5.0 on Galaxy, a sophisticated toolset for microbial analysis. This pipeline is a ported version of the same well-established amplicon pipeline used by MGnify, a leading metagenomics platform dedicated to the analysis and archiving of microbiome sequences. MGnify has seen significant adoption and growth in recent years ({%cite Richardson2022%}); as of the time of writing, over 500k analyses have been conducted using this pipeline for amplicon data alone ({%cite ebi_metagenomics%}). One of the key advantages of this workflow is that it does not require users to know their target region in advance—it automatically processes all amplicons for ITS, SSU, and LSU, and can even handle mixed amplicon datasets. This workflow does not currently generate ASVs (Amplicon Sequence Variants). If ASV-level resolution is required, we recommend reading the [DADA2 tutorial]({% link topics/microbiome/tutorials/dada-16S/tutorial.md %}) or utilizing {% tool [LotuS2](toolshed.g2.bx.psu.edu/repos/earlhaminst/lotus2/lotus2/2.32+galaxy0) %}.

![Graphical overview of the full pipeline](./images/pipeline_v5.0_amplicon.png "<strong>The workflow of the MGnify amplicon pipeline v5.0</strong> ({%cite ebi_amplicon_pipeline%}). The figure illustrates the workflow of the MGnify amplicon pipeline, beginning with quality control to filter low-quality reads. The processed reads then proceed to the rRNA-prediction step, where SSU and LSU regions are classified and their taxonomic abundance is visualized. Following this, the pipeline handles the ITS regions, classifying them and visualizing their taxonomic abundance using pie charts. Each stage is clearly outlined, showcasing how data moves through the pipeline and is analyzed to provide comprehensive insights into microbial communities."){: style="width:45%; align-self:center"}

Our journey through this tutorial will provide a comprehensive understanding of the different sub-workflows that make up the pipeline and how each contributes to the overall analysis process. We will explore how these workflows are executed within the Galaxy platform. Additionally, we will explore which databases and tools are integrated into the pipeline, giving us a clearer picture of the resources that power these analyses. By the end of this tutorial you will have the knowledge to apply the MGnify amplicon pipeline effectively in your own research.<br>

Before diving into the tutorial, it is strongly recommended to review the Introduction to Microbiome Analysis slides or watch the corresponding video, which are linked in the requirements section above, especially if you are new to this field. These slides will provide essential background knowledge and help you better understand the concepts covered in this tutorial.

> <warning-title>Slight discrepancy between runs</warning-title>
> The outputs generated by this workflow may vary slightly between runs due to the non-deterministic behavior of the MAPseq version used. While this variation does not always occur, certain processing steps involve elements of randomness that can lead to minor differences in the results on different executions.
>
> ![Beta diversity benchmark](./images/beta_div_plot_mapseq2.1.1.png "<b>Beta diversity-based benchmark</b>. Dissimilarity between MAPseq runs. Executed 10 times using identical configuration and the same sample from analysis MGYA00578954 to demonstrate MAPseq inconsistency. The term ’All’ refers to all ranks combined. Species rank was excluded, due to taxa absence at this rank. Bars represent the average dissimilarity values, while individual data points represent the values for each of the ten different runs.")
>
{: .warning}

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

In this tutorial, we'll cover four different versions:
1. Executing the amplicon workflow using ENA data (default)
2. Executing the amplicon workflow using own reads, which requires some minor adjustments
2. Executing each subworkflow individually using ENA data
2. Executing each subworkflow individually using own reads

{% include _includes/cyoa-choices.html option1="Executing the full workflow using ENA data" option2="Executing subworkflows using ENA data" option3="Executing the full workflow using own data" option4="Executing subworkflows using own data" default="Executing the full workflow using ENA data" %}


# Import datasets

Let's begin with creating a history and giving it a suitable name.

> <hands-on-title>Create history</hands-on-title>
>
> 1. Create a new history
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the created history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}



<div class="Executing-the-full-workflow-using-ENA-data" markdown="1">

> <hands-on-title>Downloading datasets</hands-on-title>
> 1. Click on {% icon galaxy-upload %} **Upload Data** at the top of left panel
>    - Select {% icon galaxy-wf-edit %} **Paste/Fetch Data** at the bottom
>    - Copy {% icon copy %} and paste the following URLs into the text box:
>    - Click on **Start**
>
>      ```
>      https://zenodo.org/records/13710235/files/ribo.claninfo
>      https://zenodo.org/records/13710235/files/accessions.csv
>      ```
>
>    - **Close** the window
>    - **Note**: The file `ribo.claninfo` is required for the workflow to run properly. `ribo.claninfo` provides classification information for the rRNA models in ribo.cm, grouping them into so called clans.
{: .hands_on}

</div>



<div class="Executing-subworkflows-using-ENA-data" markdown="1">

> <hands-on-title>Downloading datasets</hands-on-title>
> 1. Click on {% icon galaxy-upload %} **Upload Data** at the top of left panel
>    - Select {% icon galaxy-wf-edit %} **Paste/Fetch Data** at the bottom
>    - Copy {% icon copy %} and paste the following URLs into the text box:
>    - Click on **Start**
>
>      ```
>      https://zenodo.org/records/13710235/files/ribo.claninfo
>      https://zenodo.org/records/13710235/files/accessions.csv
>      ```
>
>    - **Close** the window
>    - **Note**: The file `ribo.claninfo` is required for the workflow to run properly. `ribo.claninfo` provides classification information for the rRNA models in ribo.cm, grouping them into so called clans.
>
> 2. {% tool [fastq-dl](toolshed.g2.bx.psu.edu/repos/iuc/fastq_dl/fastq_dl/3.0.0+galaxy0) %} using the parameters:
>    - {% icon param-files %} *"select input type"*: `A list of ENA accession IDs, one per row`
>    - {% icon param-files %} *"Accession"*: `accessions.csv`
{: .hands_on}

</div>

<div class="Executing-subworkflows-using-own-data" markdown="1">

> <hands-on-title>Downloading datasets</hands-on-title>
>
> 1. Click on {% icon galaxy-upload %} **Upload Data** at the top of left panel
>    - Select {% icon galaxy-wf-edit %} **Paste/Fetch Data** at the bottom
>    - Copy {% icon copy %} and paste the following URLs into the text box:
>    - Click on **Start**
>
>      ```
>      https://zenodo.org/records/13710235/files/ribo.claninfo
>      ```
>
>   - **Close** the window
>   - **Note**: The file `ribo.claninfo` is required for the workflow to run properly. `ribo.claninfo` provides classification information for the rRNA models in ribo.cm, grouping them into so called clans.
>
{: .hands_on}

</div>



<div class="Executing-the-full-workflow-using-own-data" markdown="1">

> <hands-on-title>Downloading datasets</hands-on-title>
> 1. Click on {% icon galaxy-upload %} **Upload Data** at the top of left panel
>    - Select {% icon galaxy-wf-edit %} **Paste/Fetch Data** at the bottom
>    - Copy {% icon copy %} and paste the following URLs into the text box:
>    - Click on **Start**
>
>      ```
>      https://zenodo.org/records/13710235/files/ribo.claninfo
>      ```
>
>   - **Close** the window
>   - **Note**: The file `ribo.claninfo` is required for the workflow to run properly. `ribo.claninfo` provides classification information for the rRNA models in ribo.cm, grouping them into so called clans.
{: .hands_on}

</div>


<div class="Executing-the-full-workflow-using-own-data" markdown="1">
# Upload own data files

> <hands-on-title>Upload own data file</hands-on-title>
>
> 1. Upload the reads in `fastqsanger` or `fastqsanger.gz` format
>
>    {% snippet faqs/galaxy/datasets_upload.md %}
>
> 2. Create a collection out of the single-end datasets
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
> 3. Create a paired-end end collection for the paired-end datasets
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
{: .hands_on}

</div>

# Workflow Selection Guidance
Before running the rRNA or ITS sub-workflows, it's important to know the nature of your amplicon data:
- If your sequencing targets 16S or 18S rRNA genes, use the rRNA-prediction workflow.
- If your sequencing targets fungal ITS regions, use the ITS workflow.
- You do not need to run both workflows unless your dataset includes both types of sequences.

# Quality control
Quality control (QC) is a crucial first step in any metagenomic analysis pipeline. It ensures that sequencing reads are free from contaminants, low-quality bases, and adapter sequences, which can negatively impact downstream analyses. In the MGnify v5.0 pipeline, QC involves trimming, filtering, and assessing read quality using tools like Trimmomatic, SeqPrep, and FastQC. This process guarantees that only high-quality data progresses, providing reliable results for subsequent steps like taxonomic classification and diversity analysis.<br>
The quality control phase in this amplicon pipeline is split into two sub-workflows: one dedicated to paired-end reads and another for single-end reads.
In addition to quality-controlled single- and paired-end reads, both subworkflows also generate a MultiQC report that summarizes quality metrics across all input files. To help you interpret the MultiQC report, we recommend reviewing the Quality Control tutorial linked in the requirements section.

## Single-end reads
The quality control sub-workflow for single-end reads involves a series of tools that perform trimming, filtering, and quality assessment to ensure high-quality sequencing data. The workflow includes the following tools:
- {% tool [Trimmomatic](toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.39+galaxy2) %} trims low quality sequencing regions. Using the following parameters:
    - *"SLIDING WINDOW - Number of bases to average across"*: `4`
    - *"SLIDING WINDOW - Average quality required"*: `15`
    - *"LEADING"*: `3`
    - *"TRAILING"*: `3`
    - *"MINLEN"*: `100`
    - *"Quality score encoding"*: `Phred33`
- {% tool [Filter FASTQ](toolshed.g2.bx.psu.edu/repos/devteam/fastq_filter/fastq_filter/1.1.5) %} filters sequences shorter than 100 nucleotides in length out
- {% tool [PRINSEQ](toolshed.g2.bx.psu.edu/repos/iuc/prinseq/prinseq/0.20.4+galaxy2) %} provides advanced filtering, trimming, and data quality control, allowing for the removal of duplicate reads, low-complexity sequences, and additional quality filtering. Using the following parameters:
    - *"Is this library paired- or single-end?"*: `Single-end`
    - *"Apply filter treatments?"*: `Yes`
    - *"Filter sequences based on their length?"*: `No`
    - *"Filter sequences based on quality score?"*: `No`
    - *"Filter sequences based on their base content?"*: `Yes`
    - *"Filter sequences based on their GC percentage?"*: `No`
    - *"Filter sequences based on their number of N bases?"*: `No`
    - *"Filter sequences based on their percentage of N bases?"*: `Yes`
    - *"Maximal N percentage threshold to conserve sequences"*: `10`
    - *"Filter sequences with characters other than A, T, C, G and N?"*: `Yes`
    - *"Filter sequences based on their complexity?"*: `No`
    - *"Apply trimming treatments?"*: `No`
- {% tool [convert FASTQ to FASTA](toolshed.g2.bx.psu.edu/repos/devteam/fastqtofasta/fastq_to_fasta_python/1.1.5) %} converts the quality control processed reads from FASTQ into FASTA format
- {% tool [FASTQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy0) %} performs comprehensive quality checks on sequencing data, providing visual summaries and detailed metrics for read quality
- {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy1) %} aggregates reports from multiple quality control FASTQC reports, into a single, consolidated report for easier interpretation. Using the following parameters:
    - **Which tool was used generate logs?**: `FastQC`
    - **Type of FastQC output?**: `Raw data`

<div class="Executing-subworkflows-using-ENA-data" markdown="1">
> <hands-on-title>Executing the single-end quality control sub-workflow</hands-on-title>
> 1. **Import** the [single-end quality control sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-quality-control-single-end.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Run** {% icon workflow-run %} the single-end sub-workflow using the following input:
>    - **Single-end reads**: `Single-end data` collection
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

<div class="Executing-subworkflows-using-own-data" markdown="1">
> <hands-on-title>Executing the single-end quality control sub-workflow</hands-on-title>
> 1. **Import** the [single-end quality control sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-quality-control-single-end.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. Click on {% icon galaxy-upload %} **Upload Data** at the top of left panel
> 3. Click on {% icon tutorial %} **Choose local file** and select the desired single-end files in `fastqsanger` or `fastqsanger.gz` format
> 4. Click on **Start** and **Close** the window
> 5. When the upload is finished create a collection for all single-end reads files:
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
> 5. **Run** {% icon workflow-run %} the single-end sub-workflow using the following input:
>    - **Single-end reads**: Single-end reads collection we just created
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

## Paired-end reads
The quality control sub-workflow for paired-end reads is designed to optimize the accuracy and reliability of sequencing data by addressing the unique challenges of handling paired-end sequences. The tools included are:
- {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.23.4+galaxy1) %} A versatile tool that performs quality filtering, trimming, adapter removal, and other preprocessing tasks specifically for paired-end reads. Using the following parameters:
    - *"Single-end or paired reads"*: `Paired`
    - *"Filter Options"*:
        - *"Quality filtering options"*:
            - *"Qualified quality phred"*: `20`
            - *"Unqualified percent limit"*: `20`
        - *"Length filtering options"*:
            - *"Length required"*: `70`
    - *"Read Modification Options"*:
        - *"Base correction by overlap analysis options"*:
            - *"Enable base correction"*: `Yes`
- {% tool [SeqPrep](toolshed.g2.bx.psu.edu/repos/iuc/seqprep/seqprep/1.3.2+galaxy0) %} Merges the forward and reverse reads. Using the default parameters.
- {% tool [Trimmomatic](toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.39+galaxy2) %} trims low quality sequencing regions. Using the following parameters:
    - *"SLIDING WINDOW - Number of bases to average across"*: `4`
    - *"SLIDING WINDOW - Average quality required"*: `15`
    - *"LEADING"*: `3`
    - *"TRAILING"*: `3`
    - *"MINLEN"*: `100`
    - *"Quality score encoding"*: `Phred33`
- {% tool [Filter FASTQ](toolshed.g2.bx.psu.edu/repos/devteam/fastq_filter/fastq_filter/1.1.5) %} filters sequences shorter than 100 nucleotides in length out
- {% tool [PRINSEQ](toolshed.g2.bx.psu.edu/repos/iuc/prinseq/prinseq/0.20.4+galaxy2) %} provides advanced filtering, trimming, and data quality control, allowing for the removal of duplicate reads, low-complexity sequences, and additional quality filtering. Using the following parameters:
    - *"Is this library paired- or single-end?"*: `Single-end`
    - *"Apply filter treatments?"*: `Yes`
    - *"Filter sequence based on their length?"*: `No`
    - *"Filter sequences based on quality score?"*: `No`
    - *"Filter sequences based on their base content?"*: `Yes`
    - *"Filter sequences based on their GC percentage?"*: `No`
    - *"Filter sequences based on their number of N bases?"*: `No`
    - *"Filter sequences based on their percentage of N bases?"*: `Yes`
    - *"Maximal N percentage threshold to conserve sequences"*: `10`
    - *"Filter sequences with characters other than A, T, C, G and N?"*: `Yes`
    - *"Filter sequences based on their complexity?"*: `No`
    - *"Apply trimming treatments?"*: `No`
- {% tool [convert FASTQ to FASTA](toolshed.g2.bx.psu.edu/repos/devteam/fastqtofasta/fastq_to_fasta_python/1.1.5) %} converts the quality control processed reads from FASTQ into FASTA format
- {% tool [FASTQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy0) %} performs comprehensive quality checks on sequencing data, providing visual summaries and detailed metrics for read quality
- {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy1) %} aggregates reports from multiple quality control FASTQC reports, into a single, consolidated report for easier interpretation. Using the following parameters:
    - *"Which tool was used generate logs?"*: `FastQC`
    - *"Type of FastQC output?"*: `Raw data`

<div class="Executing-subworkflows-using-ENA-data" markdown="1">
> <hands-on-title>Executing the paired-end quality control sub-workflow</hands-on-title>
> 1. **Import** the [paired-end quality control sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-quality-control-paired-end.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Run** {% icon workflow-run %} the paired-end sub-workflow using the following input:
>    - **Paired-end reads**: `Pair-end data` collection
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

<div class="Executing-subworkflows-using-own-data" markdown="1">
> <hands-on-title>Executing the paired-end quality control sub-workflow</hands-on-title>
> 1. **Import** the [paired-end quality control sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-quality-control-paired-end.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. Click on {% icon galaxy-upload %} **Upload Data** at the top of left panel
> 3. Click on {% icon tutorial %}**Choose local file** and select the desired paired-end files in `fastqsanger` or `fastqsanger.gz` format
> 4. Click on **Start** and **Close** the window
> 5. When the upload is finished create a paired collection for all paired-end reads files:
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
> 6. **Run** {% icon workflow-run %} the paired-end sub-workflow using the following input:
>    - **Paired-end reads**: Paired-end reads collection, which was created in the steps above
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

# rRNA-prediction
The rRNA prediction sub-workflow focuses on identifying and classifying ribosomal RNA (rRNA) sequences, particularly small subunit (SSU) and large subunit (LSU) rRNAs. This workflow uses several tools for comparison against covariance models, filtering, classification, and visualization:
- {% tool [CMseach](toolshed.g2.bx.psu.edu/repos/bgruening/infernal/infernal_cmsearch/1.1.4+galaxy0) %} the datasets are compared against the LSU and SSU models from Rfam (v13.0). Using the following parameters:
    - *"Subject covariance models"*: `Locally installed covariance models`
    - *"Covariance models"*: `Mgnify 5.0 RFAM Models`
    - *"Calculate E-values as if the search space size is 'x' megabases (Mb)"*: `1000`
    - *"Options controlling acceleration heuristics"*: `Use HMM only, don't use a CM at all (--hmmonly)`
    - *"Omit the alignment section from the main input"*: `Yes`
- {% tool [cmsearch-deoverlap](toolshed.g2.bx.psu.edu/repos/rnateam/cmsearch_deoverlap/cmsearch_deoverlap/0.08+galaxy2) %} removes overlapping reads that belong to the same clan. Using the default parameters.
- {% tool [bedtools getfasta](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_getfastabed/2.30.0+galaxy1) %} extract sequences from a FASTA file
- {% tool [MAPseq](toolshed.g2.bx.psu.edu/repos/iuc/mapseq/mapseq/2.1.1+galaxy0) %} classifies matching OTUs to the reads, used in conjunction with SILVA (v132) SSU and LSU databases. Using the following parameters:
    - *"Use cached database or database from history"*: `Cached database`
    - *"Using built-in mapseq DB**. MAPseq is used in conjunction with the following databases:
        1. `MGnify LSU (v5.0.7) - silva_lsu-20200130 downloaded at 2023-09-26`
        2. `MGnify SSU (v5.0.7) - silva_ssu-20200130 downloaded at 2023-09-26`
    - *"Create OTU table"*: `Yes`
    - *"Create taxon table for Krona"*: `Yes`
    - *"Top hits"*: `80`
    - *"Top OTUs"*: `40`
    - *"Output format"*: `simple`
- {% tool [Krona](toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1+galaxy0) %} generates pie charts demonstrating the taxonomic abundance. Using the following parameters:
    - *"What is the type of your input data"*: `Tabular`
    - *"Provide a name for the basal rank"*: `Root`
- {% tool [biom-convert](toolshed.g2.bx.psu.edu/repos/iuc/biom_convert/biom_convert/2.1.15+galaxy1) %} converts the taxonomic abundance tables to JSON and HDF5 formats. Using the following parameters:
    - *"Choose the source BIOM format"*: `Tabular File`
    - *"Process metadata associated with observations when converting"*: `taxonomy`
    - *"Choose the output type"*: `BIOM`
    - *"Table Type"*: `OTU table`
    - *"BIOM format type"*:
        1. `JSON-formatted table(BIOM1)`
        2. `HDF5-formatted table(BIOM2)`

<div class="Executing-subworkflows-using-ENA-data" markdown="1">

> <hands-on-title>Executing the rRNA-prediction sub-workflow</hands-on-title>
>
> 1. **Import** the [rRNA-prediction sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-rrna-prediction.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. Using {% tool [Merge collections](__MERGE_COLLECTION__) %} merge both quality processed single- and paired-end reads collections: `Single-end post quality control FASTA files` and `Paired-end post quality control FASTA files`
> 3. **Run** {% icon workflow-run %} the rRNA-prediction sub-workflow using the following inputs:
>    - **Processed sequences**: Quality processed sequences (merged collection)
>    - **Clan information file**: `ribo.claninfo`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

<div class="Executing-subworkflows-using-own-data" markdown="1">
> <hands-on-title>Executing the rRNA-prediction sub-workflow</hands-on-title>
> 1. **Import** the [rRNA-prediction sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-rrna-prediction.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
> 2. Using {% tool [Merge collections](__MERGE_COLLECTION__) %} merge both quality processed single- and paired-end reads collections: `Single-end post quality control FASTA files` and `Paired-end post quality control FASTA files`
> 3. **Run** {% icon workflow-run %} the rRNA-prediction sub-workflow using the following inputs:
>    - **Processed sequences**: Quality processed sequences (merged collection)
>    - **Clan information file**: `ribo.claninfo`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

# ITS
The ITS (Internal Transcribed Spacer) workflow is designed for the identification and taxonomic classification of fungal sequences, specifically focusing on ITS regions. The sub-workflow includes several tools to mask, classify, and visualize taxonomic data:
- {% tool [bedtools MaskFastaBed](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_maskfastabed/2.30.0) %} masks SSU and LSU regions from a FASTA file based on intervals defined in a BED regions file. Using the default parameters.
- {% tool [MAPseq](toolshed.g2.bx.psu.edu/repos/iuc/mapseq/mapseq/2.1.1+galaxy0) %} classifies matching OTUs to the reads, used in conjunction with the UNITE (v8.0) and ITSoneDB (v1.138) databases. Using the following parameters:
    - *"Use cached database or database from history"*: `Cached database`
    - *"Using built-in mapseq DB**. MAPseq is used in conjunction with the following databases:
        1. `MGnify ITS ITSonedb (v5.0.7) - ITSoneDB-20200214 downloaded at 2023-09-26`
        2. `MGnify ITS UNITE (v5.0.7) - UNITE-20200214 downloaded at 2023-09-26`
    - *"Create OTU table"*: `Yes`
    - *"Create taxon table for Krona"*: `Yes`
    - *"Top hits"*: `80`
    - *"Top OTUs"*: `40`
    - *"Output format"*: `simple`
- {% tool [Krona](toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1+galaxy0) %} generates pie charts demonstrating the taxonomic abundance. Using the following parameters:
    - *"What is the type of your input data"*: `Tabular`
    - *"Provide a name for the basal rank"*: `Root`
- {% tool [biom-convert](toolshed.g2.bx.psu.edu/repos/iuc/biom_convert/biom_convert/2.1.15+galaxy1) %} converts the taxonomic abundance tables to JSON and HDF5 formats. Using the following parameters:
    - *"Choose the source BIOM format"*: `Tabular File`
    - *"Process metadata associated with observations when converting"*: `taxonomy`
    - *"Choose the output type"*: `BIOM`
    - *"Table Type"*: `OTU table`
    - *"BIOM format type"*:
        1. `JSON-formatted table(BIOM1)`
        2. `HDF5-formatted table(BIOM2)`

<div class="Executing-subworkflows-using-ENA-data" markdown="1">
> <hands-on-title>Executing the ITS sub-workflow</hands-on-title>
> 1. **Import** the [ITS sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-its.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Run** {% icon workflow-run %} the ITS sub-workflow using the following inputs:
>    - *"LSU and SSU BED"*: `LSU and SSU BED regions`
>    - *"Processed sequence"*: Quality processed sequences (merged single-end and paired-end collection)
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

<div class="Executing-subworkflows-using-own-data" markdown="1">
> <hands-on-title>Executing the ITS sub-workflow</hands-on-title>
> 1. **Import** the [ITS sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-its.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Run** {% icon workflow-run %} the ITS sub-workflow using the following inputs:
>    - *"LSU and SSU BED"*: `LSU and SSU BED regions`
>    - *"Processed sequence"*: Quality processed sequences (merged single-end and paired-end collection)
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

# Summary tables
The summary tables sub-workflow takes as input the taxonomic abundance tables generated from all datasets within the pipeline. It processes these tables to create two comprehensive taxonomic summary tables:
1. Full Taxonomic Summary Table: This table provides detailed taxonomic information spanning all levels, from superkingdom down to species. It offers a complete overview of the taxonomic composition across all samples, facilitating in-depth analyses of microbial communities.

2. Phylum-Level Summary Table: Focused specifically on the phylum level, this table aggregates data to provide a clear view of the phylum distribution within the datasets. This level-specific summary is particularly useful for high-level comparisons and identifying major trends within the microbial communities.

Both summary tables serve as essential inputs for downstream interpretation, helping to visualize and compare the taxonomic profiles across multiple samples (for example, see our tutorial on[Calculating α and β diversity]({% link topics/microbiome/tutorials/diversity/tutorial.md %})).

<div class="Executing-subworkflows-using-ENA-data" markdown="1">

> <hands-on-title>Executing the summary tables sub-workflow</hands-on-title>
> 1. **Import** the [summary tables sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-summary-tables.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Run** {% icon workflow-run %} the summary tables sub-workflow 4 times using the following inputs:
>    - First run:
>      - *"OTU tables"*: `LSU OTU tables (SILVA DB)`
>      - *"Taxonomic abundance summary table name"*:
>        ```
>        LSU taxonomic abundance summary table
>        ```
>      - *"Phylum level taxonomic abundance summary table name"*:
>        ```
>        LSU phylum level taxonomic abundance summary table
>        ```
>    - Second run:
>      - *"OTU tables"*: `SSU OTU tables (SILVA DB)`
>      - *"Taxonomic abundance summary table name"*:
>        ```
>        SSU taxonomic abundance summary table
>        ```
>      - *"Phylum level taxonomic abundance summary table name"*:
>        ```
>        SSU phylum level taxonomic abundance summary table
>        ```
>    - Third run:
>      - *"OTU tables"*: `ITS OTU tables (UNITE DB)`
>      - *"Taxonomic abundance summary table name"*:
>        ```
>        UNITE taxonomic abundance summary table
>        ```
>      - *"Phylum level taxonomic abundance summary table name"*:
>        ```
>        UNITE phylum level taxonomic abundance summary table
>        ```
>    - Fourth run:
>      - *"OTU tables"*: `ITS OTU tables (ITSoneDB)`
>      - *"Taxonomic abundance summary table name"*:
>        ```
>        ITSoneDB taxonomic abundance summary table
>        ```
>      - *"Phylum level taxonomic abundance summary table name"*:
>        ```
>        ITSoneDB phylum level taxonomic abundance summary table
>        ```
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

<div class="Executing-subworkflows-using-own-data" markdown="1">
> <hands-on-title>Executing the summary tables sub-workflow</hands-on-title>
> 1. **Import** the [summary tables sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-summary-tables.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Run** {% icon workflow-run %} the summary tables sub-workflow 4 times using the following inputs:
>    - First run:
>      - *"OTU tables"*: `LSU OTU tables (SILVA DB)`
>      - *"Taxonomic abundance summary table name"*:
>        ```
>        LSU taxonomic abundance summary table
>        ```
>      - *"Phylum level taxonomic abundance summary table name"*:
>        ```
>        LSU phylum level taxonomic abundance summary table
>        ```
>    - Second run:
>      - *"OTU tables"*: `SSU OTU tables (SILVA DB)`
>      - *"Taxonomic abundance summary table name"*:
>        ```
>        SSU taxonomic abundance summary table
>        ```
>      - *"Phylum level taxonomic abundance summary table name"*:
>        ```
>        SSU phylum level taxonomic abundance summary table
>        ```
>    - Third run:
>      - *"OTU tables"*: `ITS OTU tables (UNITE DB)`
>      - *"Taxonomic abundance summary table name"*:
>        ```
>        UNITE taxonomic abundance summary table
>        ```
>      - *"Phylum level taxonomic abundance summary table name"*:
>        ```
>        UNITE phylum level taxonomic abundance summary table
>        ```
>    - Fourth run:
>      - *"OTU tables"*: `ITS OTU tables (ITSoneDB)`
>      - *"Taxonomic abundance summary table name"*:
>        ```
>        ITSoneDB taxonomic abundance summary table
>        ```
>      - *"Phylum level taxonomic abundance summary table name"*:
>        ```
>        ITSoneDB phylum level taxonomic abundance summary table
>        ```
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

# Complete Workflow (including all sub-workflows)
If you're working with SSU, LSU, and ITS data, it's beneficial to run the entire workflow to take full advantage of its capabilities. Even if you're unsure which of these datasets you have, executing the complete workflow ensures that all relevant analyses are applied, maximizing the insight gained from your microbial data.

<div class="Executing-the-full-workflow-using-ENA-data" markdown="1">
> <hands-on-title>Executing the complete MGnify amplicon pipeline</hands-on-title>
> 1. **Import** the [complete MGnify amplicon workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-complete.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Run** {% icon workflow-run %} the workflow using the following inputs:
>    - *"SRA accession list"*: `accessions.csv`
>    - *"Clan information file"*: `ribo.claninfo`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

<div class="Executing-the-full-workflow-using-own-data" markdown="1">
> <hands-on-title>Executing the complete MGnify amplicon pipeline</hands-on-title>
> 1. **Import** the [complete MGnify amplicon workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-complete.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. Adjust workflow:
>    - Click on {% icon galaxy-workflows-activity %} **Workflows** in the left side-bar
>    - Search for *"MGnify's amplicon pipeline v5.0 - complete"* workflow
>    - Click on {% icon galaxy-wf-edit %} **Edit**
>    - **Remove** the steps:
>      - *"1: SRA accession list"*
>      - *"22: fastq-dl"*
>      - *"23: Convert compressed file to uncompressed"
>      - *"24: Convert compressed file to uncompressed"
>      - *"25: Text reformatting"
>      - *"26: Text reformatting"
>      - *"27: Convert uncompressed file to compressed"
>      - *"28: Convert uncompressed file to compressed"
>    - Click on **Inputs** in the left side-panel and add an **Input dataset collection**
>      - Add the *"Label"*: `Single-end reads`
>      - Connect it to the *"Single-end reads"* input in step *"29: MGnify's amplicon pipeline v5.0 - Quality control SE"*
>    - Click on **Inputs** in the left side-panel and add an **Input dataset collection**
>      - Add the *"Label"*: `Paired-end reads`
>      - Change *"Collection type"* to `list:paired`
>        - Connect it to the *"Paired-end reads"* input in step *"30: MGnify's amplicon pipeline v5.0 - Quality control PE"*
>    - Click on {% icon galaxy-save %} **Save** and then {% icon workflow-run %} **Run** in the right side-panel
> 3. **Run** {% icon workflow-run %} the workflow using the following inputs:
>    - *"Single-end reads"*: Single-end reads collection
>    - *"Paired-end reads"*: Paired-end reads collection
>    - *"Clan information file"*: Upload the clan information file, which contains details about the sequence family assignments required for accurate taxonomic classification `ribo.claninfo`
>    - *"Covariance models"*: Provide the covariance models necessary for the rRNA prediction step, enabling accurate alignment and prediction of ribosomal RNA sequences `ribo.cm`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
> > <tip-title>Input reformatting</tip-title>
> > To achieve results comparable to those from MGnify, it is recommended that your sequence headers follow this format: `@accession.spotID-$spotName-$readNumber`. For example: `@ERR2015153.1-M01170:20:000000000-AMCTR:1:1101:17071:1623-2`
> {: .tip}
>
{: .hands_on}
</div>

# MAPseq-to-ampvis2
The MAPseq to ampvis2 workflow transforms MAPseq OTU tables and accompanying metadata into a format compatible with ampvis2. This process restructures the MAPseq output into well-organized object files that can be directly used in ampvis2 for downstream analyses such as rank abundance plots, heatmaps, and other visualizations.

> <hands-on-title>Downloading datasets</hands-on-title>
> 1. Click on {% icon galaxy-upload %} **Upload Data** at the top of left panel
>    - Select {% icon galaxy-wf-edit %} **Paste/Fetch Data** at the bottom
>    - Copy {% icon copy %} and paste the following:
>      ```
>      https://zenodo.org/records/13347829/files/SRR11032273.tsv
>      https://zenodo.org/records/13347829/files/SRR11032274.tsv
>      https://zenodo.org/records/13347829/files/SRR11032275.tsv
>      https://zenodo.org/records/13347829/files/SRR11032276.tsv
>      https://zenodo.org/records/13347829/files/SRR11032277.tsv
>      https://zenodo.org/records/13347829/files/SRR11038211.tsv
>      https://zenodo.org/records/13347829/files/SRR11038212.tsv
>      https://zenodo.org/records/13347829/files/SRR11038213.tsv
>      https://zenodo.org/records/13347829/files/SRR11038214.tsv
>      https://zenodo.org/records/13347829/files/SRR11038215.tsv
>      https://zenodo.org/records/13347829/files/test_metadata_formatted.tabular
>      ```
>    - Click on **Start**
> 2. Create a collection out of the tsv files
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
{: .hands_on}

> <hands-on-title>Executing the MAPseq-to-ampvis2 workflow</hands-on-title>
> 1. **Import** the [MAPseq-to-ampvis2 workflow]({{ site.baseurl }}{{ page.dir }}workflows/mapseq-to-ampvis2.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Run** {% icon workflow-run %} the workflow using the following inputs:
>      - *"MAPseq outputs"*: Provide the OTU tables generated by MAPseq, which need to be reformatted for ampvis2
>      - *"Metadata"*: Contextual information associated with the data from the OTU tables
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}

With the ampvis2 object now successfully generated, users can proceed to create a variety of informative visualizations—such as heatmaps—to explore and interpret microbial community composition, identify patterns across samples, and support further ecological or statistical analysis.

> <hands-on-title>Utilizing the ampvis2 object to generate a heatmap</hands-on-title>
>
> 1. {% tool [ampvis2 heatmap](toolshed.g2.bx.psu.edu/repos/iuc/ampvis2_heatmap/ampvis2_heatmap/2.8.9+galaxy1) %}:
>    - **Ampvis2 RDS dataset**: Choose `Ampvis2 object` from your history
>    - **The taxonomic level to aggregate the OTUs**: `Order`
>    - **How to show OTUs without taxonomic information**: `Remove OTUs without taxonomic information`
>    - Select light blue for **Start color for the heatmap**
>    - Select dark red for **End color for the heatmap**
>
{: .hands_on}

Let us now take a look at the generated heatmap.

![Heatmap produced by the workflow](./images/heatmap.png){: style="width:75%;"}


> <question-title>i</question-title>
>
> 1. Which sample has the highest relative abundance of Lactobacillales?
> 2. Which taxonomic order is most dominant in sample SRR1038213?
>
> > <solution-title></solution-title>
> >
> > 1. Sample SRR11032276 has the highest relative abundance of Lactobacillales at 97.7%, indicating it is strongly dominated by this bacterial order.
> > 2. In sample SRR11038213, Tremellales is the most dominant order, with a relative abundance of 50%.
> >
> {: .solution}
>
{: .question}

This was a brief glimpse into the capabilities of ampvis2. We encourage you to further explore the full range of features available in Ampvis2, as well as other powerful visualization tools offered on the Galaxy platform.

# Conclusion
This tutorial provided a step-by-step guide to executing the different sub-workflows of the MGnify v5.0 amplicon pipeline on Galaxy. By following these steps, you should be able to process, analyze, and visualize your amplicon sequencing data efficiently.


topic: microbiome
tutorial: metatranscriptomics
title: Metatranscriptomics analysis using microbiome RNA-seq data
difficulty: introductory
time_estimation: 5H
content: {% include topics/microbiome/tutorials/metatranscriptomics/content.md short=false %}
tools_json: []
requirements_json: []
tags_json: ["metatranscriptomics", "microgalaxy"]
content_hash: 29f09e32001e8c823803990143837078
last_modified: 2026-02-26T21:12:17.319815
zenodo_link: https://zenodo.org/record/4776250
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/metatranscriptomics/tutorial.html


topic: microbiome
tutorial: clinical-mp-2-discovery
title: Clinical Metaproteomics 2: Discovery
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["[label-TMT11]"]
content_hash: 6dfe42e96d7b7bfa9a0f8ec8e07c93a9
last_modified: 2026-02-26T21:12:17.463276
zenodo_link: https://doi.org/10.5281/zenodo.10105821
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/clinical-mp-2-discovery/tutorial.html

content:
This tutorial can be followed with any user-defined database but would work best if the clinical metaproteomics database generation module was used (see [Database Generation tutorial](https://github.com/subinamehta/training-material/blob/main/topics/proteomics/tutorials/clinical-mp-database-generation/tutorial.md)). The MetaNovo tool generates a more manageable database that contains identified proteins. The MetaNovo-generated database merged with Human SwissProt (reviewed only) and contaminants (cRAP) databases to generate a compact database (~21.2k protein sequences) that will be used for peptide identification.


# Peptide identification
The MSMS data will be searched against the compact database `Human UniProt Microbial Proteins (from MetaNovo) and cRAP` to identify peptide and protein sequences via sequence database searching. For this tutorial, two peptide identification programs will be used: SearchGUI/PeptideShaker and MaxQuant. However, you could use other software too, such as Fragpipe or Scribe. For the purpose of this tutorial, a dataset of the 4 RAW/MGF files will be used as the MS/MS input.


![Discovery Workflow]({% link topics/proteomics/images/clinical-mp/clinical-mp-discovery.JPG %})


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Database Searching
This step is to identify proteins based on mass spectrometry data. The algorithms identify peptides in the spectra and search a protein sequence database to match observed peptide data with theoretical peptide masses and spectra. Scoring and false discovery rate control help assess the reliability of matches, followed by protein inference to determine the proteins present in the sample. These algorithms are essential for interpreting mass spectrometry data, aiding in protein identification, quantification, and insights into biological processes and disease mechanisms in proteomics research.

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/10105821/files/Human_UniProt_Microbial_Proteins_(from_MetaNovo)_and_cRAP.fasta
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F10_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F11_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F13_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F15_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/Experimental-Design_Discovery_MaxQuant.tabular
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Add to each database a tag corresponding to user.
> 6. Create a dataset collection of all the raw files and MGF files.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Import Workflow


> <hands-on-title>Running the Workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/proteomics/tutorials/clinical-mp-2-discovery/workflows/WF2_Discovery-Workflow.ga" title="Discovery Workflow" %}
>
> 2. Run **Workflow** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *" RAW files"*: `RAW dataset collection`
>    - {% icon param-file %} *" Human UniProt Microbial Proteins (from MetaNovo) and cRAP"*: `Human_UniProt_Microbial_Proteins_(from_MetaNovo)_and_cRAP.fasta`
>    - {% icon param-file %} *" Experimental Design Discovery MaxQuant"*: `Experimental-Design_Discovery_MaxQuant.tabular`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}


# Peptide identification
Using the compact database generated by MetaNovo as the input database, we will match MS/MS data to peptide sequences via sequence database searching.

For this tutorial, two peptide identification programs will be used: SearchGUI/PeptideShaker and MaxQuant. For both programs, the created dataset of the four MS datasets  in the history will be used as the MS/MS input. The RAW MS/MS data files will be converted into mascot generic format (MGF) files as that is the standard format in which MS/MS searches are performed.

Peptides identified from each program will be verified with the PepQuery tool to generate a master list of confident verified microbial peptides.

## Appending decoy sequenced to FASTA database with **FastaCLI**

Using the FastaCLI tool, decoy sequences will be appended to the FASTA database. Decoy sequences are protein sequences are not expected to be present in samples. For more information on how to generate and append decoy sequences, see [GTN Protein FASTA Database Handling](https://training.galaxyproject.org/archive/2019-12-01/topics/proteomics/tutorials/database-handling/tutorial.html#creating-a-decoy-database).

> <hands-on-title> FastaCLI </hands-on-title>
>
> 1. {% tool [FastaCLI](toolshed.g2.bx.psu.edu/repos/galaxyp/peptideshaker/fasta_cli/4.0.41+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Protein Database"*: `output` (Input dataset)
>
>
{: .hands_on}

## Converting RAW files to MGF files with **msconvert**

The msconvert tool allows for the conversion of mass spectrometry data files between different formats, such as thermo.raw, mgf, or mzml.

> <hands-on-title> msconvert: RAW to MGF </hands-on-title>
>
> 1. {% tool [msconvert](toolshed.g2.bx.psu.edu/repos/galaxyp/msconvert/msconvert/3.0.20287.2) %} with the following parameters:
>    - {% icon param-collection %} *"Input unrefined MS data"*: `output` (Input dataset collection)
>    - *"Do you agree to the vendor licenses?"*: `Yes`
>    - *"Output Type"*: `mgf`
>    - In *"Data Processing Filters"*:
>        - *"Apply peak picking?"*: `Yes`
>        - *"(Re-)calculate charge states?"*: `no`
>
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Why do we need to use MGF instead of RAW files for Search GUI?
>
> > <solution-title></solution-title>
> >
> > 1. SearchGUI is compatible only with MGF files, hence you have to use msconvert or Thermofile converter tools to convert the RAW format to MGF fomat.
> >
> {: .solution}
>
{: .question}

## Perform Database searching with **Search GUI**
SearchGUI is a database-searching tool that comprises different search engines to match sample MS/MS spectra to known peptide sequences. In our analysis, we will use X!Tandem and MS-GF+ as search algorithms within SearchGUI for matching spectra from mass spectrometry data against peptides from the protein sequence database.

The SearchGUI tool will perform a database search based on the parameters we've set and will generate a file (called a SearchGUI archive file) that will serve as the input for the PeptideShaker tool. The SearchGUI archive file contains Peptide-Spectral Matches (PSMs), and PeptideShaker is a post-processing software that will assess the confidence of the data. PeptideShaker also infers the identities of proteins based on the matched peptide sequences, and users are able to visualize these outputs to interpret results.  More information about database searching using SearchGUI and PeptideShaker is accessible at [Metaproteomics tutorial](https://gxy.io/GTN:T00221).

> <hands-on-title> Peptide discovery using SearchGUI </hands-on-title>
>
> 1. {% tool [Search GUI](toolshed.g2.bx.psu.edu/repos/galaxyp/peptideshaker/search_gui/4.0.41+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Identification Parameters file"*: `Identification_Parameters_File` (output of **Identification Parameters** {% icon tool %})
>    - {% icon param-file %} *"Fasta file"*: `input_database_concatenated_target_decoy` (output of **FastaCLI** {% icon tool %})
>    - {% icon param-file %} *"Input Peak Lists"*: `output` (output of **msconvert** {% icon tool %})
>    - *"SearchGUI Options"*: `Default`
>
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Why do we need to add decoy sequences to our FASTA database for Search GUI? And how many do we need to add?
>
> > <solution-title></solution-title>
> >
> > 1. Adding decoy sequences helps in FDR estimation, discriminating true positives from false positives, and quality control of the data. The number of decoy sequences you need to add to your database depends on the desired FDR level you want to achieve. A common practice is to use a 1:1 ratio of target sequences to decoy sequences. In other words, for every real protein sequence in your database, you would add a decoy sequence. This allows you to estimate the FDR at 1%, 5%, or any other chosen threshold.
> >
> {: .solution}
>
{: .question}
> <question-title></question-title>
>
> 1. What is the Identification Parameters tool?
>
> > <solution-title></solution-title>
> >
> > 1. Identification Parameters tool is an input required by the search GUI tool, it contains all the parameters required to run the search algorithms.
> >
> {: .solution}
>
{: .question}

## Post-processing of SearchGUI output using with **Peptide Shaker**

> <hands-on-title> Peptide Shaker </hands-on-title>
>
> 1. {% tool [Peptide Shaker](toolshed.g2.bx.psu.edu/repos/galaxyp/peptideshaker/peptide_shaker/2.0.33+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Compressed SearchGUI results"*: `searchgui_results` (output of **Search GUI** {% icon tool %})
>    - In *"Exporting options"*:
>        - *"Follow-up analysis export options"*: `Do not export`
>        - *"Identification features reports to be generated"*: `PSM Report` `Peptide Report` `Protein Report` `Certificate of Analysis`
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. What are the differences between the following reports from PeptideShaker: PSM report, Peptide report, and Protein report?
>
> > <solution-title></solution-title>
> >
> > 1. PSM reports focus on individual peptide-spectrum matches, providing detailed information about each spectrum and its assigned peptide sequence. Peptide reports summarize information about unique peptides and their properties. Protein reports, on the other hand, focus on proteins, including protein inference, grouping, and quantification, making them more suitable for understanding the overall protein composition in a sample. These reports serve different purposes in proteomic data analysis and are used to extract various levels of information from mass spectrometry results.
> >
> {: .solution}
>
{: .question}

##  Using Text Manipulation Tools to Manage Microbial Outputs from SearchGUI/PeptideShaker

> <hands-on-title> Selecting microbial peptides from SearchGUI/PeptideShaker with Select tool </hands-on-title>
>
> 1. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `output_peptides` (output of **Peptide Shaker** {% icon tool %})
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_HUMAN)|(_REVERSED)|(REV_)|(CON)|(con)`
>    - *"Keep header line"*: `Yes`
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. What is the purpose of this step?
>
> > <solution-title></solution-title>
> >
> > 1. This step is to extract microbial peptides or to remove any peptides that match humans, reverse, contaminants, etc.
> >
> {: .solution}
>
{: .question}

> <hands-on-title> Selecting microbial PSMs from SearchGUI/PeptideShaker with Select </hands-on-title>
>
> 1. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `output_psm` (output of **Peptide Shaker** {% icon tool %})
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_HUMAN)|(_REVERSED)|(REV_)|(CON)|(con)`
>    - *"Keep header line"*: `Yes`
>
>
{: .hands_on}


> <hands-on-title> Filtering confident microbial peptides from SGPS  with Filter </hands-on-title>
>
> 1. {% tool [Filter](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `out_file1` (output of **Select** {% icon tool %})
>    - *"With following condition"*: `c17=='Confident'`
>    - *"Number of header lines to skip"*: `1`
>
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. In the Filtering steps, what does “Confidence” mean quantitatively, i.e. what is the percentage cutoff?
>
> > <solution-title></solution-title>
> >
> > 1. The term "Confidence" in the context of proteomic data analysis often refers to a measure of how reliable or trustworthy a particular protein or peptide identification is. However, the specific numerical value or percentage cutoff for confidence can vary depending on the software or approach you are using and the goals of your analysis. In many proteomics studies, researchers use a false discovery rate (FDR) to set a quantitative confidence threshold. Here we have set it as 1%FDR, which means that you're accepting only 1% or less of your reported identifications as likely to be false positives.
> >
> {: .solution}
>
{: .question}


> <hands-on-title> Filtering confident microbial PSMs from SGPS with Filter </hands-on-title>
>
> 1. {% tool [Filter](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `out_file1` (output of **Select** {% icon tool %})
>    - *"With following condition"*: `c24=='Confident'`
>    - *"Number of header lines to skip"*: `1`
>
>
{: .hands_on}


We will generate and merge the Human SwissProt Protein Database and contaminants (cRAP) and convert the resulting FASTA file to a tabular file that will be used in the Query Tabular tool to generate distinct microbial peptides from SearchGUI/PeptideShaker.

> <hands-on-title> Merging Human SwissProt and cRAP databases for Query Tabular with FASTA Merge Files and Filter Unique Sequences </hands-on-title>
>
> 1. {% tool [FASTA Merge Files and Filter Unique Sequences](toolshed.g2.bx.psu.edu/repos/galaxyp/fasta_merge_files_and_filter_unique_sequences/fasta_merge_files_and_filter_unique_sequences/1.2.0) %} with the following parameters:
>    - *"Run in batch mode?"*: `Merge individual FASTAs (output collection if input is collection)`
>        - In *"Input FASTA File(s)"*:
>            - {% icon param-repeat %} *"Insert Input FASTA File(s)"*
>                - {% icon param-file %} *"FASTA File"*: `Human Swissprot Protein Database` (output of **Protein Database Downloader** {% icon tool %})
>                - - {% icon param-file %} *"FASTA File"*: `Contaminants cRAP database` (output of **Protein Database Downloader** {% icon tool %})
>
>
{: .hands_on}


> <hands-on-title> Converting FASTA sequences to TAB-delimited file with FASTA-to-Tabular </hands-on-title>
>
> 1. {% tool [FASTA-to-Tabular](toolshed.g2.bx.psu.edu/repos/devteam/fasta_to_tabular/fasta2tab/1.1.0) %} with the following parameters:
>    - {% icon param-file %} *"Convert these sequences"*: `output` (output of **FASTA Merge Files and Filter Unique Sequences** {% icon tool %})
>
>
{: .hands_on}


> <hands-on-title> Filtering out accession numbers from TAB-delimited file  with Filter Tabular </hands-on-title>
>
> 1. {% tool [Filter Tabular](toolshed.g2.bx.psu.edu/repos/iuc/filter_tabular/filter_tabular/3.3.0) %} with the following parameters:
>    - {% icon param-file %} *"Tabular Dataset to filter"*: `output` (output of **FASTA-to-Tabular** {% icon tool %})
>    - In *"Filter Tabular Input Lines"*:
>        - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>            - *"Filter By"*: `select columns`
>                - *"enter column numbers to keep"*: `1`
>        - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>            - *"Filter By"*: `regex replace value in column`
>                - *"enter column number to replace"*: `1`
>                - *"regex pattern"*: `^[^|]+[|]([^| ]+).*$`
>                - *"replacement expression"*: `\1`
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. What’s the difference between a FASTA and Tabular output?
>
> > <solution-title></solution-title>
> >
> > 1. FASTA Output: Typically used to report identified peptide or protein sequences, which are useful for building or updating sequence databases, for downstream sequence analysis, or for re-searching against the sequences.
> > Tabular Output: Used for presenting various information related to identified peptides or proteins, such as accession numbers, scores, abundance values, and other attributes. Tabular output facilitates data analysis, comparisons, and custom data processing.
> >
> >
> {: .solution}
>
{: .question}


> <hands-on-title> Querying protein accession numbers and peptide sequences of confident microbial PSMs (from SGPS) with Query Tabular </hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.0) %} with the following parameters:
>    - In *"Database Table"*:
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `out_file1` (output of **Filter** {% icon tool %})
>            - In *"Filter Dataset Input"*:
>                - In *"Filter Tabular Input Lines"*:
>                    - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>                        - *"Filter By"*: `skip leading lines`
>                            - *"Skip lines"*: `1`
>                    - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>                        - *"Filter By"*: `prepend a line number column`
>            - In *"Table Options"*:
>                - *"Specify Name for Table"*: `psms`
>                - *"Specify Column Names (comma-separated list)"*: `ln,id,Proteins,Sequence`
>                - *"Only load the columns you have named into database"*: `Yes`
>                - In *"Table Index"*:
>                    - {% icon param-repeat %} *"Insert Table Index"*
>                        - *"Index on Columns"*: `ln`
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `out_file1` (output of **Filter** {% icon tool %})
>            - In *"Filter Dataset Input"*:
>                - In *"Filter Tabular Input Lines"*:
>                    - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>                        - *"Filter By"*: `skip leading lines`
>                            - *"Skip lines"*: `1`
>                    - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>                        - *"Filter By"*: `prepend a line number column`
>                    - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>                        - *"Filter By"*: `normalize list columns, replicates row for each item in list`
>                            - *"enter column numbers to normalize"*: `3`
>            - In *"Table Options"*:
>                - *"Specify Name for Table"*: `prots`
>                - *"Specify Column Names (comma-separated list)"*: `ln,id,prot`
>                - *"Only load the columns you have named into database"*: `Yes`
>                - In *"Table Index"*:
>                    - {% icon param-repeat %} *"Insert Table Index"*
>                        - *"This is a unique index"*: `Yes`
>                        - *"Index on Columns"*: `prot,ln`
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `output` (output of **Filter Tabular** {% icon tool %})
>            - In *"Table Options"*:
>                - *"Specify Name for Table"*: `Uniprot`
>                - *"Specify Column Names (comma-separated list)"*: `prot`
>                - In *"Table Index"*:
>                    - {% icon param-repeat %} *"Insert Table Index"*
>                        - *"Index on Columns"*: `prot`
>    - *"SQL Query to generate tabular output"*: `SELECT id,Proteins,Sequence FROM psms WHERE psms.ln NOT IN (SELECT distinct prots.ln FROM prots JOIN Uniprot ON prots.prot = Uniprot.prot) ORDER BY psms.ln`
>    - *"include query result column headers"*: `Yes`
>
>
{: .hands_on}


> <hands-on-title> Cutting out peptide sequences from Query Tabular with Cut </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c3`
>    - {% icon param-file %} *"From"*: `output` (output of **Query Tabular** {% icon tool %})
>
>
{: .hands_on}


> <hands-on-title> Grouping distinct (unique) peptides from SGPS with Group </hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `out_file1` (output of **Cut** {% icon tool %})
>    - *"Group by column"*: `c1`
>
>
{: .hands_on}


## Perform peptide discovery with **MaxQuant**
MaxQuant is an MS-based proteomics platform that is capable of processing raw data and provides improved mass precision and high precursor mass accuracy (HPMA), which resulted in increased protein identification and more in-depth proteomic analysis. Raw MS/MS spectra will be searched against the reduced MetaNovo-generated database (~21.2k sequences). More information about analysis using MaxQuant is available, including [Label-free data analysis](https://gxy.io/GTN:T00218) and [MaxQuant and MSstats for the analysis of TMT data](https://gxy.io/GTN:T00220).

> <hands-on-title> Peptide discovery using MaxQuant </hands-on-title>
>
> 1. {% tool [MaxQuant](toolshed.g2.bx.psu.edu/repos/galaxyp/maxquant/maxquant/2.0.3.0+galaxy0) %} with the following parameters:
>    - In *"Input Options"*:
>        - {% icon param-file %} *"FASTA files"*: `output` (Input dataset)
>    - In *"Search Options"*:
>        - {% icon param-file %} *"Specify an experimental design template (if needed). For detailed instructions see the help text."*: `output` (Input dataset)
>        - *"minimum peptide length"*: `8`
>        - *"Match between runs"*: `Yes`
>        - *"Maximum peptide length for unspecific searches"*: `50`
>    - In *"Protein quantification"*:
>        - *"Use only unmodified peptides"*: `Yes`
>            - *"Modifications used in protein quantification"*: `Oxidation (M)`
>        - In *"LFQ Options"*:
>            - *"iBAQ (calculates absolute protein abundances by normalizing to copy number and not protein mass)"*: `No`
>    - In *"Parameter Group"*:
>        - {% icon param-repeat %} *"Insert Parameter Group"*
>            - {% icon param-collection %} *"Infiles"*: `output` (Input dataset collection)
>            - *"fixed modifications"*: `Carbamidomethyl (C)`
>            - *"variable modifications"*: `Oxidation (M)`
>            - *"enzyme"*: `Trypsin/P`
>            - *"Quantitation Methods"*: `reporter ion MS2`
>                - *"isobaric labeling"*: `TMT11plex`
>                - *"Filter by PIF"*: `True`
>    - *"Generate PTXQC (proteomics quality control pipeline) report? (experimental setting)"*: `False`
>    - In *"Output Options"*:
>        - *"Select the desired outputs."*: `Protein Groups` `mqpar.xml` `Peptides` `MSMS` `msms scans` `summary` `MaxQuant and PTXQC log` `yaml config file`
>
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. What is the Experimental Design file for MaxQuant?
> 
> > <solution-title></solution-title>
> >
> > 1. In MaxQuant, the **Experimental Design** file is used to specify the experimental conditions, sample groups, and the relationships between different samples in a proteomics experiment. This file is a crucial component of the MaxQuant analysis process because it helps the software correctly organize and analyze the mass spectrometry data. The **Experimental Design** file typically has a ".txt" extension and is a tab-delimited text file. Here's what you might include in an Experimental Design file for MaxQuant: **Sample Names** (You specify the names of each sample in your experiment. These names should be consistent with the naming conventions used in your raw data files.), **Experimental Conditions** (You define the experimental conditions or treatment groups associated with each sample. For example, you might have control and treated groups, and you would assign the appropriate condition to each sample.), **Replicates** (You indicate the replicates for each sample, which is important for assessing the statistical significance of your results. Replicates are typically denoted by numeric values (e.g., "1," "2," "3") or by unique identifiers (e.g., "Replicate A," "Replicate B")), **Labels** (If you're using isobaric labeling methods like TMT (Tandem Mass Tag) or iTRAQ (Isobaric Tags for Relative and Absolute Quantitation), you specify the labels associated with each sample. This is important for quantification.), **Other Metadata** (You can include additional metadata relevant to your experiment, such as the biological source, time points, or any other information that helps describe the samples and experimental conditions.)
> >
> {: .solution}
>
{: .question}


## Using Text Manipulation Tools to Manage Microbial Outputs from MaxQuant

> <hands-on-title> Selecting microbial peptides from MaxQuant with Select </hands-on-title>
>
> 1. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `peptides` (output of **MaxQuant** {% icon tool %})
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_HUMAN)|(_REVERSED)|(REV_)|(CON)|(con)`
>    - *"Keep header line"*: `Yes`
>
>
{: .hands_on}


> <hands-on-title> Cutting out microbial peptide sequences with Cut </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - {% icon param-file %} *"From"*: `out_file1` (output of **Select** {% icon tool %})
>
>
{: .hands_on}


> <hands-on-title> Remove header line from MaxQuant peptide output with Remove beginning </hands-on-title>
>
> 1. {% tool [Remove beginning](Remove beginning1) %} with the following parameters:
>    - {% icon param-file %} *"from"*: `out_file1` (output of **Cut** {% icon tool %})
>
>
{: .hands_on}


> <hands-on-title> Grouping distinct (unique) peptide sequences from MaxQuant with Group </hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `out_file1` (output of **Remove beginning** {% icon tool %})
>    - *"Group by column"*: `c1`
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. How case-sensitive is the Group tool? Can I only group by column values, and not row values?
>
> > <solution-title></solution-title>
> >
> > 1. You can make it case sensitive, by default it is not. The tool here does column grouping only.
> >
> {: .solution}
>
{: .question}


## Process SGPS and MaxQuant peptides to compile one list of unique microbial peptides


> <hands-on-title> Concatenate SGPS and MaxQuant peptides into a singular database with Concatenate datasets </hands-on-title>
>
> 1. {% tool [Concatenate datasets](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/0.1.1) %} with the following parameters:
>    - {% icon param-files %} *"Datasets to concatenate"*: `out_file1` (output of **Group** {% icon tool %}), `out_file1` (output of **Group** {% icon tool %})
>
>
{: .hands_on}


> <hands-on-title> Group the peptides from SGPS and MaxQuant to remove duplicates with Group </hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `out_file1` (output of **Concatenate datasets** {% icon tool %})
>    - *"Group by column"*: `c1`
>
>
{: .hands_on}


# Conclusion
By following this tutorial, you have effectively conducted a search of your MS/MS data against the compact database and successfully retrieved reliable microbial peptides. After identifying these microbial peptides with the assistance of MaxQuant and SearchGUI, the next step is to verify the presence of these peptides. This compiled list of unique peptides will serve as the input for PepQuery to validate the confident identification of microbial peptides with the help of the verification workflow.


topic: microbiome
tutorial: mothur-miseq-sop-short
title: 16S Microbial Analysis with mothur (short)
difficulty: intermediate
time_estimation: 2h
content: {% include topics/microbiome/tutorials/mothur-miseq-sop/content.md short=true %}
tools_json: []
requirements_json: []
tags_json: ["metabarcoding", "16S", "microgalaxy"]
content_hash: 6efb0c6c4c5f1e7bc21f79187177bb02
last_modified: 2026-02-26T21:12:17.324597
zenodo_link: https://doi.org/10.5281/zenodo.800651
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/mothur-miseq-sop-short/tutorial.html


topic: microbiome
tutorial: clinical-mp-4-quantitation
title: Clinical Metaproteomics 4: Quantitation
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["[label-TMT11]"]
content_hash: c177e1b90366ad012fb84648be68fcb3
last_modified: 2026-02-26T21:12:17.465149
zenodo_link: https://doi.org/10.5281/zenodo.10105821
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/clinical-mp-4-quantitation/tutorial.html

content:
The next step of the clinical metaproteomics workflow is the quantification workflow. Running a quantification workflow in proteomics is essential for several critical purposes. It allows researchers to measure and compare the abundance of proteins or peptides in biological samples, offering valuable insights into biomarker discovery, comparative analysis, and differential expression studies. Quantitative proteomics helps reveal the functional roles of proteins, the stoichiometry of protein complexes, and the effects of drugs on protein expression in pharmacological studies. Additionally, it serves as a quality control measure, validating initial protein identifications, and providing data normalization for increased accuracy. Quantitative data are indispensable for hypothesis testing, systems biology, and their clinical relevance in areas such as disease diagnosis, prognosis, and therapeutic decision-making. In summary, the quantitation workflow in proteomics is a cornerstone for deciphering the complexities of protein expression and regulation, facilitating a wide array of biological and clinical applications.

In this current workflow, we perform Quantification using the MaxQuant tool and the output will be interpreted in our next module.

![Quantitation workflow]({% link topics/proteomics/images/clinical-mp/clinical-mp-quantification.JPG %})



> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F10_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F11_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F13_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F15_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/Experimental-Design_Discovery_MaxQuant.tabular
>    https://zenodo.org/records/10105821/files/Quantitation_Database_for_MaxQuant.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Add to each database a tag corresponding to input files.
> 6. Create a dataset of the RAW files.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
{: .hands_on}


# Import Workflow

> <hands-on-title>Running the Workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/proteomics/tutorials/clinical-mp-4-quantitation/workflows/WF4_Quantitation_Workflow.ga" title="Quantitation Workflow" %}
>
> 2. Run **Workflow** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *" Quantitation_Database-For-MaxQuant * "*: `Quantitation_Database_for_MaxQuant.fasta`
>    - {% icon param-file %} *" Experimental-Design Discovery MaxQuant"*: `Experimental-Design_Discovery_MaxQuant.tabular`
>    - {% icon param-file %} *" Input Raw-files"*: `RAW dataset collection`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}


# Peptide quantification

In the [Discovery Module](https://github.com/subinamehta/training-material/blob/main/topics/proteomics/tutorials/clinical-mp-discovery/tutorial.md), we used MaxQuant to identify peptides for verification. Now, we will again use MaxQuant to further quantify the PepQuery-verified peptides, both microbial and human. More information about quantitation using MaxQuant is available, including [Label-free data analysis](https://gxy.io/GTN:T00218) and [MaxQuant and MSstats for the analysis of TMT data](https://gxy.io/GTN:T00220).

The outputs we are most interested in consist of the `MaxQuant Evidence file`, `MaxQuant Protein Groups`, and `MaxQuant Peptides`. The `MaxQuant Peptides` file will allow us to group them to generate a list of quantified microbial peptides.

> <hands-on-title> Quantify verified peptides (from PepQuery2) </hands-on-title>
>
> 1. {% tool [MaxQuant](toolshed.g2.bx.psu.edu/repos/galaxyp/maxquant/maxquant/1.6.17.0+galaxy4) %} with the following parameters:
>    - In *"Input Options"*:
>        - {% icon param-file %} *"FASTA files"*: `Quantitation Database for MaxQuant` (Input dataset)
>    - In *"Search Options"*:
>        - {% icon param-file %} *"Specify an experimental design template (if needed). For detailed                           instructions see the help text."*: `output` (Input dataset)
>        - *"minimum peptide length"*: `8`
>        - *"Match between runs"*: `Yes`
>        - *"Maximum peptide length for unspecific searches"*: `50`
>    - In *"Protein quantification"*:
>        - *"Use only unmodified peptides"*: `Yes`
>            - *"Modifications used in protein quantification"*: `Oxidation (M)`
>        - In *"LFQ Options"*:
>            - *"iBAQ (calculates absolute protein abundances by normalizing to copy number and not protein mass)"*: `No`
>    - In *"Parameter Group"*:
>        - {% icon param-repeat %} *"Insert Parameter Group"*
>            - {% icon param-collection %} *"Infiles"*: `output` (Input dataset collection)
>            - *"fixed modifications"*: `Carbamidomethyl (C)`
>            - *"variable modifications"*: `Oxidation (M)`
>            - *"enzyme"*: `Trypsin/P`
>            - *"Quantitation Methods"*: `reporter ion MS2`
>                - *"isobaric labeling"*: `TMT11plex`
>                - *"Filter by PIF"*: `Yes`
>    - In *"Output Options"*:
>        - *"Select the desired outputs."*: `Protein Groups` `mqpar.xml` `Peptides` `Evidence` `MSMS`
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Why can we switch back to using RAW files for MaxQuant, instead of using MGF files?
>
> > <solution-title></solution-title>
> >
> > 1. MaxQuant prefers RAW format compared to MGF as it has more information compared to MGF.
> >
> {: .solution}
>
{: .question}
> <question-title></question-title>
>
> 1. Previously, we used MaxQuant in the Discovery workflow. Why are we using MaxQuant again, instead of Search GUI/PeptideShaker?
>
> > <solution-title></solution-title>
> >
> > 1. We are using MaxQuant for quantification purposes only. SearchGUI Peptide Shaker doesn't have the capability to perform quantification of peptides or proteins.
> >
> {: .solution}
>
{: .question}

## Using Text Manipulation Tools to Manage MaxQuant Outputs

> <hands-on-title> Select microbial protein groups from MaxQuant with Select </hands-on-title>
>
> 1. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `proteinGroups` (output of **MaxQuant** {% icon tool %})
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_HUMAN)|(_REVERSED)|(REV_)|(CON)|(con)`
>
>
> 2. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `peptides` (output of **MaxQuant** {% icon tool %})
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_HUMAN)|(_REVERSED)|(REV_)|(CON)|(con)`
>
>
> 3. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - {% icon param-file %} *"From"*: `out_file1` (output of **Select** {% icon tool %})
>
>
>
> 4. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - {% icon param-file %} *"From"*: `out_file1` (output of **Select** {% icon tool %})
>
>
{: .hands_on}

## Generating a list of quantified proteins and peptides

> <hands-on-title> Group quantified proteins </hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `out_file1` (output of **Cut** {% icon tool %})
>    - *"Group by column"*: `c1`
>
>
{: .hands_on}
> <hands-on-title> Group quantified peptides </hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `out_file1` (output of **Cut** {% icon tool %})
>    - *"Group by column"*: `c1`
>
>
{: .hands_on}


# Conclusion

In summary, the implementation of a quantitation workflow using MaxQuant represents a significant advancement in quantitative proteomic research. This approach enables precise measurement of protein and peptide abundances, enhancing our ability to unravel the complexities of biological systems. This workflow is instrumental in biomarker discovery, comparative analysis, and understanding differential protein expression by offering detailed insights into quantitative changes across different experimental conditions. Its capacity to generate accurate data supports a wide spectrum of applications, including disease research, drug development, and systems biology investigations. Furthermore, the MaxQuant-based quantitation workflow ensures data quality, enabling reliable and reproducible results. It serves as a vital step for quality control, allowing researchers to draw meaningful conclusions from proteomic experiments confidently.


topic: microbiome
tutorial: clinical-mp-1-database-generation
title: Clinical Metaproteomics 1: Database-Generation
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["[label-TMT11]"]
content_hash: 113e6a7484bc258c821fcfc28ea5ed66
last_modified: 2026-02-26T21:12:17.462483
zenodo_link: https://doi.org/10.5281/zenodo.10105821
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/clinical-mp-1-database-generation/tutorial.html

content:
Metaproteomics is the large-scale characterization of the entire complement of proteins expressed by microbiota. However, metaproteomics analysis of clinical samples is challenged by the presence of abundant human (host) proteins which hampers the confident detection of lower abundant microbial proteins {% cite Batut2018 %} ; {% cite Jagtap2015 %} .

To address this, we used tandem mass spectrometry (MS/MS) and bioinformatics tools on the Galaxy platform to develop a metaproteomics workflow to characterize the metaproteomes of clinical samples. This clinical metaproteomics workflow holds potential for general clinical applications such as potential secondary infections during COVID-19 infection, microbiome changes during cystic fibrosis as well as broad research questions regarding host-microbe interactions.

![Clinical Metaproteomics workflow]({% link topics/proteomics/images/clinical-mp/clinical-mp-overview.JPG %})


The first workflow for the clinical metaproteomics data analysis is the Database generation workflow. The Galaxy-P team has developed a workflow wherein a large database is generated by downloading protein sequences of known disease-causing microorganisms and then generating a compact database from the comprehensive database using the Metanovo tool.


![Database Generation Workflow]({% link topics/proteomics/images/clinical-mp/clinical-mp-database-generation.JPG %})



> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data Upload

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/10105821/files/HUMAN_SwissProt_Protein_Database.fasta
>    https://zenodo.org/records/10105821/files/Species_UniProt_FASTA.fasta
>    https://zenodo.org/records/10105821/files/Contaminants_(cRAP)_Protein_Database.fasta
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F10_9Aug19_Rage_Rep-19-06-08.mgf
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F11_9Aug19_Rage_Rep-19-06-08.mgf
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F13_9Aug19_Rage_Rep-19-06-08.mgf
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F15_9Aug19_Rage_Rep-19-06-08.mgf
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
>
> 5. Optional-Add to each database a tag corresponding to the file name.
> 6. Create a dataset collection of the 4 MGF datasets.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Import Workflow


> <hands-on-title>Running the Workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/proteomics/tutorials/clinical-mp-1-database-generation/workflows/WF1_Database_Generation_Workflow.ga" title="Database Generation" %}
>
>
> 2. Run **Workflow** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *" Input Dataset collection"*: `MGF dataset collection`
>    - {% icon param-file %} *" Species_tabular"*: `Species_tabular.tabular`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}

# Step-by-step analysis

## Download Protein Sequences using taxon names
First, we want to generate a large comprehensive protein sequence database using the UniProt XML Downloader to extract sequences for species of interest. To do so, you will need a tabular file that contains a list of species.

For this tutorial, a literature survey was conducted to obtain 118 taxonomic species of organisms that are commonly associated with the female reproductive tract {% cite AfiuniZadeh2018 %}. This species list was used to generate a protein sequence FASTA database was generated using the UniProt XML Downloader tool within the Galaxy framework. In this tutorial, the Species FASTA database (~3.38 million sequences) has already been provided as input. However, if you have your own list of species of interest as a tabular file (`Your_Species_tabular.tabular`), steps to generate a FASTA file from a tabular file are included:


## Download Species Protein Sequences using UniProt XML downloader with **UniProt**

> <hands-on-title> UniProt XML downloader</hands-on-title>
> 1. {% tool [UniProt](toolshed.g2.bx.psu.edu/repos/galaxyp/uniprotxml_downloader/uniprotxml_downloader/2.3.0) %} with the following parameters:
>    - *"Select"*: `Your_Species_tabular.tabular`
>        - {% icon param-file %} *"Dataset (tab separated) with Taxon ID/Name column"*: `output` (Input dataset)
>        - *"Column with Taxon ID/name"*: `c1`
>    - *"UniProt output format"*: `fasta`
> 2. Rename the output as Species_UniProt_FASTA.fasta
>
>    > <comment-title> UniProt description </comment-title>
>    >
>    > This tool will help download the protein fasta sequences by inputting the taxon names.
>    {: .comment}
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Can we use a higher taxonomy clade than species for the UniProt XML downloader?
> 2. Why are we using the tools separately? Can we run it all together?
> 3. Can we select multiple files together?
> 4. How many FASTA files can be merged at once, i.e. is there a limit on the number/size of files?
>
> > <solution-title></solution-title>
> >
> >
> > 1. Yes, the UniProt XML downloader can also be used for generating a database from Genus, Family, Order, or any other higher taxonomy clade.
> > 2. The tools are run separately to reduce the load on the server and tool. If you have a limited number of taxon names, then you can run it all together.
> > 3. Yes, that certainly can be done. We used one input file at a time to maintain the order of sequences in the database.
> > 4. There is no limit.
> >
> {: .solution}
>
{: .question}


## Merging databases to obtain a large comprehensive database for MetaNovo
Once generated, the Species UniProt database (~3.38 million sequences) will be merged with the Human SwissProt database (reviewed only; ~20.4K sequences) and contaminant (cRAP) sequences database (116 sequences) and filtered to generate the large comprehensive database (~2.59 million sequences). The large comprehensive database will be used to generate a compact database using MetaNovo, which is much more manageable.

> <hands-on-title> Download contaminants with **Protein Database Downloader** </hands-on-title>
>
> 1. {% tool [Protein Database Downloader](toolshed.g2.bx.psu.edu/repos/galaxyp/dbbuilder/dbbuilder/0.3.4) %} with the following parameters:
>    - *"Download from?"*: `cRAP (contaminants)`
> 2. Rename as "Protein Database Contaminants (cRAP)"
>
{: .hands_on}

> <hands-on-title> Human SwissProt (reviewed) database</hands-on-title>
> 1. {% tool [Protein Database Downloader](toolshed.g2.bx.psu.edu/repos/galaxyp/dbbuilder/dbbuilder/0.3.4) %} with the following parameters:
>    - *"Download from?"*: `UniProtKB(reviewed only)`
>        - In *"Taxonomy"*: `Homo sapiens (Human)`
>        - In *"reviewed"*: `UniProtKB/Swiss-Prot (reviewed only)`
>	 - In  *"Proteome Set"*: `Reference Proteome Set`
>	 - In  *"Include isoform data"*: `False`
> 2. Rename as "Protein Database Human SwissProt".
{: .hands_on}

> <question-title></question-title>
>
> 1. How often is the Protein Database Downloader updated?
>
>
> > <solution-title></solution-title>
> >
> >
> > 1. It is updated every 3 months.
> >
> {: .solution}
>
{: .question}


> <hands-on-title> FASTA Merge Files and Filter Unique Sequences </hands-on-title>
>
> 1. {% tool [FASTA Merge Files and Filter Unique Sequences](toolshed.g2.bx.psu.edu/repos/galaxyp/fasta_merge_files_and_filter_unique_sequences/fasta_merge_files_and_filter_unique_sequences/1.2.0) %} with the following parameters:
>    - *"Run in batch mode?"*: `Merge individual FASTAs (output collection if input is collection)`
>        - In *"Input FASTA File(s)"*:
>            - {% icon param-repeat %} *"Insert Input FASTA File(s)"*
>                - {% icon param-file %} *"FASTA File"*: `Protein Database Human SwissProt` (output of **Protein Database Downloader** {% icon tool %})
>                - {% icon param-file %} *"FASTA File"*: `Protein Database Contaminants (cRAP)` (output of **Protein Database Downloader** {% icon tool %})
>                - {% icon param-file %} *"FASTA File"*: `Species_UniProt_FASTA ` (output of **UniProt XML downloader** {% icon tool %})
>
> 2. Rename out as "Human UniProt Microbial Proteins cRAP for MetaNovo".
>
{: .hands_on}

# Reducing Database size

## Metanovo tool generates a compact database from your comprehensive database with **MetaNovo**
Next, the large comprehensive database of ~2.59 million sequences can be reduced using the MetaNovo tool to generate a more manageable database that contains identified proteins.

The compact MetaNovo-generated database (~1.9K sequences) will be merged with Human SwissProt (reviewed only) and contaminants (cRAP) databases to generate the reduced database (~21.2k protein sequences) that will be used for peptide identification (see [Discovery Module tutorial](https://github.com/subinamehta/training-material/blob/main/topics/proteomics/tutorials/clinical-mp-discovery/tutorial.md)).

> <hands-on-title> MetaNovo </hands-on-title>
>
> 1. {% tool [MetaNovo](toolshed.g2.bx.psu.edu/repos/galaxyp/metanovo/metanovo/1.9.4+galaxy4) %} with the following parameters:
>    - *"MGF Input Type"*: `Collection`
>        - {% icon param-collection %} *"MGF Collection"*: `output` (Input dataset collection)
>    - {% icon param-file %} *"FASTA File"*: `output` (output of **FASTA Merge Files and Filter Unique Sequences** {% icon tool %})
>    - In *"Spectrum Matching Parameters"*:
>        - *"Fragment ion mass tolerance"*: `0.01`
>        - *"Enzyme"*: `Trypsin (no P rule)`
>        - *"Fixed modifications as comma separated list"*: `Carbamidomethylation of C` `TMT 10-plex of K` `TMT 10-plex of peptide N-term`
>        - *"Variable modifications as comma separated list"*: `Oxidation of M`
>        - *"Maximal charge to search for"*: `5`
>    - In *"Import Filters"*:
>        - *"The maximal peptide length to consider when importing identification files"*: `50`
> 2. Rename as "MetaNovo Compact Database".
>
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Why are we reducing the size of the database?
> 2. Why is this running TMT10 plex modification when the data is 11-plex?
> 3. Regarding MetaNovo Spectrum Matching parameters, what are the most “important” parameters? Meaning, that if a user wants to reduce or increase the sensitivity/number of output sequences, what should they change?
>
> > <solution-title></solution-title>
> >
> >
> > 1. Reducing the size of the database improves search speed, FDR, and sensitivity.
> > 2. There is no option for 11-plex modifications in Metanovo, hence we use the TMT-10plex.
> > 3. The most important parameters are the tolerance (MS1 and MS2) and any modifications introduced during the processing of the data.
> >
> {: .solution}
>
{: .question}

## Merging databases to obtain reduced MetaNovo database for peptide discovery with **FASTA Merge Files and Filter Unique Sequences**

> <hands-on-title> FASTA Merge Files and Filter Unique Sequences</hands-on-title>
>
> 1. {% tool [FASTA Merge Files and Filter Unique Sequences](toolshed.g2.bx.psu.edu/repos/galaxyp/fasta_merge_files_and_filter_unique_sequences/fasta_merge_files_and_filter_unique_sequences/1.2.0) %} with the following parameters:
>    - *"Run in batch mode?"*: `Merge individual FASTAs (output collection if input is collection)`
>        - In *"Input FASTA File(s)"*:
>            - {% icon param-repeat %} *"Insert Input FASTA File(s)"*
>                - {% icon param-file %} *"FASTA File"*: `Protein Database Human SwissProt` (output of **Protein Database Downloader** {% icon tool %})
>                - {% icon param-file %} *"FASTA File"*: `Protein Database Contaminants (cRAP)` (output of **Protein Database Downloader** {% icon tool %})
>                - {% icon param-file %} *"FASTA File"*: `MetaNovo Compact Database` (output of **MetaNovo** {% icon tool %})
>
>
{: .hands_on}

# Conclusion

The first step for the Clinical Metaproteomics study is database generation. As we didn’t have a reference database or information from 16srRNA-seq data, we generated a fasta database doing a literature survey, however, if 16S rRNA data is present, the taxon identified can be used for a customized database generation. As the size of the comprehensive database is generally  too large, we used the Metanovo tool to reduce the size of the database. This reduced database will be then used for clinical metaproteomics discovery workflow.


