# tutorials

topic: microbiome
tutorial: metasbt
title: Indexing and profiling microbes with MetaSBT
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: []
tags_json: ["metagenomics", "diversity", "taxonomic profiling"]
content_hash: ad8b761840067ff33ab4770dac54c6ff
last_modified: 2026-05-27T09:01:49.359047
zenodo_link: https://zenodo.org/records/15882806
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/metasbt/tutorial.html

content:
## Introduction

The ever-increasing volume of sequenced viral genomes, from global surveillance efforts and metagenomic studies, presents a significant computational challenge. How can we efficiently catalog all known viral genomes and then quickly determine the identity of a newly sequenced one?

**MetaSBT** is a powerful computational method designed to address this challenge. It uses a data structure called a **Sequence Bloom Tree (SBT)**, which is a tree-like index where each leaf contains a **Bloom filter**. A Bloom filter is a space-efficient probabilistic data structure that allows for rapid checking of whether an element (in this case, a k-mer from a genome) is part of a set. By organizing these filters into a tree, MetaSBT can quickly search across thousands of genomes at once.

This approach gives MetaSBT several key advantages:
- **Speed:** It can query massive databases much faster than traditional alignment-based methods.
- **Low Memory Footprint:** It uses significantly less RAM than other methods, making it accessible on standard hardware.
- **Novelty Detection:** It can distinguish k-mers from a query genome that are present in the database from those that are new or unknown.

In this tutorial, we will learn how to use the MetaSBT suite in Galaxy to perform three key tasks with viral genomes:

1.  **Create** a new MetaSBT database from a small set of reference viruses.
2.  **Update** this database with new viruses.
3.  **Profile** a query virus against our database to determine its identity and assess its novelty.

Let's get started!

> <hands-on-title>Prepare the History</hands-on-title>
>
> 1.  Create a new history for this tutorial and give it a name like "MetaSBT Viral Profiling".
> 2.  Import the 29 viral reference genomes for this tutorial belonging to 5 different viral species. Open the Galaxy {% icon galaxy-upload %} **Upload Manager** and choose **Paste/Fetch data**. Paste the following URLs into the text box:
>
>     ```
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/882/815/GCA_000882815.1_ViralProj36615/GCA_000882815.1_ViralProj36615_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/002/366/285/GCA_002366285.1_ViralProj411812/GCA_002366285.1_ViralProj411812_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/004/787/195/GCA_004787195.1_ASM478719v1/GCA_004787195.1_ASM478719v1_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/004/787/735/GCA_004787735.1_ASM478773v1/GCA_004787735.1_ASM478773v1_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/004/788/155/GCA_004788155.1_ASM478815v1/GCA_004788155.1_ASM478815v1_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/864/885/GCA_000864885.1_ViralProj15500/GCA_000864885.1_ViralProj15500_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/009/858/895/GCA_009858895.3_ASM985889v3/GCA_009858895.3_ASM985889v3_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/009/937/885/GCA_009937885.1_ASM993788v1/GCA_009937885.1_ASM993788v1_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/009/937/895/GCA_009937895.1_ASM993789v1/GCA_009937895.1_ASM993789v1_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/009/937/905/GCA_009937905.1_ASM993790v1/GCA_009937905.1_ASM993790v1_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/857/045/GCA_000857045.1_ViralProj15142/GCA_000857045.1_ViralProj15142_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/006/458/325/GCA_006458325.1_ASM645832v1/GCA_006458325.1_ASM645832v1_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/006/458/385/GCA_006458385.1_ASM645838v1/GCA_006458385.1_ASM645838v1_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/006/458/425/GCA_006458425.1_ASM645842v1/GCA_006458425.1_ASM645842v1_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/006/458/665/GCA_006458665.1_ASM645866v1/GCA_006458665.1_ASM645866v1_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/851/145/GCA_000851145.1_ViralMultiSegProj14892/GCA_000851145.1_ViralMultiSegProj14892_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/864/105/GCA_000864105.1_ViralMultiSegProj15617/GCA_000864105.1_ViralMultiSegProj15617_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/865/085/GCA_000865085.1_ViralMultiSegProj15622/GCA_000865085.1_ViralMultiSegProj15622_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/865/725/GCA_000865725.1_ViralMultiSegProj15521/GCA_000865725.1_ViralMultiSegProj15521_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/866/645/GCA_000866645.1_ViralMultiSegProj15620/GCA_000866645.1_ViralMultiSegProj15620_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/862/125/GCA_000862125.1_ViralProj15306/GCA_000862125.1_ViralProj15306_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/865/065/GCA_000865065.1_ViralProj15599/GCA_000865065.1_ViralProj15599_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/866/625/GCA_000866625.1_ViralProj15598/GCA_000866625.1_ViralProj15598_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/871/845/GCA_000871845.1_ViralProj20183/GCA_000871845.1_ViralProj20183_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/004/786/575/GCA_004786575.1_ASM478657v1/GCA_004786575.1_ASM478657v1_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/029/745/045/GCA_029745045.1_ASM2974504v1/GCA_029745045.1_ASM2974504v1_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/029/745/055/GCA_029745055.1_ASM2974505v1/GCA_029745055.1_ASM2974505v1_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/029/745/065/GCA_029745065.1_ASM2974506v1/GCA_029745065.1_ASM2974506v1_genomic.fna.gz
>     https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/029/744/035/GCA_029744035.1_ASM2974403v1/GCA_029744035.1_ASM2974403v1_genomic.fna.gz
>     ```
>
> 3.  Also import the table from the [Zenodo record](https://zenodo.org/records/15882806):
>
>     ```
>     https://zenodo.org/record/15882806/files/taxonomies.tsv
>     ```
>
> 4.  Click **Start** and then **Close** the upload manager.
> 5.  Once uploaded, you will have 30 items in your history. The table retrieved at the previous step contains a mapping between the name of the genomes and their complete taxonomic labels. Note that four genomes are missing from this table. Don't worry, we are going to use them later for updating our database and demonstrate how to profile new genomes:
>     - `GCA_000882815.1_ViralProj36615_genomic` -> `k__Viruses|p__Kitrinoviricota|c__Flasuviricetes|o__Amarillovirales|f__Flaviviridae|g__Flavivirus|s__Zika_virus`
>     - `GCA_002366285.1_ViralProj411812_genomic` -> `k__Viruses|p__Kitrinoviricota|c__Flasuviricetes|o__Amarillovirales|f__Flaviviridae|g__Flavivirus|s__Zika_virus`
>     - `GCA_004787195.1_ASM478719v1_genomic` -> `k__Viruses|p__Kitrinoviricota|c__Flasuviricetes|o__Amarillovirales|f__Flaviviridae|g__Flavivirus|s__Zika_virus`
>     - `GCA_004787735.1_ASM478773v1_genomic` -> `k__Viruses|p__Kitrinoviricota|c__Flasuviricetes|o__Amarillovirales|f__Flaviviridae|g__Flavivirus|s__Zika_virus`
>     - `GCA_004788155.1_ASM478815v1_genomic` -> `k__Viruses|p__Kitrinoviricota|c__Flasuviricetes|o__Amarillovirales|f__Flaviviridae|g__Flavivirus|s__Zika_virus`
>     - `GCA_000864885.1_ViralProj15500_genomic` -> `k__Viruses|p__Pisuviricota|c__Pisoniviricetes|o__Nidovirales|f__Coronaviridae|g__Betacoronavirus|s__Severe_acute_respiratory_syndrome_related_coronavirus`
>     - `GCA_009858895.3_ASM985889v3_genomic` -> `k__Viruses|p__Pisuviricota|c__Pisoniviricetes|o__Nidovirales|f__Coronaviridae|g__Betacoronavirus|s__Severe_acute_respiratory_syndrome_related_coronavirus`
>     - `GCA_009937885.1_ASM993788v1_genomic` -> `k__Viruses|p__Pisuviricota|c__Pisoniviricetes|o__Nidovirales|f__Coronaviridae|g__Betacoronavirus|s__Severe_acute_respiratory_syndrome_related_coronavirus`
>     - `GCA_009937895.1_ASM993789v1_genomic` -> `k__Viruses|p__Pisuviricota|c__Pisoniviricetes|o__Nidovirales|f__Coronaviridae|g__Betacoronavirus|s__Severe_acute_respiratory_syndrome_related_coronavirus`
>     - `GCA_009937905.1_ASM993790v1_genomic` -> `k__Viruses|p__Pisuviricota|c__Pisoniviricetes|o__Nidovirales|f__Coronaviridae|g__Betacoronavirus|s__Severe_acute_respiratory_syndrome_related_coronavirus`
>     - `GCA_000857045.1_ViralProj15142_genomic` -> `k__Viruses|p__Nucleocytoviricota|c__Pokkesviricetes|o__Chitovirales|f__Poxviridae|g__Orthopoxvirus|s__Monkeypox_virus`
>     - `GCA_006458325.1_ASM645832v1_genomic` -> `k__Viruses|p__Nucleocytoviricota|c__Pokkesviricetes|o__Chitovirales|f__Poxviridae|g__Orthopoxvirus|s__Monkeypox_virus`
>     - `GCA_006458385.1_ASM645838v1_genomic` -> `k__Viruses|p__Nucleocytoviricota|c__Pokkesviricetes|o__Chitovirales|f__Poxviridae|g__Orthopoxvirus|s__Monkeypox_virus`
>     - `GCA_006458425.1_ASM645842v1_genomic` -> `k__Viruses|p__Nucleocytoviricota|c__Pokkesviricetes|o__Chitovirales|f__Poxviridae|g__Orthopoxvirus|s__Monkeypox_virus`
>     - `GCA_006458665.1_ASM645866v1_genomic` -> `k__Viruses|p__Nucleocytoviricota|c__Pokkesviricetes|o__Chitovirales|f__Poxviridae|g__Orthopoxvirus|s__Monkeypox_virus`
>     - `GCA_000851145.1_ViralMultiSegProj14892_genomic` -> `k__Viruses|p__Negarnaviricota|c__Insthoviricetes|o__Articulavirales|f__Orthomyxoviridae|g__Alphainfluenzavirus|s__Influenza_A_virus`
>     - `GCA_000864105.1_ViralMultiSegProj15617_genomic` -> `k__Viruses|p__Negarnaviricota|c__Insthoviricetes|o__Articulavirales|f__Orthomyxoviridae|g__Alphainfluenzavirus|s__Influenza_A_virus`
>     - `GCA_000865085.1_ViralMultiSegProj15622_genomic` -> `k__Viruses|p__Negarnaviricota|c__Insthoviricetes|o__Articulavirales|f__Orthomyxoviridae|g__Alphainfluenzavirus|s__Influenza_A_virus`
>     - `GCA_000865725.1_ViralMultiSegProj15521_genomic` -> `k__Viruses|p__Negarnaviricota|c__Insthoviricetes|o__Articulavirales|f__Orthomyxoviridae|g__Alphainfluenzavirus|s__Influenza_A_virus`
>     - `GCA_000866645.1_ViralMultiSegProj15620_genomic` -> `k__Viruses|p__Negarnaviricota|c__Insthoviricetes|o__Articulavirales|f__Orthomyxoviridae|g__Alphainfluenzavirus|s__Influenza_A_virus`
>     - `GCA_000862125.1_ViralProj15306_genomic` -> `k__Viruses|p__Kitrinoviricota|c__Flasuviricetes|o__Amarillovirales|f__Flaviviridae|g__Flavivirus|s__Dengue_virus`
>     - `GCA_000865065.1_ViralProj15599_genomic` -> `k__Viruses|p__Kitrinoviricota|c__Flasuviricetes|o__Amarillovirales|f__Flaviviridae|g__Flavivirus|s__Dengue_virus`
>     - `GCA_000866625.1_ViralProj15598_genomic` -> `k__Viruses|p__Kitrinoviricota|c__Flasuviricetes|o__Amarillovirales|f__Flaviviridae|g__Flavivirus|s__Dengue_virus`
>     - `GCA_000871845.1_ViralProj20183_genomic` -> `k__Viruses|p__Kitrinoviricota|c__Flasuviricetes|o__Amarillovirales|f__Flaviviridae|g__Flavivirus|s__Dengue_virus`
>     - `GCA_004786575.1_ASM478657v1_genomic` -> `k__Viruses|p__Kitrinoviricota|c__Flasuviricetes|o__Amarillovirales|f__Flaviviridae|g__Flavivirus|s__Dengue_virus`
>
> > <tip-title>Using Pre-built Databases</tip-title>
> >
> > Building a comprehensive database from all NCBI viral reference genomes can be time-consuming. For many applications, you can directly use a pre-built database. Many Galaxy servers provide access to these via a **CVMFS (CernVM File System) data directory**.
> >
> > For example, you might find pre-built MetaSBT databases at a path like:
> > `/cvmfs/data.galaxyproject.org/byhand/MetaSBT/`
> >
> > You can find more information about [how Galaxy uses CVMFS for reference data]({% link topics/admin/tutorials/cvmfs/tutorial.md %}).
> >
> {: .tip}
>
{: .hands_on}

---

## Part 1: Creating a MetaSBT Database from Scratch

Our first step is to build a small, custom database using our reference viruses.

> <hands-on-title>Build the Initial Index</hands-on-title>
>
> 1.  Run the {% icon tool %} **metasbt_index** tool with the following parameters:
>     - {% icon param-collection %} *"Input genomes"*: Select all the reference genomes previously imported into the history `*.fna.gz`.
>     - {% icon param-file %} *"Input table with taxonomic labels"*: Select the table with the mapping between the name of the genomes and their complete taxonomic labels.
>     - Under the *"Advanced options"* section, {% icon param-file %} *"MetaSBT database"* is automatically set to *"Build your own MetaSBT database from scratch"*. This gives us access to a set of advanced options for the estimation of the best k-mer length and the estimation of a proper Bloom filter size according to our specific set of input genomes, and a quality control filter to get rid of genomes that do not satisfy certain quality criteria based on a completeness and contamination threshold. For the specific purpose of this tutorial, we should set the following options:
>         - *"K-mer length"* > *"Set a k-mer length"*: `9`
>         - *"Bloom filter size"* > *"Set a bloom filter size"*: `10000`
>
> 2.  Click **Execute**. The tool will produce three output files:
>     - A `clusters` table with the list of clusters defined by MetaSBT according to the taxonomic organization of the input genomes, together with the number of genomes in each cluster and the density of the bloom filters.
>     - A `genomes` table with the list of genomes in the MetaSBT clusters and their assigned taxonomic labels.
>     - A `database` compressed tarball representing the actual MetaSBT database.
>
> > <question-title></question-title>
> >
> > 1. What does the k-mer size parameter represent?
> > 2. Why did we compress the output database tarball?
> > 3. There are three genomes missing in the database. Why?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. The k-mer size is the length of the short DNA/RNA sequences (k-mers) that are extracted from the genomes and stored in the Bloom filters.
> > > 2. The MetaSBT database is actually a directory containing multiple files. Compressing it into a single `.tar.gz` file makes it much easier to manage and pass to other tools or share it with other users within Galaxy.
> > > 3. Although we selected the whole set of genomes in our history `*.fna.gz`, the input table with the mapping between the name of the genomes and their complete taxonomic labels does not report any information about four genomes. Thus, they are automatically excluded. This has been done on purpose specifically for this tutorial. We are now going to use these three of these missing genomes for updating the database!
> > >
> > {: .solution}
> >
> {: .question}
>
{: .hands_on}

We now have a database containing the genomic information for a specific selection of viral species.

---

## Part 2: Updating an Existing MetaSBT Database

Viral databases are rarely static. New strains and species are constantly being discovered. Instead of rebuilding our index from scratch every time, we can simply update it.

We will now add three more *Monkeypox virus* genomes to the database we just created.

> <hands-on-title>Update the Database</hands-on-title>
>
> 1.  Run the {% icon tool %} **metasbt_index** tool again with these parameters:
>     - {% icon param-collection %} *"Input genomes"*: Select `GCA_029745045.1_ASM2974504v1_genomic.fna.gz`, `GCA_029745055.1_ASM2974505v1_genomic.fna.gz`, and `GCA_029745065.1_ASM2974506v1_genomic.fna.gz`.
>     - Under the *"Advanced options"* section, *"MetaSBT database"*: *"Update a MetaSBT database"*.
>     - *"Will you use a MetaSBT database from your history or a public database?"*: *"Use a database from the history"*.
>     - {% icon param-file %} *"Select a MetaSBT database"*: `database`.
>
> 2.  Click **Execute**. This will produce a *new* MetaSBT database, thus, three new `clusters`, `genomes`, and `database` files will appear in your history. These three new genomes have been profiled against our database and assigned to the closest species cluster, i.e., *Monkeypox virus* in this specific case.
>
> > <question-title></question-title>
> >
> > 1. Why is updating faster?
> > 2. Why didn't we specify the k-mer length nor the Bloom filter size?
> >
> > > <solution-title></solution-title>
> > > 1. When updating, MetaSBT only needs to process the new genomes and insert them into the existing tree structure. It doesn't need to re-process the genomes that are already in the index. For very large databases, this can save hours or even days of computation time.
> > > 2. In case of an update, the way we build the Bloom filter representation of the new genomes must be consistent with how we previously built our database. Thus, this information is implicitly inherited from the selected MetaSBT database.
> > >
> > {: .solution}
> >
> {: .question}
>
{: .hands_on}

---

## Part 3: Profiling a Query Virus Against the Database

Now we have a database containing five known viral species, and we have a new "query" genome whose identity we want to determine. Let's use MetaSBT to find out what our query virus is and how it relates to the viruses in our database.

Our query genome is `GCA_029744035.1_ASM2974403v1_genomic.fna.gz`. We will profile it against our database.

> <hands-on-title>Profile the Query Virus</hands-on-title>
>
> 1.  Run the {% icon tool %} **metasbt_profile** tool with the following parameters:
>     - {% icon param-collection %} *"Input genomes"*: Select `GCA_029744035.1_ASM2974403v1_genomic.fna.gz`.
>     - *"Database source"*: *"Use a database from the history"*.
>     - {% icon param-file %} *"Select a MetaSBT database tarball"*: Select the **second** `database` file (the one from the update step).
>
> 2.  Click **Execute**. The tool will generate a collection containing a profiling report for each of the input query genomes in tabular format.
>
> 3.  Click the {% icon galaxy-eye %} (eye) icon on the generated report to view its contents.
>
> > <question-title></question-title>
> >
> > The output report has different rows. Why there are multiple matches under the same taxonomic level?
> >
> > > <solution-title></solution-title>
> > >
> > > MetaSBT may report multiple taxonomic units under the same taxonomic level if their distance from the input query genome is below a specific threshold which is established considering the distance to the closest taxonomic unit minus its 20% (by default). This last percentage is called *uncertainty percentage* and can be changed under the *"Advanced options"* section.
> > >
> > {: .solution}
> >
> {: .question}
>
{: .hands_on}

### Analyzing the Results

When you view one of the profiles in the output collection, you should see a report where each row represents a taxonomic level and its corresponding Average Nucleotide Identity (ANI) value. ANI is a measure of genomic similarity between two genomes. Here, the ANI is expressed as a distance measure, so the lower it is, the closer a specific taxonomic unit is to the input genome.

The report is structured into three columns:
- *level*: This column indicates the taxonomic rank (i.e., *kingdom*, *phylum*, *class*, *order*, *family*, *genus*, and *species*).
- *closest*: This column displays the lineage of the closest match found in the database at that specific taxonomic level.
- *ani*: This column shows the ANI distance for the best match found. A lower ANI distance suggests a closer relationship with the input query genome.

This output helps you understand the genomic relatedness of your query to the genomes present in the database at various taxonomic resolutions. You can observe how the ANI changes as you move down the taxonomic hierarchy, providing insights into the closest classification of your genome.

---

## Conclusion

In this tutorial, you have learned the complete workflow for using MetaSBT in Galaxy. You have successfully:
1.  Built a custom database from a selection of viral reference genomes;
2.  Efficiently updated that database with new viruses;
3.  Profiled an unknown virus against the database to determine its identity (or lack thereof);
4.  Interpreted the similarity reports to identify a query virus.

You are now equipped to use MetaSBT for your own research. You can create custom databases for curated sets of genomes, or leverage large, pre-built databases to rapidly identify and characterize newly sequenced genomes from clinical or environmental samples.


topic: microbiome
tutorial: mothur-miseq-sop
title: 16S Microbial Analysis with mothur (extended)
difficulty: intermediate
time_estimation: 6h
content: {% include topics/microbiome/tutorials/mothur-miseq-sop/content.md short=false %}
tools_json: []
requirements_json: []
tags_json: ["metabarcoding", "16S", "microgalaxy"]
content_hash: 1d06ac28023f03d9fc9361067e5e0e14
last_modified: 2026-05-27T09:01:49.375526
zenodo_link: https://doi.org/10.5281/zenodo.800651
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/mothur-miseq-sop/tutorial.html


topic: microbiome
tutorial: metagenomics-assembly
title: Assembly of metagenomic sequencing data
difficulty: introductory
time_estimation: 2H
requirements_json: []
tags_json: ["assembly", "metagenomics", "microgalaxy"]
content_hash: 0f2c5783ec6a46920b4227c792889ec4
last_modified: 2026-05-27T09:01:49.354276
zenodo_link: https://zenodo.org/record/7818827
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/metagenomics-assembly/tutorial.html

content:
Metagenomics involves the extraction, sequencing and analysis of combined genomic DNA from **entire microbiome** samples. It includes then DNA from **many different organisms**, with different taxonomic background.

Reconstructing the genomes of microorganisms in the sampled communities is critical step in analyzing metagenomic data. To do that, we can use **assembly** and assemblers, *i.e.* computational programs that stich together the small fragments of sequenced DNA produced by sequencing instruments.

Assembling seems intuitively similar to putting together a jigsaw puzzle. Essentially, it looks for reads "that work together" or more precisely, reads that overlap. Tasks like this are **not straightforward**, but rather complex because of the complexity of the genomics (specially the repeats), the missing pieces and the errors introduced during sequencing.

> <comment-title></comment-title>
>
> Do you want to learn more about the principles behind single genome assembly? Follow our [tutorials]({% link topics/assembly/index.md %}).
{: .comment}

**Metagenomic assembly** is further complicated by
- the large volume of data produced
- the quality of the sequence
- the unequal representation of members of the microbial community
- the presence of closely related microorganisms with similar genomes
- the presence of several strains of the same microorganism
- an insufficient amount of data for minor community members

> <details-title>Strategies for assembly</details-title>
> For assembly, there are 3 main strategies:
> 1. Greedy extension
> 2. Overlap Layout Consensus
> 3. De Bruijn graphs. The following figure illustrates these strategies in brief.
>
> ![Image shows greedy extention, overlap layout consensus, and de Brujin graphs assembly algorithms.](./images/assembly-algorithms.png "Assembly algorithms. Image from {% cite carpentries %}."){:width="70%"}
>
> The nice paper {% cite miller2010 %} on assemblers based on these algorithms will help you to better understand how they work.
{: .details}

For metagenomic assembly, several tools exist: metaSPAdes ({% cite nurk2017 %}), MEGAHIT ({% cite li2015 %}), etc. The different assemblers have different computational characteristics and their performance varies according to the microbiome as shown in by both rounds of  Critical Assessment of Metagenome Interpretation initiative ({% cite sczyrba2017 %}, {% cite meyer2022 %}, {% cite meyer2021 %}). The preference of one assembler over another depends on the purpose at hand.

In this tutorial, we will learn how to run metagenomic assembly tool and evaluate the quality of the generated assemblies. To do that, we will use data from the study: [Temporal shotgun metagenomic dissection of the coffee fermentation ecosystem](https://www.ebi.ac.uk/metagenomics/studies/MGYS00005630#overview). For an in-depth analysis of the structure and functions of the coffee microbiome, a temporal shotgun metagenomic study (six time points) was performed. The six samples have been sequenced with Illumina MiSeq utilizing whole genome sequencing.

Based on the 6 original dataset of the coffee fermentation system, we generated mock datasets for this tutorial.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare analysis history and data

To run assembly, we first need to get the data into Galaxy. Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Prepare the Galaxy history</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

We need to get the data into our history.

In case of a not very large dataset it's more convenient to upload data directly from your computer to Galaxy.

> <hands-on-title>Upload data into Galaxy</hands-on-title>
>
> 1. Import the sequence read raw data (\*.fastqsanger.gz) from [Zenodo]({{ page.zenodo_link }}) or a data library:
>
>    ```text
>    {{ page.zenodo_link }}/files/ERR2231567_1.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231567_2.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231568_1.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231568_2.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231569_1.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231569_2.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231570_1.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231570_2.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231571_1.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231571_2.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231572_1.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231572_2.fastqsanger.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    > <comment-title></comment-title>
>    > In case of large dataset, we can use FTP server or the [Galaxy Rule-based Uploader]({% link topics/galaxy-interface/tutorials/upload-rules/tutorial.md %}).
>    {: .comment}
>
> 2. Create a paired collection named `Raw reads`, rename your pairs with the sample name
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
{: .hands_on}

# Assembly

As explained before, there are many challenges to metagenomics assembly, including:

1. Differences in coverage between samples, resulting from differences in abundance;
2. The fact that different species often share conserved regions ({%cite kececioglu2001%}), and
3. The presence of multiple strains of a single species ({%cite miller2010%}).

To reduce the differences in coverage between samples, we can use a **co-assembly** approach, where reads from all samples are aligned together:

![Image show one pile of sample1 reads and another pile of sample2 reads, then, green arrow leads to assembled reads from both piles](./images/co-assembly.png "Co-assembly"){:width="60%"}

Pros of co-assembly | Cons of co-assembly
--- | ---
More data | Higher computational overhead
Better/longer assemblies | Risk of shattering your assembly
Access to lower abundant organisms | Risk of increased contamination

Co-assembly is then not always beneficial:
- Changes in strain can cause the assembly graph to collapse
- Binned contigs are likely to be misclassified: MAGs must be treated as a population genome.

![Image shows the process of assembled reads from 2 samples followed by binning and there is detailed information about broken contigs, dominant coverage, chimeric contigs found during assembly process](./images/co-assembly-process.png "Co-assembly process"){:width="80%"}

In these cases, co-assembly is reasonable if:
- Same samples
- Same sampling event
- Longitudinal sampling of the same site
- Related samples

{% snippet faqs/galaxy/fastq_groupmerge.md %}

Examples where co-assembly would be reasonable:
- Repeated sampling of the **same patient** along a particular amount of time.
- Multiple samples taken from the **same site** and **similar environmental conditions**, eg. a patch of soil during the same sampling season.

Examples where co-assembly would NOT be recommended:
- Samples from different patients.
If samples differ as described, **individual assembly** is preferred. In the case of individual assembly, **contigs should be binned** per sample and an extra step of **de-replication** should be used as binning:

If samples differ as described, **individual assembly** is preferred. In the case of individual assembly, **contigs should be binned** per sample and an extra step of **de-replication** should be used after binning:


> <comment-title></comment-title>
> For more information on dereplication, check out the [metagenomic binning tutorial]({% link topics/microbiome/tutorials/metagenomics-binning/tutorial.md %}).
{: .comment}

In this tutorial, to show all steps, we will run an **individual assembly**.

> <comment-title>Why not both?</comment-title>
> It is also possible to run both individual assembly and co-assembly, and this approach can recover MAGs effectively. In this case: individual assembly can recover MAGs with a low amount of contamination, while co-assembly also allows for the recovery of low-abundance MAGs, with the downside of potentially more contamination. Although this approach can be effective, it also requires high computational resources and should be considered carefully.
>
> {% snippet faqs/galaxy/fastq_groupmerge.md %}
>
{: .comment}

As mentioned in the introduction, several tools are available for metagenomic assembly. But 2 are the most used ones:

- **MetaSPAdes** ({%cite nurk2017%}): an short-read assembler designed specifically for large and complex metagenomics datasets.

  MetaSPAdes is part of the SPAdes toolkit, which has several assembly pipelines. Since SPAdes handles non-uniform coverage, it is useful for assembling simple communities, but metaSPAdes also handles other problems, allowing it to assemble complex communities' metagenomes.

  As input for metaSPAdes it can accept short reads. However, there is an option to use additionally long reads besides short reads to produce hybrid input.

- **MEGAHIT** ({% cite li2015 %}): a single node assembler for large and complex metagenomics NGS reads, such as soil

  It makes use of the Succinct de Bruijn Graph (SdBG) approach to achieve low memory assembly.

Both tools are available in Galaxy. But currently, only **MEGAHIT** can be used in individual mode for several samples, as a option in the wrapper.
**MetaSPAdes** performs co-assembly by default if multiple samples are supplied. However, individually assembly can be enforced:

{% snippet faqs/galaxy/collections_force_mapping_over.md %}

# Assembly with MEGAHIT

> <hands-on-title>Individual assembly of short-reads with MEGAHIT</hands-on-title>
> 1.  {% tool [MEGAHIT](toolshed.g2.bx.psu.edu/repos/iuc/megahit/megahit/1.2.9+galaxy2) %} with parameters:
>     - *"Select your input option"*: `Paired-end collection`
>       - *"Run in batch mode?"*: `Run individually`
>
>          > <comment-title></comment-title>
>          > To run as co-assembly, select `Merge all fastq pair-end`, instead of `Run individually`
>          {: .comment}
>
>       - {% icon param-collection %} *"Select a paired collection"*: `Raw reads`
>     - In *Basic assembly options*
>       - *"K-mer specification method"*: `Specify min, max, and step values`
>         - *"Minimum kmer size"*: 21
>         - *"Maximum kmer size"*: 91
>         - *"Increment of kmer size of each iteration"*: 12
>
{: .hands_on}

**MEGAHIT** produced a collection of output assemblies - one per sample - that can be used for the subsequent step of **metagenomic binning**. The output contains **contigs**, contiguous lengths of genomic sequences in which bases are known to a high degree of certainty.

> <comment-title>Scaffolds</comment-title>
> Contrary to **MetaSPAdes**, **MEGAHIT** does not output **scaffolds**. **Scaffolds** are segments of genome sequence reconstructed fron contigs and gaps. The gaps occur when reads from the two sequenced ends of at least one fragment overlap with other reads from two different contigs (as long as the arrangement is otherwise consistent with the contigs being adjacent). It is possible to estimate the number of bases between contigs based on fragment lengths.
{: .comment}

> <comment-title></comment-title>
>
> Since the assembly process would take ~1h we are just going to import the results of the assembly previously run.
>
> > <hands-on-title>Import generated assembly files</hands-on-title>
> >
> > 1. Import the six contig files from [Zenodo]({{ page.zenodo_link }}) or the Shared Data library:
> >
> >    ```text
> >    {{ page.zenodo_link }}/files/contigs_ERR2231567.fasta
> >    {{ page.zenodo_link }}/files/contigs_ERR2231568.fasta
> >    {{ page.zenodo_link }}/files/contigs_ERR2231569.fasta
> >    {{ page.zenodo_link }}/files/contigs_ERR2231570.fasta
> >    {{ page.zenodo_link }}/files/contigs_ERR2231571.fasta
> >    {{ page.zenodo_link }}/files/contigs_ERR2231572.fasta
> >    ```
> >
> >
> > 2. Create a collection named `MEGAHIT Contigs`, rename your pairs with the sample name
> >
> {: .hands_on}
{: .comment}

> <question-title>Contig metrics</question-title>
>
> 1. How many contigs have been assembled for sample ERR2231568 ?
> 2. And how many have been assembled for ERR2231572?
> 3. What is the minimum length of the contigs?
>
> > <solution-title></solution-title>
> >
> > 1. There are 228,719 sequences in the file so 228,719 contigs
> > 2. 122,526 contigs
> > 3. Sequences seems bigger than 200 bp (`len` attribute of the sequence information in Fasta files). It is the default value set up in MEGAHIT.
> >
> {: .solution}
>
{: .question}

> <details-title>Co-assembly with MetaSPAdes</details-title>
> MetaSPAdes supports co-assembly by passing a list of paired-end read files.
>
> > <hands-on-title>Assembly with MetaSPAdes</hands-on-title>
> > 1. {% tool [MetaSPAdes](toolshed.g2.bx.psu.edu/repos/nml/metaspades/metaspades/4.2.0+galaxy0) %} with the following parameters:
>     - *"Pair-end reads input format"*: `Paired-end: list of dataset pairs`
> >      - {% icon param-collection %} *"FASTQ file(s): collection"*: `Raw reads`
> >
> >    > <comment-title>Tip</comment-title>
> >    > To run **individual assembly** instead, follow the previous FAQ to create a **nested list of samples**.
> >     {: .comment}
> >
> >   - *"Select k-mer detection option"*: `User specific`
> >    - *"K-mer size values"*: `21,33,55,77`
> {: .hands_on}
{: .details}

# Quality control of assembly

Once assembly is done, it is important to check its quality.

Assemblies can be evaluated with **metaQUAST** ({%cite mikheenko2016%}), the metagenomics mode of **QUAST** ({%cite gurevich2013%}).

> <hands-on-title>Evaluation of assembly quality with metaQUAST</hands-on-title>
>
> 1. {% tool [QUAST](toolshed.g2.bx.psu.edu/repos/iuc/quast/quast/5.2.0+galaxy1) %} with parameters:
>     - *"Assembly mode?"*: `Individual assembly (1 contig file per samples)`
>       - *"Use customized names for the input files?"*: `No, use dataset names`
>         - {% icon param-collection %} *"Contigs/scaffolds file"*: output **MEGAHIT**
>       - *"Reads options"*: `Illumina paired-end reads in paired collection`
>
>       > <comment-title>Tip</comment-title>
>       > To speed up the processing, you may choose `Disabled` here. In this case, raw reads will **not** be mapped back to the assembly to compute metrics such as read coverage.
>       {: .comment}
>
>       - {% icon param-collection %} *"FASTQ/FASTA files"*: `Raw reads`
>     - *"Type of assembly"*: `Metagenome`
>     - *"Output files"*: `HTML report`, `PDF report`, `Tabular reports`, `Log file`, `Key metric summary (metagenome mode)`, `Krona charts (metagenome mode without reference genomes)`
>
> 2. Inspect the HTML reports:
> ![Screenshot of QUAST HTML report](./images/quast_html_report.png)
{: .hands_on}


> <tip-title>Preview MetaQUAST results</tip-title>
>
> 1. Since the QUAST process takes some time, we also provide an example history here:
>
>    - [Galaxy EU](https://usegalaxy.eu/histories/view?id=11ac94870d0bb33aeff2ed936aab8446)
>
{: .tip}

QUAST main output are HTML reports which aggregate different metrics.

## Assembly statistics

On the top of each report is a table with in rows statistics for contigs larger than 500 bp for the different sample assemblies (columns). Let's now look at the table and go from top to bottom:

1. **Genome statistics**

    - **Genome fraction (%)**: percentage of aligned bases in the reference genome

      A base in the reference genome is counted as aligned if at least one contig has at least one alignment to this base.

      We did not provide any reference genome, but QUAST tries to identify the genome content of the metagenome by aligning contigs to [SILVA](https://www.arb-silva.de/) 16S rRNA database. For each assembly, 50 reference genomes with top scores are chosen. The full reference genomes of the identified organisms are downloaded from NCBI to map the assemblies on them and compute the genome fractions.

      > <comment-title>Metagenome reference</comment-title>
      > The alignment to automatically downloaded genomes for metagenomes is rather ambiguous and time-consuming. Most large-scale pipelines skip this step and set the **Maximum number of reference genomes (per each assembly) to download after searching in the SILVA database*** option to `0`.
      {: .comment}

      For each identified genomes, the genome fraction is given when clicking on **Genome fraction (%)**

      > <question-title></question-title>
      >
      > 1. What is the genome fraction for ERR2231568? And for ERR2231572?
      > 2. Which reference genome has the highest genome fraction for ERR2231568? And for ERR2231572?
      >
      > > <solution-title></solution-title>
      > >
      > > 1. The genome fraction is 29.429% for ERR2231568 and 60.463% for ERR2231572
      > > 2. The highest genome fraction was found for *Leuconostoc pseudomesenteroides* for ERR2231568 (84.2%) and for *Lactobacillus vaccinostercus* for ERR2231572 (92.2%). The genomes of *Leuconostoc pseudomesenteroides* and *Lactobacillus vaccinostercus* could be then almost completely recovered from the assemblies of ERR2231568 and ERR2231572 respectively.
      > >
      > {: .solution}
      >
      {: .question}

    - **Duplication ratio**: total number of aligned bases / genome fraction * reference length

      If an assembly contains many contigs that cover the same regions of the reference, the duplication ratio may be much larger than 1.

      > <question-title></question-title>
      >
      > 1. What is the duplication ratio for ERR2231568? And for ERR2231572?
      > 2. Which reference genome has the highest duplication ratio for ERR2231568? And for ERR2231572?
      >
      > > <solution-title></solution-title>
      > >
      > > 1. The duplication ratio is 1.062 for ERR2231568 and 1.112 for ERR2231572 (column ERR2231572 in ERR2231572 report)
      > > 2. The highest duplication ratio was found for *Gluconobacter oxydans* for ERR2231568 (1.156) and for *Lactobacillus pseudomesenteroides* for ERR2231572 (1.204).
      > >
      > {: .solution}
      >
      {: .question}

2. **Read mapping**: results of the mapping of the raw reads on the different assemblies (only if the *"Reads options"* is not disabled)

    > <question-title></question-title>
    >
    > 1. What is the % of read mapped for ERR2231568 assembly to ERR2231568 raw reads? And for ERR2231572 assembly to ERR2231572 raw reads?
    > 2. What is the percentage of reads used to build the assemblies for ERR2231568? and ERR2231572?
    >
    > > <solution-title></solution-title>
    > >
    > > 1. 79.47% of ERR2231568 raw reads were mapped to ERR2231568 assembly and 86.98% of ERR2231572 raw reads to ERR2231572 assembly.
    > > 2. 79.47% of reads were used to the assemblies for ERR2231568 and  86.98% for ERR2231572.
    > {: .solution}
    >
    {: .question}

    > <details-title>Alternative ways to compute coverage</details-title>
    >
    > There are 2 alternative ways to compute coverage:
    > 1. Use [CoverM](https://github.com/wwood/CoverM), which is available in Galaxy
    >
    >    > <hands-on-title>Calculate coverage using CoverM</hands-on-title>
    >    >
    >    > 1. {% tool [CoverM contig](toolshed.g2.bx.psu.edu/repos/iuc/coverm_contig/coverm_contig/0.7.0+galaxy0) %} with parameters:
    >    >    - *"Have the reads already been mapped to contigs?"*: `No`
    >    >       - *"Assembly mode?"*: `Individual assembly (1 contig file per sample)`
    >    >          - *"Read type"*: `Paired collection`
    >    >             - {% icon param-collection %} *"Collection of paired-end FASTA/Q files(s) for mapping"*: `Raw reads`
    >    >    - {% icon param-collection %} *"Contigs"*: output of MEGAGIT
    >    >
    >    > 2. Open the {% icon dataset-info %} "Dataset details" for ERR2231568
    >    > 3. Expand the `Tool Standard Error`
    >    {: .hands_on}
    >
    >    > <question-title></question-title>
    >    >
    >    > 1. What is the overall alignment rate for ERR2231568
    >    > 2. What is the percentage of reads used in assemblies for ERR2231568?
    >    > 3. Are the results similar to the ones in QUAST report?
    >    >
    >    > > <solution-title></solution-title>
    >    > >
    >    > > 1. The overall alignment rate for ERR2231567 is 75.48%
    >    > > 2. 75.48% of the reads were used in assemblies for ERR2231567 and 73.67% for ERR2231571.
    >    > > 3. Slighlty lower results that the ones found by QUAST (79.47% for ERR2231568)
    >    > >
    >    > {: .solution}
    >    >
    >    {: .question}
    >
    > 2. Map the original reads onto contigs and extract the percentage of mapped reads:
    >
    >    > <hands-on-title>Computation of the % reads used in assemblies</hands-on-title>
    >    >
    >    > 1. {% tool [Bowtie2](toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.3+galaxy1) %} with the following parameters:
    >    >    - *"Is this single or paired library"*: `Paired-end Dataset Collection`
    >    >      - {% icon param-collection %} *"FASTQ Paired Dataset"*: `Raw reads`
    >    >    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from the history and build index`
    >    >      - {% icon param-collection %} *"Select reference genome"*: MEGAHIT output
    >    >    - *"Save the Bowtie2 mapping statistics to the history"*: `Yes`
    >    > 3. Inspect the mapping statistics output
    >    {: .hands_on}
    >
    >    > <question-title></question-title>
    >    >
    >    > 1. What is the overall alignment rate for ERR2231568? and ERR2231572?
    >    > 2. What is the percentage of reads used in assemblies for ERR2231568? and ERR2231571?
    >    > 3. Are the results similar to the ones in QUAST report?
    >    >
    >    > > <solution-title></solution-title>
    >    > >
    >    > > 1. The overall alignment rate for ERR2231568 is 65.67% and 78.42% for ERR2231572.
    >    > > 2. 65.97% of the reads were used in assemblies for ERR2231568 and 78.42% for ERR2231572.
    >    > > 3. Lower results that the ones found by QUAST (79.47% for ERR2231568, 86.98%  for ERR2231572)
    >    > >
    >    > {: .solution}
    >    >
    >    {: .question}
    {: .details}

3. **Misassemblies**: joining sequences that should not be adjacent.

    QUAST identifies missassemblies by mapping the contigs to the reference genomes of the identified organisms. 3 types of misassemblies can be identified:

    ![Image shows on the top a contig with a blue and a gren parts with white arrows (pointing on the right) on them and below a reference with 2 chromosomes. The 3 types of misassemblies are after schematized. Relocation: the blue and gren parts of the contig are on chr 1 but separated. Inversion: the blue and gren parts of the contig are on chr 1 but separated and with the arrows facing each other. Translocation: the blue part is on chr 1 and gren part on chr 2.](./images/quast_misassemblies.png "Source: <a href="https://quast.sourceforge.net/docs/manual.html#sec3.1.2">QUAST manual</a>"){:width="60%"}

    1. **Relocation** occur based on signal from two mappings of the same contig against the same chromosome, with 2 cases:
        1. the 2 mappings are separated by an unmapped region of at least 1 kbp
        2. they map on the same chromosome with a shared mapping area of at least 1 kbp

        ![Image shows on the top a contig with a blue and a gren parts with white arrows (pointing on the right) on them and below a reference with 2 chromosomes. The 3 types of misassemblies are after schematized. Relocation: the blue and gren parts of the contig are on chr 1 but separated. Inversion: the blue and gren parts of the contig are on chr 1 but separated and with the arrows facing each other. Translocation: the blue part is on chr 1 and gren part on chr 2.](./images/relocation.svg "Source: <a href="https:///blog.pierre.marijon.fr/misassemblies-in-noisy-assemblies/">Yet Another Bioinformatic blog by Pierre Marijon</a>"){:width="80%"}

        > <question-title></question-title>
        >
        > 1. How many relocations has been found for ERR2231568? And for ERR2231572?
        > 2. For which reference genomes are there the most relocation found for ERR2231568? And for ERR2231572?
        >
        > > <solution-title></solution-title>
        > >
        > > 1. 83 for ERR2231568 and 159 for ERR2231572
        > > 2. *Leuconostoc pseudomesenteroides* and *Tatumella morbirosei* for ERR2231568 and *Lactobacillus hordei* for ERR2231572
        > {: .solution}
        >
        {: .question}

    2. **Translocation** occur when a contig has mapped on more than one reference chromosomes

        > <question-title></question-title>
        >
        > 1. How many translocations has been found for ERR2231568? And for ERR2231572?
        > 2. For which reference genomes are there the most translocations found for ERR2231568? And for ERR2231572?
        > 3. What are the interspecies translocations?
        > 4. How many interspecies translocations has been found for ERR2231568? And for ERR2231572?
        >
        > > <solution-title></solution-title>
        > >
        > > 1. 30 for ERR2231568 and 90 for ERR2231572.
        > > 2. *Leuconostoc pseudomesenteroides* for ERR2231568 and *Lactobacillus vaccinostercus* for ERR2231572.
        > > 3. Interspecies translocations are translocations where a contig has mapped on different reference genomes.
        > > 4. 114 for ERR2231568 and 85 for ERR2231572.
        > {: .solution}
        >
        {: .question}

    3. **Inversion** occurs when a contig has two consecutive mappings on the same chromosome but in different strands

        > <question-title></question-title>
        >
        > 1. How many inversion has been found for ERR2231568? And for ERR2231572?
        > 2. For which reference genomes are there the most inversions found for ERR2231568? And for ERR2231572?
        >
        > > <solution-title></solution-title>
        > >
        > > 1. 4 for ERR2231568 and 9 for ERR2231572.
        > > 2. *Tatumella morbirosei* for ERR2231568 and *Lactobacillus plantarum* for ERR2231572.
        > {: .solution}
        >
        {: .question}

4. **Mismatches** or mismatched bases in the contig-reference alignment

    > <question-title></question-title>
    >
    > 1. How many mismatches have been identified for ERR2231568? And for ERR2231572?
    > 2. For which reference genomes are there the most mismatches for ERR2231568? And for ERR2231572?
    >
    > > <solution-title></solution-title>
    > >
    > > 1. 469,292 for ERR2231568 and 279,694 for ERR2231572.
    > > 2. *Pantoea rwandensis* for ERR2231568 and *Leuconostoc pseudomesenteroides* for ERR2231572.
    > {: .solution}
    >
    {: .question}

5. **Statistics without reference**

    - **# contigs**: total number of contigs

        > <question-title></question-title>
        >
        > 1. How many contigs are for ERR2231568? And for ERR2231572?
        > 2. How many sequences are in the output of MEGAHIT for ERR2231568? And for ERR2231572?
        > 3. Why are these numbers different from the number of sequences in the output of MEGAHIT?
        > 4. Which statistics in the metaQUAST report corresponds to number of sequences in the output of MEGAHIT?
        > 5. Which reference genomes have the most contigs ($$\geq$$ 500 bp) in ERR2231568? And in ERR2231572?
        >
        > > <solution-title></solution-title>
        > >
        > > 1. 66,434 contigs for ERR2231568 and 36,112 for ERR2231572.
        > > 2. In the outputs of MEGAHIT, there are 228,719 contigs for ERR2231568 and 122,526 contigs.
        > > 3. The numbers are lower in the metaQUAST results because metaQUAST reports there only the contigs longer than 500bp.
        > > 4. The **# contigs (>= 0 bp)**
        > > 5. Except the non aligned contigs, *Tatumella morbirosei* for ERR2231568 and *Leuconostoc brevis* for ERR2231572.
        > {: .solution}
        >
        {: .question}

    - **Largest contig**: length of the longest contig in the assembly

        > <question-title></question-title>
        >
        > 1. What is the length of the longest contig in ERR2231568? And in ERR2231572?
        > 2. Is the longest contig assigned to a reference genome in ERR2231568? And in ERR2231572?
        >
        > > <solution-title></solution-title>
        > >
        > > 1. 63,871 bp in ERR2231568 and 65,608 for ERR2231572.
        > > 2. It is assigned to *Leuconostoc pseudomesenteroides KCTC 3652* in ERR2231568 and not assigned (`not_aligned`) in ERR2231572.
        > {: .solution}
        >
        {: .question}

    - **N50**: length for which the collection of all contigs of that length or longer covers at least half an assembly

        N50 statistic defines assembly quality in terms of contiguity. If all contigs in an assembly are ordered by length, the N50 is the minimum length of contigs that contains 50% of the assembled bases. For example, an N50 of 10,000 bp means that 50% of the assembled bases are contained in contigs of at least 10,000 bp.

        Another example. Let's consider 9 contigs with the lengths 2, 3, 4, 5, 6, 7, 8, 9, and 10:
        - The sum of the length is 54
        - Half of the sum is 27
        - 10 + 9 + 8 = 27 (half the length of the sequence)
        - N50 = 8, i.e. the size of the contig which, along with the larger contigs, contain half of sequence of a particular genome

        > <question-title></question-title>
        >
        > 1. What is N50 for ERR2231568? And for ERR2231572?
        > 2. What is N90?
        >
        > > <solution-title></solution-title>
        > >
        > > 1. 921 for ERR2231568 and 1,233 for ERR2231572.
        > > 2. N90 is similar to the N50 metric but with 90% of of the sum of the lengths of all contigs
        > {: .solution}
        >
        {: .question}

        When comparing N50 values from different assemblies, the assembly sizes must be the same size in order for N50 to be meaningful.

        Also the N50 alone is not a useful measure to assess the quality of an assembly. For example, the assemblies with the following contig lengths:
        - 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 25, 25, 150, 1500
        - 50, 500, 530, 650

        Both assemblies have the same N50 although one is more contiguous than the other.

    - **L50**: number of contigs equal to or longer than N50

        In other words, L50, for example, is the minimal number of contigs that cover half the assembly.

        If we take the previous example in N50, L50 = 3.

        > <question-title></question-title>
        >
        > 1. What is the L50 for ERR2231568? And for ERR2231572?
        >
        > > <solution-title></solution-title>
        > >
        > > 1. 17,280 for ERR2231568 and 7,496 for ERR2231572.
        > {: .solution}
        >
        {: .question}

## Icarus contig browser

Icarus generates contig size viewer and one or more contig alignment viewers (if reference genome/genomes are provided) that are accessible from the HTML report, by clicking on `View on Icarus contig browser`.

### Contig size viewer

This viewer draws contigs ordered from longest to shortest. Let's inspect this viewer for ERR2231568.

> <question-title></question-title>
>
> Open the Contig size viewer for ERR2231568 and define start as `0` and end as `500000`
>
> ![Image shows on the Icarus Contig Size Viewer for ERR2231568, with a zoom between 0 500000. Below the menu, the contigs are drawn from the longest on the left to the shortest on the right. Each contig is filled with a different color: green for correct, red for misassembled, etc. Below the contigs is dispayed a bar to navigate through the contigs](./images/ERR2231568-contig-size-viewer.png)
>
> 1. What is the color of the first contig? Why?
> 2. What is the red contig?
>
> > <solution-title></solution-title>
> >
> > 1. The first contig is white because >50% of the contigs is unaligned. By clicking on the contig, we see that only a small block is aligned: 223.41 – 223.65 kbp to Leuconostoc_pseudomesenteroides_KCTC_3652_NZ_BMBP01000002.1.
> > 2. The red contig is a missamblied contig: it contains 2 blocks, with a translocation between them.
> {: .solution}
>
{: .question}

Click `Main menu` on the top left to go back to the main Icarus page.

### Contig alignment viewer

If a reference genome is provided, there should be a table on the main Icarus page that looks like:

Genome | # fragments | Length, bp | Mean genome fraction, % | # misassembled blocks
--- | --- | --- | --- | ---
Gluconobacter oxydans H24 | 2 | 3 816 232 | 10.989 | 38
Kosakonia cowanii | 5 | 4 806 998 | 28.224 | 60
Lactococcus lactis subsp. lactis CV56 | 6 | 2 518 737 | 18.940 	23

When clicking on the genome name, the contigs are displayed according to their mapping to the reference genome. The viewer can additionally visualize genes, operons, and read coverage distribution along the genome, if any of those were fed to QUAST.

> <question-title></question-title>
>
> Open the Contig alignment viewer for ERR2231568 and *Leuconostoc pseudomesenteroides* KCTC 3652, the most covered by contigs
>
> ![Image shows on the Icarus alignment size viewer of ERR2231568 contigs for Leuconostoc pseudomesenteroides KCTC 3652. On the top are displayed the contigs aligend on the reference genome, with a zoom. Each contig is filled with a different color: green for correct, red for misassembled, etc. Below the contigs is dispayed a bar to navigate through the contigs. Below is a graph representing the GC percentage and coverage along the reference genome](./images/ERR2231568-contig-alignment-viewer-leuconostoc.png)
>
> 1. How are the organized the contigs on the top?
> 2. What the different colors for the contigs?
> 3. Why is there a big red block on the right?
> 4. What is the graph on the bottom?
>
> > <solution-title></solution-title>
> >
> > 1. The contigs are displayed based on their mapping on the reference genome of *Leuconostoc pseudomesenteroides* KCTC 3652
> > 2. The different colors represent the different status for the contig: correct contigs, correct contigs but with >50% of the contig unaligned, misassembled blocks. unchecked misassembled blocks, ambiguously mapped contigs, alternative blocks of misassembled contigs, etc.
> > 3. The big red block on the right is contig k91_183138 with a misassembly on the left side, and overlap with 2 other contigs
> > 4. The graph on the bottom represents the GC percentage and coverage by contigs along the reference genome
> {: .solution}
>
{: .question}

> <details-title>Visualization of the *de novo* assembly graph</details-title>
>
> Current metagenome assemblers like MEGAHIT and MetaSPAdes use **graphs**, most typically a de Bruijn graph to stich reads together. In an ideal case, the graph would contain one distinct path for each genome of each micro-organisms, but complexities such as repeated sequences usually prevent this.
>
> Assembly graphs contain then **branching structures**: one node may lead into multiple others. **Contigs** correspond to the longest sequences in the graph that can be determined unambiguously. They are the final results of most assembler. But the assembly graph contains more information. It can be useful for finding sections of the graph, such as rRNA, or to try to find parts of a genome.
>
> **Bandage** ({% cite wick2015bandage %}) is a tool creating interactive visualisations of assembly graphs.
>
> > <hands-on-title>Visualization the assembly graph</hands-on-title>
> >
> > 1. {% tool [megahit contig2fastg](toolshed.g2.bx.psu.edu/repos/iuc/megahit_contig2fastg/megahit_contig2fastg/1.1.3+galaxy10) %} with parameters:
> >    - {% icon param-collection %} *"Contig file"*: Output of **MEGAHIT**
> >    - *"K-mer length"*: `91`
> >
> >      > <comment-title></comment-title>
> >      > To get the value, you need to
> >      > 1. Go into the **MEGAHIT** output collection
> >      > 2. Expand one of the contig file by clicking on it in the history
> >      > 3. Check in the dataset peek the name of the contig
> >      > 4. Extract the value after the first `k` in the contig names
> >      {: .comment}
> >
> > 2. {% tool [Bandage Image](toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_image/0.8.1+galaxy4) %} with parameters:
> >    - {% icon param-collection %} *"Graphical Fragment Assembly"*: Output of **megahit contig2fastg**
> >
> > 3. Inspect the generated image for ERR2231571
> {: .hands_on}
>
> ![Image shows the assembly graphs, with longer stretch on the top and many small contigs on the bottom](./images/ERR2231571_graph.jpg "Assembly graph for ERR2231571 sample")
>
> The graph is quite disconnected. On the top, we can see the longer stretches, that includes multiples contigs (each contig having a different color). On the bottom are the shortest stretches or single contigs.
>
> But it is really hard to read or extract any information from the graph. Let's inspect the information about the assembly graph
>
> > <hands-on-title>Visualization the *de novo* assembly graph</hands-on-title>
> >
> > 1. {% tool [Bandage Info](toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_info/0.8.1+galaxy2) %} with parameters:
> >    - {% icon param-collection %} *"Graphical Fragment Assembly"*: Output of **megahit contig2fastg**
> >
> > 2. {% tool [Column join](toolshed.g2.bx.psu.edu/repos/iuc/collection_column_join/collection_column_join/0.0.3) %} with parameters:
> >    - {% icon param-collection %} *"Tabular files"*: Output of **Bandage Info**
> >
> > 3. Inspect the generated output
> {: .hands_on}
>
> > <question-title></question-title>
> >
> > 1. How many nodes are in the graph for ERR2231568? And for ERR2231572? What does they correspond to?
> > 2. How many edges are in the graph for ERR2231568? And for ERR2231572? What is the impact of these numbers in relation to the number of nodes on the graph?
> > 3. How many connected components are there for ERR2231568? And for ERR2231572? What does they correspond to?
> > 4. What is the percentage of dead ends are there for ERR2231568? And for ERR2231572?
> > 5. What are the smallest and larges edge overlaps?
> > 6. What is the largest component? For which sample?
> > 7. What is the shortest node? What does they correspond to?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. There are 228,719 nodes for ERR2231568 and 122,526 for ERR2231572. They correspond to the number of contigs
> > > 2. There are 16,580 edges for ERR2231568 and 13,993 for ERR2231572. There are less edges than nodes in the graph. It means that many nodes/contigs are disconnected
> > > 3. There are 212,598 connected components, i.e. number of regions of the graph which are disconnected from each other, for ERR2231568 and 109,044 for ERR2231572
> > > 4. There are 94.0702% dead ends, i.e. the end of a node not connected to any other nodes, for ERR2231568 and 90.7032% for ERR2231572. It confirms the previous observation
> > > 5. The smallest and larges edge overlaps are 91bp, i.e. the k-mer length
> > > 6. The largest component is 340,003 bp for ERR2231567
> > > 7. The shortest node is 200 bp, i.e. the minimal size for a contig
> > >
> > {: .solution}
> >
> {: .question}
{: .details}

# Conclusion

Metagenomic data can be assembled to, ideally, obtain the genomes of the species that are represented within the input data. But metagenomic assembly is **complex** and there are
- **different approaches** like de Bruijn graphs methods
- **different strategies**, such as co-assembly, when we assembly all samples together, and individual assembly, when we assembly samples one by one
- **different tools** like MetaSPAdes and MEGAHIT

Once the choices made, metagenomic assembly can start:
1. Input data are assembled to obtain contigs and sometimes scaffolds.
2. Assembly quality is evaluated with various metrics.
3. The assembly graph can be visualized.

Once all these steps done, we can move to the next phase to build Metagenomics Assembled Genomes (MAGs): [metagenomic binning]({% link topics/microbiome/tutorials/metagenomics-binning/tutorial.md %}).

tools_json:
["MEGAHIT", "toolshed.g2.bx.psu.edu/repos/iuc/megahit/megahit/1.2.9+galaxy2", "MetaSPAdes", "toolshed.g2.bx.psu.edu/repos/nml/metaspades/metaspades/4.2.0+galaxy0", "QUAST", "toolshed.g2.bx.psu.edu/repos/iuc/quast/quast/5.2.0+galaxy1", "CoverM contig", "toolshed.g2.bx.psu.edu/repos/iuc/coverm_contig/coverm_contig/0.7.0+galaxy0", "Bowtie2", "toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.3+galaxy1", "megahit contig2fastg", "toolshed.g2.bx.psu.edu/repos/iuc/megahit_contig2fastg/megahit_contig2fastg/1.1.3+galaxy10", "Bandage Image", "toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_image/0.8.1+galaxy4", "Bandage Info", "toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_info/0.8.1+galaxy2", "Column join", "toolshed.g2.bx.psu.edu/repos/iuc/collection_column_join/collection_column_join/0.0.3"]


topic: microbiome
tutorial: plasmid-metagenomics-nanopore
title: Antibiotic resistance detection
difficulty: intermediate
time_estimation: 3h
requirements_json: []
tags_json: ["nanopore", "plasmids", "assembly", "amr", "microgalaxy"]
content_hash: d215f107ae167b6ffef1f8df8c1569a6
last_modified: 2026-05-15T22:28:14.185624
zenodo_link: https://doi.org/10.5281/zenodo.3247504
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/plasmid-metagenomics-nanopore/tutorial.html

content:
# Overview


Pervasive use (and misuse) of antibiotics for human disease treatment, as well as for various agricultural purposes, has resulted in the evolution of multi-drug resistant (MDR) pathogenic bacteria. The [Center for Disease Control estimates](https://www.cdc.gov/drugresistance/) that in the U.S. alone, every year at least 2 million people get an antibiotic-resistant infection, and at least 23,000 people die. Antibiotic resistance poses a major public health challenge, and its causes and mitigations are widely studied.

Plasmids are small DNA molecules within a cell which are physically separated from chromosomal DNA and can replicate independently. They are most commonly found as small circular, double-stranded DNA molecules in bacteria.

![Depiction of plasmids](../../images/plasmid-metagenomics-nanopore/plasmids.png)

Plasmids are considered a major vector facilitating the transmission of drug resistant genes among bacteria via [horizontal transfer](https://en.wikipedia.org/wiki/Horizontal_gene_transfer) ({% cite Beatson2014 %}, {% cite Smillie2010 %}). Careful characterization of plasmids and other MDR mobile genetic elements is vital for understanding their evolution and transmission and adaptation to new hosts.

![Illustration of transformation of a bacteria to drug resistance](../../images/plasmid-metagenomics-nanopore/bacterial_transformation.png)

Due to the high prevalence of repeat sequences and inserts in plasmids, using traditional NGS short-read sequencing to assemble plasmid sequences is difficult and time-consuming. With the advent of third-generation single-molecule long-read sequencing technologies, full assembly of plasmid sequences is now possible.

In this tutorial we will recreate the analysis described in the paper by {% cite LiXie2018 %} entitled *Efficient generation of complete sequences of MDR-encoding plasmids by rapid assembly of MinION barcoding sequencing data*. We will use data sequenced by the [Nanopore](https://nanoporetech.com/) MinION sequencer.

The assembly will be performed with **Minimap2** {% icon tool %} ({% cite Li2018 %}),
**Miniasm** {% icon tool %} ({% cite Li2016 %}), **Racon** {% icon tool %} ({% cite Vaser2017 %}) and **Unicycler** {% icon tool %} ({% cite Wick2017 %}).
The downstream analysis will use **Nanoplot** {% icon tool %} ({% cite DeCoster2018 %}),
**Bandage** {% icon tool %} ({% cite Wick2015 %}), **PlasFlow** {% icon tool %} ({% cite Krawczyk2018 %}) and **starmr** {% icon tool %} ([GitHub](https://github.com/phac-nml/staramr)).

A schematic view of the workflow we will perform in this tutorial is given below:

![Workflow representation of this tutorial](../../images/plasmid-metagenomics-nanopore/Workflow.png)


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


{% snippet faqs/galaxy/analysis_results_may_vary.md %}

In this tutorial we use metagenomic Nanopore data, but similar pipelines can be used for other types of datasets or other long-read sequencing platforms.


# Obtaining and preparing data

We are interested in the reconstruction of full plasmid sequences and determining the presence of any antimicrobial resistance genes.
We will use the plasmid dataset created by {% cite LiXie2018 %} for their evaluation of the efficiency of MDR plasmid sequencing by MinION platform. In the experiment, 12 MDR plasmid-bearing bacterial strains were selected for plasmid extraction, including *E. coli*, *S. typhimurium*, *V. parahaemolyticus*, and *K. pneumoniae*.

{% snippet faqs/galaxy/sequencing_nanopore.md %}

## Importing the data into Galaxy

For this tutorial, in order to speed up the analysis time, we will use 6 of the 12 samples from the original study.

> <hands-on-title>Obtaining our data</hands-on-title>
>
> 1. Make sure you have an empty analysis history. Give it a name.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Import Sample Data** [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3247504.svg)](https://doi.org/10.5281/zenodo.3247504)
>    ```
>    https://zenodo.org/record/3247504/files/RB01.fasta
>    https://zenodo.org/record/3247504/files/RB02.fasta
>    https://zenodo.org/record/3247504/files/RB04.fasta
>    https://zenodo.org/record/3247504/files/RB05.fasta
>    https://zenodo.org/record/3247504/files/RB10.fasta
>    https://zenodo.org/record/3247504/files/RB12.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. **Build a list collection** containing all fasta files. Name it `Plasmids`
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
{: .hands_on}

# Quality Control

## NanoPlot to explore data


The first thing we want to do is to get a feeling for our input data and its quality. This is done
using the NanoPlot tool. This will create several plots, a statisical report and an HTML
report page.

![NanoPlot example](../../images/plasmid-metagenomics-nanopore/NanoPlot.png)

> <hands-on-title>Plotting scripts for long read sequencing data</hands-on-title>
>
> 1. {% tool [Nanoplot](toolshed.g2.bx.psu.edu/repos/iuc/nanoplot/nanoplot/1.28.2+galaxy1) %} with the following parameters
>    - {% icon param-select %} *"Select multifile mode"*: `batch`
>    - {% icon param-select %} *"Type of the file(s) to work on"*: `fasta`
>    - {% icon param-collection %} *"files"*: The `Plasmids` dataset collection you just created
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
{: .hands_on}

The `HTML report` gives an overview of various QC metrics for each sample. For example, it will
plot the read length distribution of each sample:
![NanoPlot Output](../../images/plasmid-metagenomics-nanopore/NanoPlot_output.png)


> <question-title></question-title>
>
> What was the mean read length for this (RB01) sample?
>
> > <solution-title></solution-title>
> > 4906.3
> >
> > This can be determined by looking at the NanoStats or HTML output of NanoPlot RB01.
> {: .solution }
{: .question}


For more information on the topic of quality control, please see our 
[sequence analysis training materials]({{site.baseurl}}/topics/sequence-analysis/)

# De-novo Assembly

## Pairwise alignment using Minimap2

In this experiment we used Nanopore sequencing; this means sequencing results with long reads, and significant overlaps between those reads.
To find this overlap, Minimap2 is used. Minimap2 is a sequence alignment program that can be used for different
purposes, but in this case we'll use it to find overlaps between long reads with an error rate up to ~15%.
Typical other use cases for Minimap2 include:

1. mapping PacBio or Oxford Nanopore genomic reads to the human genome
2. splice-aware alignment of PacBio Iso-Seq or Nanopore cDNA or Direct RNA reads against a reference genome
3. aligning Illumina single- or paired-end reads
4. assembly-to-assembly alignment
5. full-genome alignment between two closely related species with divergence below ~15%.

Minimap2 is faster and more accurate than mainstream long-read mappers such as BLASR, BWA-MEM, NGMLR and GMAP and
therefore widely used for Nanopore alignment. Detailed evaluations of Minimap2 are available in
the Minimap2 publication ({% cite Li2018 %}).

![Pairwise alignment](../../images/plasmid-metagenomics-nanopore/Minimap2.png)


> <hands-on-title>Pairwise sequence alignment</hands-on-title>
>
> 1. {% tool [Map with minimap2](toolshed.g2.bx.psu.edu/repos/iuc/minimap2/minimap2/2.17+galaxy2) %} with the following parameters
>    - {% icon param-select %} *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>      - {% icon param-collection %} *"Use the following data collection as the reference sequence"*: `Plasmids` dataset collection we just created
>    - {% icon param-select %} *"Single or Paired-end reads"*: `Single`
>      - {% icon param-collection %} *"Select fastq dataset"*: The `Plasmids` dataset collection
>      - {% icon param-select%} *"Select a profile of preset options"*: `Oxford Nanopore all-vs-all overlap mapping`
>    - In the section **Set advanced output options**:
>      - {% icon param-select %} *"Select an output format"*: `paf`
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
{: .hands_on}

This step maps the Nanopore sequence reads against itself to find overlaps. The result is a PAF file.
PAF is a text format describing the approximate mapping positions between two
set of sequences. PAF is TAB-delimited with each line consisting of the
following predefined fields:

|Col|Type  |Description                               |
|--:|:----:|:-----------------------------------------|
|1  |string|Query sequence name                       |
|2  |int   |Query sequence length                     |
|3  |int   |Query start (0-based)                     |
|4  |int   |Query end (0-based)                       |
|5  |char  |Relative strand: "+" or "-"               |
|6  |string|Target sequence name                      |
|7  |int   |Target sequence length                    |
|8  |int   |Target start on original strand (0-based) |
|9  |int   |Target end on original strand (0-based)   |
|10 |int   |Number of residue matches                 |
|11 |int   |Alignment block length                    |
|12 |int   |Mapping quality (0-255; 255 for missing)  |

View the output of **Minimap2** {% icon tool %} of the collection against RB12, it should look something like this:
```
channel_100_69f2ea89-01c5-45f4-8e1b-55a09acdb3f5_template	4518	114	2613	+	channel_139_250c7e7b-f063-4313-8564-d3efbfa7e38d_template	3657	206	2732	273	2605	0	tp:A:S	cm:i:29	s1:i:240	dv:f:0.2016	rl:i:1516
channel_100_69f2ea89-01c5-45f4-8e1b-55a09acdb3f5_template	4518	148	1212	+	channel_313_35f447cb-7e4b-4c3d-977e-dc0de2717a4d_template	3776	2433	3450	218	1064	0	tp:A:S	cm:i:31	s1:i:210	dv:f:0.1291	rl:i:1516
channel_100_69f2ea89-01c5-45f4-8e1b-55a09acdb3f5_template	4518	251	1328	+	channel_313_a83f7257-52db-46e4-8e2a-1776500c7363_template	3699	2327	3382	208	1082	0	tp:A:S	cm:i:29	s1:i:203	dv:f:0.1363	rl:i:1516
```

## Ultrafast de novo assembly using Miniasm

The mapped reads are ready to be assembled with **Miniasm** {% icon tool%} ({% cite Li2016 %}). Miniasm is a very fast Overlap Layout Consensus based de-novo assembler for noisy long reads.
It takes all-vs-all read self-mappings (typically by **Minimap2** {% icon tool%}) as input and outputs an assembly graph in the GFA format.

Different from mainstream assemblers, miniasm does not have a consensus step.
It simply concatenates pieces of read sequences to generate the final sequences.
The optimal case would be to recreate a complete chromosome or plasmid.
Thus the per-base error rate is similar to the raw input reads.

![Pairwise alignment](../../images/plasmid-metagenomics-nanopore/Miniasm.png "Sequencing and Assembly Schematic. A longer sequence is constructed from many smaller, overlapping fragments. These are aligned and reconstructed during assembly to produce the best guess assembly.")

> <hands-on-title>De novo assembly</hands-on-title>
>
> 1. {% tool [miniasm](toolshed.g2.bx.psu.edu/repos/iuc/miniasm/miniasm/0.3+galaxy0) %} with the following parameters
>    - {% icon param-collection %} *"Sequence Reads"*: The `Plasmids` dataset collection
>    - {% icon param-collection %} *"PAF file"*: `Output Minimap dataset collection` created by **Minimap2** {% icon tool %}
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
{: .hands_on}

The `Assembly Graph` output file gives information about the steps taken in the assembly.

The output should look like:

```
S	utg000001l	GAAATCATCAGGCGTTTTTCACGATATGGACGGGAAGATGCGGAAATAGGCAGGAGGACATAGAA [..]
a	utg000001l	0	channel_364_204a2254-2b6f-4f10-9ec5-6d40f0b870e4_template:101-4457	+	4357
```

## Remapping using Minimap2

Remapping is done with the original reads, using the Miniasm assembly as a reference, in order to improve the consensus base call per position. This is used by **Racon** {% icon tool %} for consensus construction. This is done as some reads which might not have mapped well during the consensus calling, will now map to your scaffold.

The Assembly graph created can be used for mapping again with Minimap2, but first the graph should be transformed to FASTA format.

> <hands-on-title>Pairwise sequence alignment</hands-on-title>
>
> 1. {% tool [GFA to Fasta](toolshed.g2.bx.psu.edu/repos/iuc/gfa_to_fa/gfa_to_fa/0.1.1) %} with the following parameters
>   - {% icon param-collection %} *"Input GFA file"*: the `Assembly Graph` (collection) created by **Miniasm** {% icon tool %}
>
>     > <question-title></question-title>
>     >
>     > How many contigs do we have for the RB05 sample after de novo assembly?
>     > <br><br>
>     > Hint: run **Nanoplot** {% icon tool %} on the output of **GFA to Fasta** {% icon tool %}
>     >
>     > > <solution-title></solution-title>
>     > > 25
>     > >
>     > > This can be determined by looking at the NanoStats output of NanoPlot.
>     > {: .solution }
>     {: .question}
>
> 2. {% tool [Map with minimap2](toolshed.g2.bx.psu.edu/repos/iuc/minimap2/minimap2/2.17+galaxy2) %} with the following parameters
>    - {% icon param-select %} *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>      - {% icon param-collection %} *"Use the following dataset as the reference sequence"*: `FASTA file` output from **GFA to Fasta** {% icon tool %} (collection)
>    - {% icon param-select %} *"Single or Paired-end reads"*: `single`
>      - {% icon param-collection %} *"Select fastq dataset"*: The `Plasmids` collection
>      - {% icon param-select %} *"Select a profile of preset options"*:: `PacBio/Oxford Nanopore read to reference mapping (-Hk19)`
>    - In the section **Set advanced output options**:
>       - {% icon param-select %} *"Select an output format"*: `paf`
>
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
{: .hands_on}

## Ultrafast consensus module using Racon

The mapped reads can be improved even more using **Racon** {% icon tool%} ({% cite Vaser2017 %}) to find a consensus sequence.
Racon is a standalone consensus module to correct raw contigs generated by rapid assembly methods which do not include a consensus step.
It supports data produced by both Pacific Biosciences and Oxford Nanopore Technologies.

![Consensus Module](../../images/plasmid-metagenomics-nanopore/Racon.png)


> <hands-on-title>Consensus module</hands-on-title>
>
> 1. {% tool [Racon](toolshed.g2.bx.psu.edu/repos/bgruening/racon/racon/1.4.13) %} with the following parameters
>   - {% icon param-collection %} *"Sequences"*: The `Plasmids` dataset collection
>   - {% icon param-collection %} *"Overlaps"*: the latest `PAF file` collection created by **Minimap2** {% icon tool %}
>   - {% icon param-collection %} *"Target sequences"*: the `FASTA file` collection created by **GFA to Fasta** {% icon tool %}
>
{: .hands_on}

The **Racon** {% icon tool %} output file gives the final contigs.

The output of RB04 should look something like:

```
>utg000001c LN:i:4653 RC:i:11 XC:f:0.888889
AATGCAGCTATGGCGCGTGCGGTGCCAAGAAAGCCCGCAGATATTCCGCTTCCTCGCTCATT [..]
```

## Visualize assemblies using Bandage

To get a sense of how well our data was assembled, and to determine whether the contigs are chomosomal or plasmid DNA (the former being linear sequences while plasmids are circular molecules), **Bandage** {% icon tool %} can give a clear view of the assembly.

**Bandage** {% icon tool%} ({% cite Wick2015 %}) (a **B**ioinformatics **A**pplication for **N**avigating **D**e novo **A**ssembly **G**raphs **E**asily), is a program that creates visualisations of assembly graphs.
Sequence assembler programs (such as **Miniasm** {% icon tool %} ({% cite Li2016 %}), **Velvet** {% icon tool %} ({% cite Zerbino2008 %}), **SPAdes** {% icon tool %} ({% cite Bankevich2012 %}), **Trinity** {% icon tool %} ({% cite Grabherr2011 %}) and **MEGAHIT** {% icon tool %} {% cite Li2015 %}) carry out assembly by building a graph, from which contigs are generated.

By visualizing these assembly graphs, Bandage allows users to better understand, troubleshoot, and improve the assemblies.

![Bandage GUI](../../images/plasmid-metagenomics-nanopore/Bandage.png)


> <hands-on-title>Visualising de novo assembly graphs</hands-on-title>
>
> 1. {% tool [Bandage image](toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_image/0.8.1+galaxy2) %} with the following parameters
>   - {% icon param-collection %} *"Graphical Fragment Assembly"*: the `Assembly graph` collection created by **Miniasm** {% icon tool %}
>
> 2. Explore {% icon galaxy-eye %} the output images
>
> > <question-title></question-title>
> >
> > In how many samples were the full plasmid sequences assembled?
> >
> > Hint: what shape do you expect plasmid molecules to be?
> >
> > > <solution-title></solution-title>
> > > Ideally, we want to see circular assemblies, indicating the full plasmid sequence was resolved.
> > > This is not the case for most of the samples, but we will improve our assemblies in the next section!
> > {: .solution }
> {: .question}
>
{: .hands_on}


For example, the assembly for sample RB01 looks something like this (your assembly will look a bit different due to randomness in several of the tools):


![Bandage output for sample RB01](../../images/plasmid-metagenomics-nanopore/bandage_pre_unicycler.png "Bandage output for sample RB01. Large fragments were assembled, but not the full circular plasmid molecules.")


As you can see from these Bandage outputs, we were able to assemble our data into fairly large fragments, but were not quite successful in assembling the full (circular) plasmid sequences.

However, all the tools we used to do the assembly have many different parameters that we did not explore, and multiple rounds of mapping and cleaning could improve our data as well. Choosing these parameters carefully could potentially improve our assembly, but this is also a lot of work and not an easy task. This is where **Unicycler** {% icon tool %} ({% cite Wick2017 %}) can help us out.


## Optimizing assemblies using Unicycler

The assembly tools we used in this tutorial are all implemented in **Unicycler** {% icon tool %}, which will repeatedly run these tools on your data using different parameter settings, in order to find the optimal assembly.

**Unicycler** {% icon tool %} has a couple of advantages over running the tools separately:

1. The first modification is to **help circular replicons assemble** into circular string graphs.
2. **Racon** {% icon tool %} polishing is carried out in **multiple rounds to improve the sequence accuracy**. It will polish until the assembly stops improving, as measured by the agreement between the reads and the assembly.
3. Circular replicons are 'rotated' (have their starting position shifted) between rounds of polishing to ensure that **no part of the sequence is left unpolished**.


Let's try it on our data!

> <hands-on-title>Unicycler assembly</hands-on-title>
>
> 1. {% tool [Create assemblies with Unicycler](toolshed.g2.bx.psu.edu/repos/iuc/unicycler/unicycler/0.4.8.0) %} with the following parameters
>   - {% icon param-select %} *"Paired or Single end data"*: `None`
>   - {% icon param-collection %} *"Select long reads. If there are no long reads, leave this empty"*: The `Plasmids` dataset collection
>
> 2. {% tool [Bandage image](toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_image/0.8.1+galaxy2) %} with the following parameters
>   - {% icon param-collection %} *"Graphical Fragment Assembly"*: the `Final Assembly Graph` collection created by **Unicycler** {% icon tool %}
>
> 3. Examine {% icon galaxy-eye %} the output images again
>
> 4. Use the **Scratchbook** {% icon galaxy-scratchbook %} to compare the two assemblies for sample `RB01`
>    - Compare the **Bandage** {% icon tool %} images for our two assemblies:
>      1. The assembly we got from running **minimap2, miniasm, racon** {% icon tool%} (first time we ran bandage)
>      2. The assembly obtained with **Unicycler** {% icon tool %}
>    - Tip: Search your history for the term `bandage` to easily find the outputs from our two bandage runs
>
>    {% snippet faqs/galaxy/features_scratchbook.md %}
>
>    {% snippet faqs/galaxy/histories_search.md %}
>
> 5. Repeat this comparison for the other samples.
>
>    > <question-title></question-title>
>    >
>    > For which samples has the plasmid assembly improved?
>    >
>    > > <solution-title></solution-title>
>    > > Exploring the outputs for all the samples reveals that many now display circular assemblies, indicating the full plasmids sequence was resolved.
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

The Assembly graph image of the RB01 assembly with **miniasm** {% icon tool %} shows one unclear hypothetical plasmid, where the output of **Unicycler** {% icon tool %} shows two clear plasmids, as also shown by {% cite LiXie2018 %}.

![Bandage output](../../images/plasmid-metagenomics-nanopore/Bandage_output.png)

# Species and plasmids

## Prediction of plasmid sequences and classes using PlasFlow

To automatically determine whether the contigs represent chromosomal or plasmid DNA, **PlasFlow** {% icon tool %} ({% cite Krawczyk2018 %}) can be used, also in the case where a full circular plasmid sequence was not assembled. Furthermore, it assigns the contigs to a bacterial class.

**PlasFlow** {% icon tool %} is a set of scripts used for prediction of plasmid sequences in metagenomic contigs.
It relies on the neural network models trained on full genome and plasmid sequences and is able to differentiate between plasmids and chromosomes with accuracy reaching 96%.

![Pairwise alignment](../../images/plasmid-metagenomics-nanopore/PlasFlow.png)


> <hands-on-title>Prediction of plasmid sequences</hands-on-title>
>
> 1. {% tool [PlasFlow](toolshed.g2.bx.psu.edu/repos/iuc/plasflow/PlasFlow/1.0) %} with the following parameters
>   - {% icon param-collection %} *"Sequence Reads"*: the `Final Assembly` collection created by **Unicycler** {% icon tool %}
>
> > <question-title></question-title>
> >
> > What is the classification of contig_id 0 in RB10? (Hint: Check the probability table created by PlasFlow)
> >
> > > <solution-title></solution-title>
> > > plasmid.Proteobacteria
> > >
> > > This can be determined by looking at the 5th column of the probability table.
> > {: .solution }
> {: .question}
{: .hands_on}


The most important output of **PlasFlow** {% icon tool %} is a tabular file containing all predictions, consisting of several columns including:

```
contig_id 	contig_name 	contig_length 	id 	label 	...
```

where:

- `contig_id`is an internal id of sequence used for the classification
- `contig_name` is a name of contig used in the classification
- `contig_length` shows the length of a classified sequence
- `id` is an internal id of a produced label (classification)
- `label` is the actual classification
- `...` represents additional columns showing probabilities of assignment to each possible class

Additionally, PlasFlow produces FASTA files containing input sequences binned to plasmids, chromosomes and unclassified.

# Antibiotic Resistance

## Scan genome contigs for antimicrobial resistance genes

To determine whether the contigs contain antimircobial resistance genes (AMR) [staramr](https://github.com/phac-nml/staramr) can be used.
**Staramr** {% icon tool %} scans bacterial genome contigs against both the **ResFinder** ({% cite Zankari2012 %}), **PointFinder** ({% cite Zankari2017 %}), and **PlasmidFinder** ({% cite Carattoli2014 %}) databases (used by the ResFinder webservice)
and compiles a summary report of detected antimicrobial resistance genes.

![Pairwise alignment](../../images/plasmid-metagenomics-nanopore/StarAmr.png)


> <hands-on-title>Prediction of AMR genes</hands-on-title>
>
> 1. {% tool [staramr](toolshed.g2.bx.psu.edu/repos/nml/staramr/staramr_search/0.7.1+galaxy2) %} with the following parameters
>   - {% icon param-collection %} *"genomes"*: the `Final Assembly` collection created by **Unicycler** {% icon tool %}
>
> > <question-title></question-title>
> >
> > Which samples contained the resistance gene: dfrA17?
> >
> > Hint: Check the resfinder.tsv created by staramr
> >
> > > <solution-title></solution-title>
> > > RB01, RB02, and RB10
> > >
> > > This can be determined by looking at the 2nd column of the resfinder.tsv output (and the first column for the sample names).
> > {: .solution }
> {: .question}
{: .hands_on}

There are 5 different output files produced by **staramr** {% icon tool %}:

1. `summary.tsv`:  A summary of all detected AMR genes/mutations in each genome, one genome per line.
2. `resfinder.tsv`: A tabular file of each AMR gene and additional BLAST information from the **ResFinder** database, one gene per line.
3. `pointfinder.tsv`: A tabular file of each AMR point mutation and additional BLAST information from the **PointFinder** database, one gene per line.
4. `settings.txt`: The command-line, database versions, and other settings used to run `staramr`.
5. `results.xlsx`: An Excel spreadsheet containing the previous 4 files as separate worksheets.

The summary file is most important and provides all the resistance genes found.

## CARD database
To get more information about these antibiotic resistant genes, you can check the [CARD database](https://card.mcmaster.ca) (**C**omprehensive **A**ntibiotic **R**esistance **D**atabase) ({% cite Jia2016 %})

![Screenshot of the CARD database](../../images/plasmid-metagenomics-nanopore/card_database.png "Screenshot of the CARD database interface. CARD gives information about the antibiotic resistance genes, as well as links to relevant publications.")



> <question-title></question-title>
>
> What is the resistance mechanism of the *dfrA17* gene?
>
> > <solution-title></solution-title>
> > antibiotic target replacement
> >
> > This can be determined by searching for [the gene on the CARD database](https://card.mcmaster.ca/ontology/39294)
> > <br><br>
> > For more information about antibiotic resistance mechanisms, see {% cite Munita %}
> {: .solution }
{: .question}




# Conclusion
You have now seen how to perform an assembly on Nanopore sequencing data, and classify the type and species of the sequences, as well as determined the presence of potential antibiotic resistance genes.

As for any analysis, there are many different tools that can do the job, and the tools presented here are just
one possible pipeline. Which tools are best for your specific data and research question depends on a number of
factors. For more information and comparisons between various tools, review papers such as {% cite DeMaio2019 %} and {% cite Jayakumar2017 %}
may provide further insight.

You have worked your way through the following pipeline:

![Workflow representation of this tutorial](../../images/plasmid-metagenomics-nanopore/Workflow.png)

tools_json:
["Nanoplot", "toolshed.g2.bx.psu.edu/repos/iuc/nanoplot/nanoplot/1.28.2+galaxy1", "Map with minimap2", "toolshed.g2.bx.psu.edu/repos/iuc/minimap2/minimap2/2.17+galaxy2", "miniasm", "toolshed.g2.bx.psu.edu/repos/iuc/miniasm/miniasm/0.3+galaxy0", "GFA to Fasta", "toolshed.g2.bx.psu.edu/repos/iuc/gfa_to_fa/gfa_to_fa/0.1.1", "Racon", "toolshed.g2.bx.psu.edu/repos/bgruening/racon/racon/1.4.13", "Bandage image", "toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_image/0.8.1+galaxy2", "Create assemblies with Unicycler", "toolshed.g2.bx.psu.edu/repos/iuc/unicycler/unicycler/0.4.8.0", "PlasFlow", "toolshed.g2.bx.psu.edu/repos/iuc/plasflow/PlasFlow/1.0", "staramr", "toolshed.g2.bx.psu.edu/repos/nml/staramr/staramr_search/0.7.1+galaxy2"]


topic: microbiome
tutorial: pathogen-detection-from-nanopore-foodborne-data
title: Pathogen detection from (direct Nanopore) sequencing data using Galaxy - Foodborne
difficulty: introductory
time_estimation: 4h
requirements_json: []
content_hash: ed8883025996b6fc5d628eed333d6eb6
last_modified: 2026-05-27T09:01:49.384171
zenodo_link: https://zenodo.org/record/11222469
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/pathogen-detection-from-nanopore-foodborne-data/tutorial.html

content:
Food contamination with pathogens is a major burden on our society. In the year 2019, foodborne pathogens caused 137 hospitalisations in Germany [(BVL 2019)](https://www.bvl.bund.de/SharedDocs/Berichte/10_BELA_lebensmittelbed_Krankheitsausbruechen_Dtl/Jahresbericht2019.pdf?__blob=publicationFile&v=4). Globally, they affect an estimated 600 million people a year and impact socioeconomic development at different levels. These outbreaks are mainly due to _Salmonella spp._ followed by _Campylobacter spp._ and Noroviruses, as studied by the [__Food safety - World Health Organization (WHO)__](https://www.who.int/publications/i/item/9789241565165).

During the investigation of a foodborne outbreak, a microbiological analysis of the potentially responsible food vehicle is performed in order to detect the responsible pathogens and identify the contamination source. By default, the [__European Regulation (EC)__](https://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2005:338:0001:0026:EN:PDF) follows ISO standards to detect bacterial pathogens in food: pathogens are detected and identified by **stepwise cultures** on selective media and/or **targeting specific genes with real-time PCRs**. The current gold standard is Pulsed-field Gel Electrophoresis (PFGE) or Multiple-Locus Variable Number Tandem Repeat Analysis (MLVA) to characterize the detected strains. These techniques have some disadvantages.

**Whole Genome Sequencing** (WGS) has been proposed as an alternative. With just one sequencing run, we can:
- detect all genes
- run phylogenetic analysis to link cases
- get information on antimicrobial resistance genes, virulence, serotype, resistance to sanitizers, root cause, and other critical factors in one assay, including historical reference to pathogen emergence.

WGS is more than a surveillance tool and was recommended by the European Centre for Disease Prevention and Control (ECDC) and the European Food Safety Authority (EFSA) for surveillance and outbreak investigation. WGS still requires isolation of the targeted pathogen, which is a time-consuming process, the execution is not always straightforward, nor the success is guaranteed. **Sequencing methods without prior isolation could solve this issue**.

The evolution of sequencing techniques in the last decades has made the development of shotgun metagenomic sequencing possible, *i.e.* the **direct sequencing of all DNA present in a sample**. This approach gives an overview of the genomic composition of all cells in the sample, including the food source itself, the microbial community, and any possible pathogens and their complete genetic information without the need for prior isolation. Several studies have demonstrated the potential of shotgun metagenomics to identify and characterize pathogens and their functional characteristics (*e.g.* virulence genes) in naturally contaminated or purposefully spiked food samples.

The currently available studies used Illumina sequencing, generating short reads. Longer read lengths, generated by third-generation sequencing platforms such as Pacific Biosciences (PacBio) and Oxford Nanopore Technologies (ONT), make it **easier and more practical to identify strains with fewer reads**. MinION (from Oxford Nanopore) is a portable, real-time device for ONT sequencing. Several proof-of-principle studies have shown the **utility of ONT long-read sequencing from metagenomic samples for pathogen identification** ({% cite CIUFFREDA20211497 %}).

{% snippet faqs/galaxy/sequencing_nanopore.md %}

To identify and track foodborne pathogens using long-read metagenomic sequencing, different samples of potentially contaminated food (at different time points or different locations) are prepared, DNA is extracted and sequenced using MinION (ONT). The generated sequencing data then need to be processed using bioinformatics tools.

In this tutorial, we will be presenting a series of Galaxy workflows whose main goals are to:
1. **agnostically detect pathogens** (What exactly is this pathogen and what virulence factors does it carry?) from data extracted directly (without prior cultivation) from a potentially contaminated sample (e.g. food like chicken, beef, etc.) and sequenced using Nanopore
2. **compare different samples to trace** the possible source of contamination

To illustrate how to process such data, we will use datasets generated by [Biolytix](https://www.biolytix.ch/) with the following approach:

![From left to right: Biolytix logo. Chicken + milk. An arrow going to the right toward Chicken + milk and a syringe with "Contamination with known pathogens" written below. An arrow going to the right toward an Eppendorf tube with "DNA extraction" written below,  An arrow going to the right toward a qPCR machine, and another arrow over the qPCR toward a Nanopore sequencing. ](./images/Biolytix_Workflow_Sequencing.png)

Food samples, here chicken, are spiked with known pathogens, here:
- _Salmonella enterica subsp. enterica_  in the sample named `Barcode 10 Spike 2`
- _Salmonella enterica subsp. houtenae_ in the sample named `Barcode 11 Spike 2b`

DNA in the samples is extracted, analyzed with qPCR, and sequenced via Nanopore. We start the tutorial from raw data generated by Nanopore.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Prepare Galaxy and data
Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

Before we can begin any Galaxy analysis, we need to upload the input data: FASTQ files with the sequenced samples.

> <hands-on-title>Import datasets</hands-on-title>
>
> 1. Import the following samples via link from [Zenodo]({{ page.zenodo_link }}) or Galaxy shared data libraries:
>
>    ```text
>    {{ page.zenodo_link }}/files/Barcode10_Spike2.fastq.gz
>    {{ page.zenodo_link }}/files/Barcode11_Spike2b.fastq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Rename the files to `Barcode10` and `Barcode11` respectively
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. Create a collection named `Samples` that includes both datasets (`Barcode10` and `Barcode11`)
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
{: .hands_on}

In this tutorial, we can offer 2 versions:
- A short version, running prebuilt workflows
- A long version, going step-by-step

{% include _includes/cyoa-choices.html option1="Short Version" option2="Long Version" default="Short Version" %}


# Preprocessing

Before starting any analysis, it is always a good idea to assess the quality of your input data and to discard poor quality base content by trimming and filtering reads.

<div class="Short-Version" markdown="1">

In this section we will run a Galaxy workflow that performs the following tasks with the following tools:
1. Assess the reads quality before and after preprocessing it using [__FastQC__](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/), [__NanoPlot__](https://github.com/wdecoster/NanoPlot) and  [__MultiQC__](https://multiqc.info/) ({% cite Ewels2016 %})
2. Trimming and filtering reads by length and quality using [__Porechop__](https://github.com/rrwick/Porechop) and **Fastp** ({% cite Chen2018 %})
3. Remove main host sequences, in this training all __chicken__ (Gallus gallus) sequences, using [__Minimap2__](https://github.com/lh3/minimap2)
4. Remove other possible hosts sequences e.g. human, cow, etc. using [__Kraken2__](https://ccb.jhu.edu/software/kraken2/) ({% cite Wood2014 %}) with the [__Kalamari__](https://github.com/lskatz/Kalamari) database, and [__Krakentools: Extract Kraken Reads By ID__](https://github.com/jenniferlu717/KrakenTools/blob/master/extract_kraken_reads.py) to remove all the hosts sequences before moving on to the next section with only the non-host sequences.

We will run all these steps using a single workflow, then discuss each step and the results in more detail.

> <hands-on-title>Pre-Processing</hands-on-title>
>
> 1. **Import the workflow** into Galaxy
>    - Copy the URL (e.g. via right-click) of [this workflow]({{ site.baseurl }}{{ page.dir }}workflows/nanopore_preprocessing.ga) or download it to your computer.
>    - Import the workflow into Galaxy
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. Run **Workflow 1: Nanopore Preprocessing** {% icon workflow %} using the following parameters
>    - *"Samples Profile"*: `PacBio/Oxford Nanopore read to reference mapping`, which is the technique used for sequencing the samples.
>
>    - {% icon param-files %} *"Collection of all samples"*: `Samples` collection created from the imported Fastq.qz files
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}

The workflow will take a little while to complete. Once tools have completed, the results will be available in your history for viewing. Note that only the most important outputs will be visible; intermediate files are hidden by default.

While you are waiting for the workflow to complete, please continue reading in the next section(s) where we will go into a bit more detail about what happens at each step of the workflow we launched and examine the results.
</div>

## Quality Control and Preprocessing

During sequencing, errors are introduced, such as incorrect nucleotides being called. These are due to the technical limitations of each sequencing platform. Sequencing errors might bias the analysis and can lead to a misinterpretation of the data. **Sequence quality control is therefore an essential first step in your analysis.**

In this tutorial we use similar tools as described in the tutorial ["Quality control"]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}), but more specific to Nanopore data:

- Quality control with
    - [__FastQC__](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) generates a web report that will aid you in assessing the quality of your data
    - [__NanoPlot__](https://github.com/wdecoster/NanoPlot) plotting tool for long read sequencing data and alignments

    <div class="Long-Version" markdown="1">

    > <hands-on-title> Initial quality assessment </hands-on-title>
    > 1. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0) %} with the following parameters:
    >    - {% icon param-files %} *"Raw read data from your current history"*: `Samples` collection created from the imported Fastq.qz files
    >
    > 2. {% tool [NanoPlot](toolshed.g2.bx.psu.edu/repos/iuc/nanoplot/nanoplot/1.28.2+galaxy1) %} with the following parameters:
    >    - *"Select multifile mode"*: `batch`
    >        - *"Type of the file(s) to work on"*: `fastq`
    >            - {% icon param-files %} *"Data input files"*: `Samples` collection created from the imported Fastq.qz files
    >
    >    > <comment-title></comment-title>
    >    > The `NanoPlot` step, as it does not require the results of FastQC to run, can be launched even if FastQC is not ready
    >    {: .comment}
    >
    > 3. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy0) %} with the following parameters:
    >    - In *"Results"*:
    >        - {% icon param-repeat %} *"Insert Results"*
    >            - *"Which tool was used generate logs?"*: `FastQC`
    >                - In *"FastQC output"*:
    >                    - {% icon param-repeat %} *"Insert FastQC output"*
    >                        - *"Type of FastQC output?"*: `Raw data`
    >                        - {% icon param-files %} *"FastQC output"*: collection of `Raw data` outputs of **FastQC** {% icon tool %}
    {: .hands_on}

    </div>


- Read trimming and filtering with [__Porechop__](https://github.com/rrwick/Porechop) and **Fastp** ({% cite Chen2018 %})

    <div class="Long-Version" markdown="1">

    > <hands-on-title> Read trimming and filtering </hands-on-title>
    >
    > 1. {% tool [Porechop](toolshed.g2.bx.psu.edu/repos/iuc/porechop/porechop/0.2.4) %} with the following parameters:
    >    - {% icon param-files %} *"Input FASTA/FASTQ"*: `Samples` collection created from the imported Fastq.qz files
    >    - *"Output format for the reads"*: `fastq.gz`
    >
    > 2. {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.20.1+galaxy0) %} with the following parameters:
    >    - *"Single-end or paired reads"*: `Single-end`
    >        - {% icon param-files %} *"Input 1"*: output collection of **Porechop** {% icon tool %}
    >    - In *Output Options*
    >        - *"Output JSON report"*: `Yes`
    >
    >    > <comment-title></comment-title>
    >    > This step can be launched even if **Porechop** is not done. It will be scheduled and wait until **Porechop** is done to start.
    >    {: .comment}
    {: .hands_on}

    </div>

- Quality recheck after read trimming and filtering with **FastQC** and **Nanoplot** and report aggregation with [__MultiQC__](https://multiqc.info/)

    <div class="Long-Version" markdown="1">

    > <hands-on-title> Final quality checks </hands-on-title>
    > 1. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0) %} with the following parameters:
    >    - {% icon param-files %} *"Raw read data from your current history"*: output collection of **fastp** {% icon tool %}
    >
    > 2. {% tool [NanoPlot](toolshed.g2.bx.psu.edu/repos/iuc/nanoplot/nanoplot/1.28.2+galaxy1) %} with the following parameters:
    >    - *"Select multifile mode"*: `batch`
    >        - *"Type of the file(s) to work on"*: `fastq`
    >            - {% icon param-files %} *"files"*: output collection of **fastp** {% icon tool %}
    >
    > 3. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy0) %} with the following parameters:
    >    - In *"Results"*:
    >        - {% icon param-repeat %} *"Insert Results"*
    >            - *"Which tool was used generate logs?"*: `FastQC`
    >                - In *"FastQC output"*:
    >                    - {% icon param-repeat %} *"Insert FastQC output"*
    >                        - *"Type of FastQC output?"*: `Raw data`
    >                        - {% icon param-files %} *"FastQC output"*: collection of `Raw data` output of **FastQC** {% icon tool %} done after **fastp**
    >        - {% icon param-repeat %} *"Insert Results"*
    >            - *"Which tool was used generate logs?"*: `fastp`
    >                - {% icon param-files %} *"Output of fastp"*: `JSON report` output of **fastp** {% icon tool %}
    {: .hands_on}

    </div>

> <question-title></question-title>
>
> Inspect the HTML two outputs of **MultiQC** for `Barcode10` before and after preprocessing tagged `MultiQC_Before_Preprocessing` and `MultiQC_After_Preprocessing`
>
> 1. How many sequences does `Barcode10` contain before and after trimming?
> 2. What is the quality score over the reads before and after trimming? And the mean score?
> 3. What is the importance of **FastQC**?
>
> > <solution-title></solution-title>
> >
> > 1. Before trimming the file has 114,986 sequences and After trimming the file has 91,434 sequences
> > 2. The "Per base sequence quality" is globally medium: the quality score stays above 20 over the entire length of reads after trimming, while quality below 20 could be seen before trimming specially at the beginning and the end of the reads.
> >
> > Sequence quality of Barcode 10 and Barcode 11 before preprocessing:
> >
> >    ![Sequence Quality of Barcode 10 and Barcode 11 Before Trimming](./images/multiqc_per_base_sequence_quality_plot_barcode10_barcode11_before_trimming.png)
> >
> >
> > Sequence quality of Barcode 10 and Barcode 11 after preprocessing:
> >
> >    ![Sequence Quality of Barcode 10 and Barcode 11 After Trimming](./images/multiqc_per_base_sequence_quality_plot_barcode10_barcode11_after_trimming.png)
> >
> > 3. After checking what is wrong, e.g. before trimming, we should think about the errors reported by **FastQC**: they may come from the type of sequencing or what we sequenced (check the ["Quality control" training]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}): [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) for more details). However, despite these challenges, we can already see sequences getting slightly better after the trimming and filtering, so now we can proceed with our analyses.
> {: .solution}
{: .question}

> <comment-title></comment-title>
> For more information about how to interpret the plots generated by **FastQC** and **MultiQC**, please see our dedicated ["Quality control"]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}) Tutorial.
{: .comment}

## Host read filtering

Generally, we are not interested in the food (host) sequences, rather only those originating from the pathogen itself. It is an important to get rid of all host sequences and to only retain sequences that might include a pathogen, both in order to speed up further steps and to avoid host sequences compromising the analysis.

In this tutorial, we know the samples come from __chicken__ meat spiked with **_Salmonella_** so we already know what will we get as the host and the main pathogen. If the host is not known, **Kraken2** with **Kalamari** database can be used to detect it.

In this tutorial we use:
1. Map reads to __chicken__ reference genome using **Map with minimap2** and **Chicken (Gallus gallus): galGal6** built in reference genome of __chicken__, and we move forward with the unmapped ones.

    <div class="Long-Version" markdown="1">

    > <hands-on-title>Read taxonomic classification for host filtering </hands-on-title>
    >
    > 1. {% tool [Map with minimap2](toolshed.g2.bx.psu.edu/repos/iuc/minimap2/minimap2/2.24+galaxy0) %} with the following parameters:
    >    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
    >        - *"Using reference genome"*: `Chicken (Gallus gallus): galGal6`
    >    - *"Single or Paired-end reads"*: `Single`
    >        - {% icon param-file %} *"Select fastq dataset"*: `out1` (output of **fastp** {% icon tool %})
    >        - *"Select a profile of preset options"*: `PacBio/Oxford Nanopore read to reference mapping (-Hk19) (map-pb)`, which is the technique used for sequencing the samples.
    >    - In *"Alignment options"*:
    >        - *"Customize spliced alignment mode?"*: `No, use profile setting or leave turned off`
    >
    > 2. {% tool [Split BAM by reads mapping status](toolshed.g2.bx.psu.edu/repos/iuc/bamtools_split_mapped/bamtools_split_mapped/2.5.2+galaxy2) %} with the following parameters:
    >    - {% icon param-file %} *"BAM dataset to split by mapped/unmapped"*: `alignment_output` (output of **Map with minimap2** {% icon tool %})
    >
    > 3. {% tool [Samtools fastx](toolshed.g2.bx.psu.edu/repos/iuc/samtools_fastx/samtools_fastx/1.15.1+galaxy2) %} with the following parameters:
    >    - {% icon param-file %} *"BAM or SAM file to convert"*: `unmapped` (output of **Split BAM by reads mapping status** {% icon tool %})
    >    - *"Output format"*: `compressed FASTQ`
    >    - *"Write index read files"*: `No`
    >
    {: .hands_on}

2. Assign filted reads, after mapping (non __chicken__ reads), to taxa using **Kraken2** ({% cite Wood2014 %}) as a further contamination detection using the **Kalamari** database. The **Kalamari** database includes mitochondrial sequences of various known hosts including food hosts.

    <div class="Long-Version" markdown="1">

    > <hands-on-title>Read taxonomic classification for host filtering </hands-on-title>
    >
    > 1. {% tool [Kraken2](toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.1.1+galaxy1) %} with the following parameters:
    >    - *"Single or paired reads"*: `Single`
    >        - {% icon param-file %} *"Input sequences"*: `output` (output of **Samtools fastx** {% icon tool %})
    >    - *"Print scientific names instead of just taxids"*: `Yes`
    >    - In *"Create Report"*:
    >        - *"Print a report with aggregrate counts/clade to file"*: `Yes`
    >        - *"Report counts for ALL taxa, even if counts are zero"*: `Yes`
    >        - *"Report minimizer data"*: `Yes`
    >    - *"Select a Kraken2 database"*: `kalamari`
    >
    {: .hands_on}

    </div>

    > <question-title></question-title>
    >
    > For the tutorial long version takers, run **Samtools fastx** on the `mapped` (output of **Split BAM by reads mapping status** {% icon tool %}), then inspect the output for `Barcode10`. If you are a short version taker then inspect the output named `host_sequences_fastq`.
    >
    > 1. How many __chicken__ sequences were found?
    >
    > > <solution-title></solution-title>
    > >
    > > 1. 53722
    > >
    > {: .solution}
    {: .question}

3. Filter host assigned reads based on **Kraken2** assignments
    1. Manipulate **Kraken2** classification to extract the sequence ids of all hosts sequences identified with **Kraken2**
    2. Filter the FASTQ files to get 1 ouput with the host-assigned sequences and 1 output without the host-assigned reads

    <div class="Long-Version" markdown="1">

    > <hands-on-title> Host read filtering </hands-on-title>
    >
    > 1. {% tool [Krakentools: Extract Kraken Reads By ID](toolshed.g2.bx.psu.edu/repos/iuc/krakentools_extract_kraken_reads/krakentools_extract_kraken_reads/1.2+galaxy1) %} with the following parameters:
    >    - *"Single or paired reads?"*: `Single`
    >        - {% icon param-file %} *"FASTQ/A file"*: `out1` (output of **fastp** {% icon tool %})
    >    - {% icon param-files %} *"Results"*: `Kraken2 with Kalamri database Results` outputs of **Kraken2** {% icon tool %}
    >    - {% icon param-files %} *"Report"*: `Kraken2 with Kalamri database Report` outputs of **Kraken2** {% icon tool %}
    >    - *"Taxonomix ID(s) to match"*:`9031 9606 9913`
    >
    >       We specify here the taxonomic ID of the hosts so we can filter reads assigned to these hosts. Kraken2 uses taxonomic IDs from NCBI, the IDs for a specific taxa can be found at [ncbi](https://www.ncbi.nlm.nih.gov/taxonomy). To be generic, we remove here:
    >         - Human (`9606`)
    >         - Chicken (`9031`)
    >         - Beef (`9913`)
    >
    >       If the contaminated food comes from and may include other animals, you can change the value here.
    >    - *"Invert output"*: `Yes`
    >    - *"Output as FASTQ"*: `Yes`
    >    - *"Include parents"*: `Yes`
    >    - *"Include children"*: `Yes`
    {: .hands_on}

    </div>


> <comment-title></comment-title>
>
> We will need the outputs from this section in the next one.  If yours is still running or you get an error you can go on and upload it so you can start the next workflow, the next hands-on is optional.
>
> > <hands-on-title>Optional Data upload</hands-on-title>
> >
> > 1. Import the quality processed samples fastqsanger files via link from [Zenodo]({{ page.zenodo_link }}) or the Shared Data library:
> >
> >    ```text
> >    {{ page.zenodo_link }}/files/preprocessed_sample_barcode10_spike2.fastq.gz
> >    {{ page.zenodo_link }}/files/preprocessed_sample_barcode11_spike2b.fastq.gz
> >    ```
> >
> > 2. Rename datasets to `Barcode10` and `Barcode11` respectively
> >
> > 3. Create a collection named `collection of preprocessed samples` from the two imported datasets
> >
> {: .hands_on}
{: .comment}


# Taxonomy Profiling

In this section we would like to identify the different organisms found in our samples by assigning taxonomy levels to the reads starting from the kingdom level down to the species level and visualize the result. It's important to check what might be the species of a possible pathogen to be found, it gets us closer to the investigation as well as discovering possible multiple food infections if any existed.

{% snippet topics/microbiome/faqs/taxon.md %}

In the previous section we ran **Kraken2** along with the **Kalamari** database, which is also a kind of taxonomy profiling but the database used is designed to include all possible host sequences. In the following part, we run **Kraken2** again; but this time with one of its built-in databases, **Standard PlusPF**, which can give us more insight into pathogen candidate species than **Kalamari**. You can test this yourself by comparing reports of both **Kraken2** runs.

{% snippet topics/microbiome/faqs/kraken.md %}

<div class="Short-Version" markdown="1">

> <hands-on-title>Taxonomy Profiling and visualisation</hands-on-title>
>
> 1. **Import the workflow** into Galaxy
>    - Copy the URL (e.g. via right-click) of [this workflow]({{ site.baseurl }}{{ page.dir }}workflows/taxonomy_profiling_and_visualisation_with_krona.ga) or download it to your computer.
>    - Import the workflow into Galaxy
>
> 2. Run **Workflow 2: Taxonomy Profiling and Visualization with Krona** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-files %} *"Collection of preprocessed samples"*: `collection of preprocessed samples` collection, output from **Krakentools: Extract Kraken Reads By ID** {% icon tool %} from the preproceesing workflow
>    - *"Kraken database"*: `Prebuilt Refseq indexes:  PlusPF (Standard plus protozoa and fungi) (Version:  2022-06-07 - Downloaded: 2022-09-04T165121Z)`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}

</div>

To assign reads to taxons, we use **Kraken2** with **Standard PlusPF** database.

<div class="Long-Version" markdown="1">

> <hands-on-title> Taxonomy Profiling </hands-on-title>
>
> 1. {% tool [Kraken2](toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.1.1+galaxy1) %} with the following parameters:
>    - *"Single or paired reads"*: `Single`
>        - {% icon param-files %} *"Input sequences"*: collection output from **Krakentools: Extract Kraken Reads By ID** {% icon tool %} from the preprocessing section
>    - In *"Create Report"*:
>        - *"Print a report with aggregrate counts/clade to file"*: `Yes`
>    - *"Select a Kraken2 database"*: `Prebuilt Refseq indexes:  PlusPF (Standard plus protozoa and fungi) (Version:  2022-06-07 - Downloaded: 2022-09-04T165121Z)`
{: .hands_on}

</div>

> <question-title></question-title>
>
> Inspect the **Kraken2** report for `Barcode10`
>
> 1. What is the most commonly found species?
> 2. What is the second most commonly found species?
> 3. How many sequences are classified and how many are unclassified?
> 4. What are the differences between **Kraken2** tool's report with **Kalamari** database and **Kraken2** tool's report with **Standard PlusPF** database regarding the previous 3 questions?
>
> > <solution-title></solution-title>
> >
> > 1. Genus level _Salmonella_ with 9,950 sequences
> > 2. Genus level _Escherichia_ with 1,949 sequences
> > 3. 33,941 sequences are classified and 3,738 are unclassified
> > 4. With **Kalamari** database the most found Genus is _Escherichia_ with 13,943 sequences and the second most found Genus is _Salmonella_ with 10,585 sequences. The number of classified sequences are 30,838 sequences and the unclassified sequences are 6,874. In conclusion, both databases are able to show the similar results of the most common identified species, but with different counts of identified sequences. As well as, the number of the classified sequences with **Standard PlusPF** database is higher than **Kalamari** database.
> {: .solution}
{: .question}

In order to view the taxonomy profiling produced by **Kraken2** tool, there are a lot of tools to be used afterwards such as **Krona pie chart**, which we will be using in this tutorial. For later, you can also check out [__Pavian__](https://academic.oup.com/bioinformatics/article/36/4/1303/5573755) tool, as well as **Phinch visualization**, which is an interactive tool that contains multiple visualization plots, it is interactive alowing you to choose between different parameters, you can visualize each taxonomic level on its own, you can have the metadata of the samples represented along with the taxonomic visualization, download all plots for publications and a lot of other benefits.


<div class="Long-Version" markdown="1">

> <hands-on-title> Visualisation </hands-on-title>
>
> 1. {% tool [Krakentools: Convert kraken report file](toolshed.g2.bx.psu.edu/repos/iuc/krakentools_kreport2krona/krakentools_kreport2krona/1.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Kraken report file"*: `report_output` (output of **Kraken2** {% icon tool %})
>
> 2. {% tool [Krona pie chart](toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1+galaxy0) %} with the following parameters:
>    - *"What is the type of your input data"*: `Tabular`
>        - {% icon param-file %} *"Input file"*: `output` (output of **Krakentools: Convert kraken report file** {% icon tool %})
>
{: .hands_on}

</div>


Now let's explore the **Krona pie chart** output for `Barcode11`

> <question-title></question-title>
>
> 1. What is the most commonly found species?
> 2. What is the second most commonly found species?
>
> > <solution-title></solution-title>
> >
> > 1. At Genus level: *Salmonella* with 16,111 sequences
> > 2. At Genus level: *Pseudomonas* with 14,251 sequences
> >
> >    ![Taxonomy Krona Pie Chart Barcode 11](./images/krona_taxonomy_pie_chart_barcode11.png)
> >
> {: .solution}
{: .question}

> <comment-title></comment-title>
> While these steps are running, you can move on to the next section **Gene based pathogenic identification** and run the steps there, as well. Both analyses can execute in parallel.
{: .comment}

You may have noticed some sequences have been assigned to the Human Genome (Homosapians) species, when we run **Kraken2** using the **Standard PlusPF** in this section. However, in the **pre-processing** section when we ran **Kraken2** with **Kalamari** no Human Genomes were found. The lab (data producers) has confirmed that these sequences assigned to human by **Standard PlusPF** database are not human and there should be no human sequences in the samples as **Kalamari** database result's confirmed. So these sequences were wrongly assigned to human by **Standard PlusPF**. That is due to resemblance between organisms and the limited species coverage of **Kraken2** databases sometimes do happen that reads corresponding to higher organisms get mapped to humans. It was a very severe problem for the **Standard PlusPF**, because yeast genes were mis-assigned to human.

We decide to keep these sequences since we do not know what are they via the **taxonomy profiling** step, which could mean that they might be identified as pathogens in the coming steps, and if we delete them we are possibly losing important information and losing the main goal of the workflow to detect pathogens and track them.

# Gene-based pathogen identification

With taxonomy profiling, we identified some bacterial species. But we want to be sure they are pathogenic, by **looking for genes known to be linked to pathogenicity or to the pathogenecity character** of the organim:

- [**Virulence Factor (VF)**](https://www.sciencedirect.com/topics/immunology-and-microbiology/virulence-factor): gene products, usually proteins, involved in pathogenicity. By identifiying them we can call a pathogen and its severity level
- [**Antimicrobial Resistance genes (AMR)**](https://www.sciencedirect.com/topics/engineering/antibiotic-resistance-genes).

    These type of genes have three fundamental mechanisms of antimicrobial resistance that are enzymatic degradation of antibacterial drugs, alteration of bacterial proteins that are antimicrobial targets, and changes in membrane permeability to antibiotics, which will lead to not altering the target site and spread throughput the pathogenic bacteria decreasing the overall fitness of the host.

To look for these genes and determine the strain of the bacteria we are testing for pathogenicity we do:

1. Genome assembly to get contigs, i.e. longer sequences, using **metaflye** ({% cite flye %})  then assembly polishing using [__medaka consensus pipeline__](https://github.com/nanoporetech/medaka) and visualizing the assembly graph using **Bandage Image** ({% cite Wick2015 %})
2. Generate reports with **AMR** genes and **VF** using [__ABRicate__](https://github.com/tseemann/abricate)

As outputs, we will get our **FASTA** and **Tabular** files to track genes and visualize our pathogenic identification.

<div class="Short-Version" markdown="1">

> <hands-on-title>Gene based Pathogenic Identification</hands-on-title>
>
> 1. **Import the workflow** into Galaxy
>    - Copy the URL (e.g. via right-click) of [this workflow]({{ site.baseurl }}{{ page.dir }}workflows/gene_based_pathogen_identification.ga) or download it to your computer.
>    - Import the workflow into Galaxy
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. Run **Workflow 3: Gene-based Pathogen Identification** {% icon workflow %} using the following parameters:
>    - {% icon param-file %} *"Collection of preprocessed samples"*: `collection of preprocessed samples` collection output from **Krakentools: Extract Kraken Reads By ID** {% icon tool %} from the preprocessing workflow
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}

</div>

## Assembly

To identify VF or AMR genes, it is better to assemble reads into longer seuqences or contigs, that can be then used to search databases for the presence of any pathogenic gene:

- Assembly of long-read metagenomic data using **metaflye** or **Flye**.

    <div class="Long-Version" markdown="1">

    > <hands-on-title> Assembly with Flye </hands-on-title>
    >
    > 1. {% tool [Build list](__BUILD_LIST__) %} with the following parameters:
    >    - In *"Dataset"*:
    >        - {% icon param-repeat %} *"Insert Dataset"*
    >            - {% icon param-collection %} *"Input Dataset"*: collection output from **Krakentools: Extract Kraken Reads By ID** {% icon tool %} from the preprocessing section
    >            - *"Label to use"*: `Index`
    >
    > 2. {% tool [Flye](toolshed.g2.bx.psu.edu/repos/bgruening/flye/flye/2.9+galaxy0) %} with the following parameters:
    >    - {% icon param-file %} *"Input reads"*: collection output from **Build list** {% icon tool %}
    >
    >      > <comment-title></comment-title>
    >      > We need to run **Flye** individually on each sample otherwise **Flye** runs by default a co-assembly mode, *i.e.* it combines reads of both samples together before running the assembly.
    >      {: .comment}
    >
    >    - *"Mode"*: `Nanopore HQ (--nano-hq)`
    >    - *"Perform metagenomic assembly"*: `Yes`
    >    - *"Reduced contig assembly coverage"*: `Disable reduced coverage for initial disjointing assembly`
    >
    {: .hands-on}

    </div>

- For the visualization of the assembly graph output from **Flye** we have chosen **Bandage Image**.

    <div class="Long-Version" markdown="1">

    > <hands-on-title> Visualization of the assembly grap </hands-on-title>
    > 3. {% tool [Bandage Image](toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_image/0.8.1+galaxy2) %} with the following parameters:
    >    - {% icon param-files %} *"Graphical Fragment Assembly"*: `assembly_graph` Assembly graph outputs of **Flye** {% icon tool %}
    {: .hands-on}

    </div>

- Contig polishing using **medaka** to correct the long, error-prone Nanopore reads

    <div class="Long-Version" markdown="1">

    > <hands-on-title> Contig polishing </hands-on-title>
    >
    > 4. {% tool [medaka consensus pipeline](toolshed.g2.bx.psu.edu/repos/iuc/medaka_consensus_pipeline/medaka_consensus_pipeline/1.7.2+galaxy0) %} with the following parameters:
    >    - {% icon param-files %} *"Select basecalls"*: collection output from **Krakentools: Extract Kraken Reads By ID** {% icon tool %} from the preprocessing section
    >    - {% icon param-files %} *"Select assembly"*: `consensus` collection output of **Flye** {% icon tool %}
    >    - *"Select model"*: `r941_min_hac_g507`
    >    - *"Select output file(s)"*: `select all`
    {: .hands-on}

    To keep information about the provenance of the contigs, we extract the sample names from the **collection of preprocessed samples** collection and add it to the contigs files.

    > <hands-on-title> Contig renaming to add sample names </hands-on-title>
    > 1. {% tool [FASTA-to-Tabular](toolshed.g2.bx.psu.edu/repos/devteam/fasta_to_tabular/fasta2tab/1.1.1) %} with the following parameters:
    >    - {% icon param-files %} *"Convert these sequences"*: collection output of **medaka consensus pipeline** {% icon tool %}
    > 2. {% tool [Extract element identifiers](toolshed.g2.bx.psu.edu/repos/iuc/collection_element_identifiers/collection_element_identifiers/0.0.2) %} with the following parameters:
    >    - {% icon param-files %} *"Dataset collection"*: collection output from **Krakentools: Extract Kraken Reads By ID** {% icon tool %} from the preprocessing section
    >
    > 3. {% tool [Split file](toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/split_file_to_collection/0.5.0) %} with the following parameters:
    >    - {% icon param-files %} *"Text file to split"*: output from **Extract element identifiers** {% icon tool %}
    >
    > 4. {% tool [Parse parameter value](param_value_from_file) %} with the following parameters:
    >    - {% icon param-files %} *"Input file containing parameter to parse out of"*: output from **Split file** {% icon tool %}
    >
    > 5. {% tool [Compose text parameter value](toolshed.g2.bx.psu.edu/repos/iuc/compose_text_param/compose_text_param/0.1.1) %} with the following parameters:
    >    - In *"components"*:
    >        - {% icon param-repeat %} *"Insert components"*
    >            - *"Choose the type of parameter for this field"*: `Text Parameter`
    >                - *"Enter text that should be part of the computed value"*: output from **Parse parameter value** {% icon tool %}
    >        - {% icon param-repeat %} *"Insert components"*
    >            - *"Choose the type of parameter for this field"*: `Text Parameter`
    >                - *"Enter text that should be part of the computed value"*: `_$1`
    >
    > 6. {% tool [Replace](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/1.1.4) %} with the following parameters:
    >    - {% icon param-file %} *"File to process"*: collection output of **FASTA-to-Tabular** {% icon tool %}
    >    - In *"Find and Replace"*:
    >        - {% icon param-repeat %} *"Insert Find and Replace"*
    >            - *"Find pattern"*: `^(.+)$`
    >            - *"Replace with"*: output from **Compose text parameter value** {% icon tool %}
    >            - *"Find-Pattern is a regular expression"*: `Yes`
    >            - *"Replace all occurences of the pattern"*: `Yes`
    >            - *"Find and Replace text in"*: `specific column`
    >                - *"in column"*: `Column: 1`
    >
    > 7. {% tool [Tabular-to-FASTA](toolshed.g2.bx.psu.edu/repos/devteam/tabular_to_fasta/tab2fasta/1.1.1) %} with the following parameters:
    >    - {% icon param-file %} *"Tab-delimited file"*: `outfile` (output of **Replace** {% icon tool %})
    >    - *"Title column(s)"*: `c1`
    >    - *"Sequence column"*: `c2`
    >
    > 8. Rename the output collection `Contigs`
    {: .hands-on}

    </div>

> <question-title></question-title>
>
> Inspect **Flye** and **Medaka consensus pipeline** output results for `Barcode10`
>
> 1. How many different contigs did you get after **Flye**, collection name: Flye Consensus Fasta?
> 2. How many were left after **Medaka consensus pipeline**, collection name: Sample all contigs, and what does that mean?
> 2. What is the result of your **Bandage Image**?
>
> > <solution-title></solution-title>
> >
> > 1. After **Flye** we have got 130 contigs
> > 2. After **Medaka consensus pipeline** all 130 contigs were kept, which means that the quality of the **Flye** run was high, and as a result the polishing did not remove any of the contigs.
> > 3. The graph looks like:
> >
> >    ![Bandage Image Barcode 10 Assembly Graph](./images/bandage_image_flye_graph.png)
> >
> {: .solution}
{: .question}

<!--## Multilocus Sequence Typing

**MLST** tool is used to scan the [pubMLST](https://pubmlst.org/) database against PubMLST typing schemes. It's one of the analyses that you can perform on your dataset to determine the allele IDs, you can also detect novel alleles. This step is not essential to identify pathogens and track them in the remainder of this tutorial, however we wanted to show some of the analysis that one can use Galaxy in to understand more about the dataset, as well as identifying the strain that might be a pathogen or not.

<div class="Long-Version" markdown="1">

> <hands-on-title> Multilocus Sequence Typing </hands-on-title>
>
> 1. {% tool [MLST](toolshed.g2.bx.psu.edu/repos/iuc/mlst/mlst/2.22.0) %} with the following parameters:
>    - {% icon param-files %} *"input_files"*:  collection output of **medaka consensus pipeline** {% icon tool %} FASTA files with contigs
>    - *"Specify advanced parameters"*: `Yes, see full parameter list`
>        - *"Output novel alleles"*: `Yes`
>        - *"Automatically set MLST scheme"*: `Automatic MLST scheme detection`
>
{: .hands-on}

</div>

The output file of the **MLST** tool is a tab-separated output file which contains:
- the filename
- the closest PubMLST scheme name
- the ST (sequence type)
- the allele IDs

> <question-title></question-title>
>
> Inspect **MLST** {% icon tool %} results
>
> 1. What is the the closest PubMLST typing scheme name detected by the tool for  `Barcode11`?
>
> > <solution-title></solution-title>
> >
> > 1. senterica_achtman_2 ({% cite sentericaachtman2 %})
> >
> {: .solution}
{: .question}
-->

## Antimicrobial Resistance Genes

Now, to search **AMR** genes among our samples' contigs, we run **ABRicate** and choose the [__NCBI Bacterial Antimicrobial Resistance Gene Database (AMRFinderPlus)__](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6811410/) from the advanced options of the tool. The tool checks if there is an AMR found or not, if found then in which contig it is, its location on the contig, what the name of the exact product is, what substance it provides resistance against and a lot of other information regarding the found **AMR**.

<div class="Long-Version" markdown="1">

> <hands-on-title> Antimicrobial Resistance Genes Identification </hands-on-title>
>
> 1. {% tool [ABRicate](toolshed.g2.bx.psu.edu/repos/iuc/abricate/abricate/1.0.1) %} with the following parameters:
>    - {% icon param-files %} *"Input file (Fasta, Genbank or EMBL file)"*: collection output of **medaka consensus pipeline** {% icon tool %} FASTA files with contigs
>    - In *"Advanced options"*:
>        - *"Database to use - default is 'resfinder'"*: `NCBI Bacterial Antimicrobial Resistance Reference Gene Database`
> 2. Rename **ABRicate** the output collection `amr_identified_by_ncbi`
{: .hands-on}

</div>

The outputs of **ABRicate** is a tabular file with different columns:
1. `FILE`: The filename this hit came from
2. `SEQUENCE`: The sequence in the filename
3. `START`: Start coordinate in the sequence
4. `END`: End coordinate
5. `STRAND`: AMR gene
6. `GENE`: AMR gene
7. `COVERAGE`: What proportion of the gene is in our sequence
8. `COVERAGE_MA`: A visual represenation
9. `GAPS`: Was there any gaps in the alignment - possible pseudogene?
10. `%COVERAGE`: Proportion of gene covered
11. `%IDENTITY`: Proportion of exact nucleotide matches
12. `DATABASE`: The database this sequence comes from
13. `ACCESSION`: The genomic source of the sequence
14. `PRODUCT`
15. `RESISTANCE`

<!--<div class="Long-Version" markdown="1">

> <hands-on-title> Antimicrobial Resistance Genes Identification </hands-on-title>
> 2. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c13`
>    - {% icon param-file %} *"From"*: `report` (output of the **ABRicate** {% icon tool %} run)
>
> 3. {% tool [Add line to file](toolshed.g2.bx.psu.edu/repos/bgruening/add_line_to_file/add_line_to_file/0.1.0) %} with the following parameters:
>    - *"text to add"*: `Spike2Barcode10` or `Spike2bBarcode11` respectively to the tag chosen for the first parameter (the sample ID)
>    - {% icon param-file %} *"input file"*: `out_file1` (output of the **Cut** {% icon tool %})
{: .hands-on}

</div>-->
To prepare the **ABRicate**{% icon tool %} output tabulars of both samples for further analysis in the **Pathogen Detection Samples Aggregation and Visualisation** section, tabular manipulation tools such as **Replace**{% icon tool %} is used. We mainly use it to add the sample ID along with which contig at which exact location.

<div class="Long-Version" markdown="1">

> <hands-on-title>Antimicrobial Resistance Genes Identification</hands-on-title>
> 1. {% tool [Replace](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/1.1.4) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `report` (output of **ABRicate** {% icon tool %})
>    - In *"Find and Replace"*:
>        - {% icon param-repeat %} *"Insert Find and Replace"*
>            - *"Find pattern"*: `#FILE`
>            - *"Replace with"*: `SampleID`
>            - *"Replace all occurences of the pattern"*: `Yes`
>            - *"Find and Replace text in"*: `specific column`
>                - *"in column"*: `c1`
>        - {% icon param-repeat %} *"Insert Find and Replace"*
>            - *"Find pattern"*: `^(.+)$`
>            - *"Replace with"*: output from **Compose text parameter value** {% icon tool %}
>            - *"Find-Pattern is a regular expression"*: `Yes`
>            - *"Replace all occurences of the pattern"*: `Yes`
>            - *"Ignore first line"*: `Yes`
>            - *"Find and Replace text in"*: `specific column`
>                - *"in column"*: `c2`
>
> 2. Rename the output collection `AMRs`
{: .hands-on}

</div>

> <question-title></question-title>
>
> Inspect **ABRicate** output files from `Barcode10`and `Barcode11` tags
>
> 1. How many **AMR** genes found in `Barcode10` sample, what are they? Give more details about them.
> 2. How many **AMR** genes found in `Barcode11` sample, what are they? Give more details about them.
>
> > <solution-title></solution-title>
> >
> > 1. 7 **AMR** genes were found:
> >    1. Tet(C), which resists [TETRACYCLINE](https://medlineplus.gov/druginfo/meds/a682098.html). It was found in contig 119 from the position 1635 till 2810, with 100% coverage, so 100% of gene is covered in this contig.
> >    2. Tet(34), which resists [TETRACYCLINE](https://medlineplus.gov/druginfo/meds/a682098.html). It was found in contig 135 from the position 69874 till 70223, with 74.41% coverage.
> >    3. aac(6')_Yersi, which resists [AMINOGLYCOSIDE](https://en.wikipedia.org/wiki/Aminoglycoside). It was found in contig 156 from the position 37698 till 38000, with  69.68 % coverage.
> >    4. 2 genes with sulfonamide-resistant dihydropteroate synthase Sul1 products
> >    5. 2 genes with oxacillin-hydrolyzing class D beta-lactamase OXA-2 products
> > 2. 2 **AMR** genes were found by the database in `Barcode11` sample, Tet(34) and aac(6')_Yersi.
> >
> {: .solution}
{: .question}

## Virulence Factor identification

In this step we return back to the main goal of the tutorial where we want to identify the pathogens: **identify if the bacteria found in our samples are pathogenic bacteria or not**. One of the ways to do that is to identify if the sequences include genes with a [Virulence Facor](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2646308/) or not, such that if the samples include gene(s) with a Virulence Factor then it for sure is a pathogen.

> <comment-title>Definitions</comment-title>
>
> **Bacterial Pathogen**: A bacterial pathogen is usually defined as any bacterium that has the capacity to cause disease. Its ability to cause disease is called pathogenicity.
>
> **Virulence**: Virulence provides a quantitative measure of the pathogenicity or the likelihood of causing a disease.
>
> **Virulence Factors**: Virulence factors refer to the properties (i.e., gene products) that enable a microorganism to establish itself on or within a host of a particular species and enhance its potential to cause disease. Virulence factors include bacterial toxins, cell surface proteins that mediate bacterial attachment, cell surface carbohydrates and proteins that protect a bacterium, and hydrolytic enzymes that may contribute to the pathogenicity of the bacterium.
>
{: .comment}

To identifly VFs, we use again **ABRicate** but this time with the [__VFDB__](https://pubmed.ncbi.nlm.nih.gov/26578559/) from the advanced options of the tool.

<div class="Long-Version" markdown="1">

> <hands-on-title> Virulence Factor identification </hands-on-title>
>
> 1. {% tool [ABRicate](toolshed.g2.bx.psu.edu/repos/iuc/abricate/abricate/1.0.1) %} with the following parameters:
>    - {% icon param-files %} *"Input file (Fasta, Genbank or EMBL file)"*: collection output of **medaka consensus pipeline** {% icon tool %} FASTA files with contigs
>    - In *"Advanced options"*:
>        - *"Database to use - default is 'resfinder'"*: `VFDB`
>
> 2. Rename **ABRicate** the output collection `vfs_of_genes_identified_by_vfdb`
{: .hands-on}

</div>

> <question-title></question-title>
>
> Inspect **VFs of genes Identified by VFDB** output file from`Barcode10`and `Barcode11`
>
> 1. How many different **VFs** gene products were foun

tools_json:
["FastQC", "toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0", "NanoPlot", "toolshed.g2.bx.psu.edu/repos/iuc/nanoplot/nanoplot/1.28.2+galaxy1", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy0", "Porechop", "toolshed.g2.bx.psu.edu/repos/iuc/porechop/porechop/0.2.4", "fastp", "toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.20.1+galaxy0", "Map with minimap2", "toolshed.g2.bx.psu.edu/repos/iuc/minimap2/minimap2/2.24+galaxy0", "Split BAM by reads mapping status", "toolshed.g2.bx.psu.edu/repos/iuc/bamtools_split_mapped/bamtools_split_mapped/2.5.2+galaxy2", "Samtools fastx", "toolshed.g2.bx.psu.edu/repos/iuc/samtools_fastx/samtools_fastx/1.15.1+galaxy2", "Kraken2", "toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.1.1+galaxy1", "Krakentools: Extract Kraken Reads By ID", "toolshed.g2.bx.psu.edu/repos/iuc/krakentools_extract_kraken_reads/krakentools_extract_kraken_reads/1.2+galaxy1", "Krakentools: Convert kraken report file", "toolshed.g2.bx.psu.edu/repos/iuc/krakentools_kreport2krona/krakentools_kreport2krona/1.2+galaxy1", "Krona pie chart", "toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1+galaxy0", "Build list", "__BUILD_LIST__", "Flye", "toolshed.g2.bx.psu.edu/repos/bgruening/flye/flye/2.9+galaxy0", "Bandage Image", "toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_image/0.8.1+galaxy2", "medaka consensus pipeline", "toolshed.g2.bx.psu.edu/repos/iuc/medaka_consensus_pipeline/medaka_consensus_pipeline/1.7.2+galaxy0", "FASTA-to-Tabular", "toolshed.g2.bx.psu.edu/repos/devteam/fasta_to_tabular/fasta2tab/1.1.1", "Extract element identifiers", "toolshed.g2.bx.psu.edu/repos/iuc/collection_element_identifiers/collection_element_identifiers/0.0.2", "Split file", "toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/split_file_to_collection/0.5.0", "Parse parameter value", "param_value_from_file", "Compose text parameter value", "toolshed.g2.bx.psu.edu/repos/iuc/compose_text_param/compose_text_param/0.1.1", "Replace", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/1.1.4", "Tabular-to-FASTA", "toolshed.g2.bx.psu.edu/repos/devteam/tabular_to_fasta/tab2fasta/1.1.1", "MLST", "toolshed.g2.bx.psu.edu/repos/iuc/mlst/mlst/2.22.0", "ABRicate", "toolshed.g2.bx.psu.edu/repos/iuc/abricate/abricate/1.0.1", "Cut", "Cut1", "Add line to file", "toolshed.g2.bx.psu.edu/repos/bgruening/add_line_to_file/add_line_to_file/0.1.0", "Clair3", "toolshed.g2.bx.psu.edu/repos/iuc/clair3/clair3/0.1.12+galaxy0", "bcftools norm", "toolshed.g2.bx.psu.edu/repos/iuc/bcftools_norm/bcftools_norm/1.9+galaxy1", "SnpSift Filter", "toolshed.g2.bx.psu.edu/repos/iuc/snpsift/snpSift_filter/4.3+t.galaxy1", "SnpSift Extract Fields", "toolshed.g2.bx.psu.edu/repos/iuc/snpsift/snpSift_extractFields/4.3+t.galaxy0", "Samtools depth", "toolshed.g2.bx.psu.edu/repos/iuc/samtools_depth/samtools_depth/1.15.1+galaxy2", "Samtools coverage", "toolshed.g2.bx.psu.edu/repos/iuc/samtools_coverage/samtools_coverage/1.15.1+galaxy2", "bcftools consensus", "toolshed.g2.bx.psu.edu/repos/iuc/bcftools_consensus/bcftools_consensus/1.15.1+galaxy3", "Remove beginning", "Remove beginning1", "Group", "Grouping1", "Filter empty datasets", "__FILTER_EMPTY_DATASETS__", "Column join", "toolshed.g2.bx.psu.edu/repos/iuc/collection_column_join/collection_column_join/0.0.3", "Column Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3", "Heatmap w ggplot", "toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_heatmap/ggplot2_heatmap/3.4.0+galaxy0", "Collapse Collection", "toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0", "Split by group", "toolshed.g2.bx.psu.edu/repos/bgruening/split_file_on_column/tp_split_on_column/0.6", "bedtools getfasta", "toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_getfastabed/2.30.0+galaxy1", "ClustalW", "toolshed.g2.bx.psu.edu/repos/devteam/clustalw/clustalw/2.1+galaxy1", "FASTTREE", "toolshed.g2.bx.psu.edu/repos/iuc/fasttree/fasttree/2.1.10+galaxy1", "Newick Display", "toolshed.g2.bx.psu.edu/repos/iuc/newick_utils/newick_display/1.6+galaxy1"]

tags_json:
["microgalaxy", "Nanopore data analysis", "Pathogens detection", "Phylogenetic tree", "Heatmap", "cyoa"]


topic: microbiome
tutorial: clinical-mp-5-data-interpretation
title: Clinical Metaproteomics 5: Data Interpretation
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: ["[label-TMT11]"]
content_hash: 7734127a4a0739c599bc2cece10a8783
last_modified: 2026-05-27T09:01:49.454120
zenodo_link: https://doi.org/10.5281/zenodo.10105821
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/clinical-mp-5-data-interpretation/tutorial.html

content:
The final workflow in the array of clinical metaproteomics tutorials is the data interpretation workflow. Interpreting MaxQuant data using MSstats involves applying a rigorous statistical framework to glean meaningful insights from quantitative proteomic datasets. The MaxQuant output is explored to understand data distribution and variability. Subsequent normalization helps account for systematic variations. MSstats allows the user to define the experimental design, including sample groups and conditions, to perform statistical analysis. The output provides valuable information about differential protein expression across conditions, estimates of fold changes, and associated p-values, aiding in the identification of biologically significant proteins. Furthermore, MSstats enables quality control and data visualization, ultimately enhancing our ability to draw meaningful conclusions from complex proteomic datasets. Additional tutorial material for using MaxQuant and MSstatTMT for TMT data analysis can be found at [MaxQuant and MSstats for the analysis of TMT data](https://gxy.io/GTN:T00220).

![Data-Interpretation-workflow]({% link topics/proteomics/images/clinical-mp/clinical-mp-data-interpretation.JPG %})

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/10105821/files/Annotation.tabular
>    https://zenodo.org/records/10105821/files/Comparison_Matrix.tabular
>    https://zenodo.org/records/10105821/files/MaxQuant_Evidence.tabular
>    https://zenodo.org/records/10105821/files/MaxQuant_Protein_Groups.tabular
>    https://zenodo.org/records/10105821/files/Quantified-Peptides.tabular
>
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Add to each database a tag corresponding to ...
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Import Workflow


> <hands-on-title>Running the Workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/proteomics/tutorials/clinical-mp-5-data-interpretation/workflows/WF5_Data_Interpretation_Worklow.ga" title="Data Interpretation Workflow" %}
>
> 2. Run **Workflow** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *" Quantified Peptides * "*: `Quantified-Peptides.tabular`
>    - {% icon param-file %} *" MaxQuant Protein Groups"*: `MaxQuant_Protein_Groups.tabular`
>    - {% icon param-file %} *" MaxQuant Evidence"*: `MaxQuant_Evidence.tabular`
>    - {% icon param-file %} *" Annotation.txt "*: `Annotation.tabular`
>    - {% icon param-file %} *" Comparison Matrix "*: `Comparison_Matrix.tabular`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}

## Taxonomic analysis with **Unipept**
Unipept serves as a vital bioinformatics platform for the analysis of mass spectrometry-based shotgun proteomics data, especially in the study of microbial communities. Its primary utility lies in taxonomic and functional analyses, enabling researchers to identify and quantify microorganisms within diverse environments. The platform facilitates comparative studies across samples, conditions, or time points, shedding light on the dynamic responses of microbial communities to environmental changes. Unipept integrates with public databases like UniProt, ensuring access to comprehensive and updated information for annotations. Being community-driven and open source, Unipept fosters collaboration and transparency, with a user-friendly web interface that accommodates researchers of varying bioinformatics expertise. In essence, Unipept is an invaluable resource, offering tools for the exploration of metaproteomic data and contributing to advancements in our understanding of microbial ecology.


> <hands-on-title> Unipept 5.0 </hands-on-title>
>
>
> 1. {% tool [Unipept](toolshed.g2.bx.psu.edu/repos/galaxyp/unipept/unipept/4.5.1) %} with the following parameters:
>    - *"Unipept application"*: `peptinfo: Tryptic peptides and associated EC and GO terms and lowest common ancestor taxonomy`
>    - *"Peptides input format"*: `tabular`
>        - {% icon param-file %} *"Tabular Input Containing Peptide column"*: `output` (Input dataset)
>        - *"Select column with peptides"*: `c1`
>    - *"Match input peptides by"*: `Match to the full input peptide`
>    - *"Choose outputs"*: `Tabular with one line per peptide` `JSON Taxomony Tree` `Peptide GO terms in normalized tabular` `Peptide InterPro entries in normalized tabular` `Peptide EC terms in normalized tabular` `JSON EC Coverage Tree`
>
{: .hands_on}

![Data-Interpretation with Unipept]({% link topics/proteomics/images/clinical-mp/clinical-mp-data-interpretation-figure2.jpg %})

## Extraction of Microbial Sequences

> <hands-on-title> Extract Microbial sequences with Select </hands-on-title>
>
> 1. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `output` (Input dataset)
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_HUMAN)|(_REVERSED)|(REV_)|(CON)|(con)`
>    - *"Keep header line"*: `Yes`
>
>
{: .hands_on}


> <hands-on-title> Select sequences matching "HUMAN" </hands-on-title>
>
> 1. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `output` (Input dataset)
>    - *"the pattern"*: `(_HUMAN)`
>    - *"Keep header line"*: `Yes`
>
>
{: .hands_on}

> <hands-on-title> Select out reverse and contaminants </hands-on-title>
>
> 1. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `out_file1` (output of **Select** {% icon tool %})
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_REVERSED)|(REV_)|(CON)|(con)`
>    - *"Keep header line"*: `Yes`
>
>
{: .hands_on}


# MSstats TMT
MSstats TMT(Tandem Mass Tag) is a computational tool designed for the robust statistical analysis of mass spectrometry-based quantitative proteomics data using TMT labeling. TMT is a widely used method for multiplexed quantitative proteomics, enabling simultaneous identification and quantification of proteins across multiple samples. MSstats TMT plays a crucial role in this process by providing a statistical framework for analyzing TMT data, and facilitating accurate and reliable protein abundance measurements. The tool offers a range of features, including quality control, normalization, and statistical modeling, allowing researchers to identify differentially expressed proteins with confidence. MSstats TMT is particularly valuable in large-scale studies where quantifying protein expression across multiple conditions is essential for understanding complex biological processes. Its application contributes to advancing our understanding of proteomic changes in response to various experimental conditions or perturbations.

## Statistical Analysis of Microbial proteins with **MSstatsTMT**

> <hands-on-title> MSstatsTMT </hands-on-title>
>
> 1. {% tool [MSstatsTMT](toolshed.g2.bx.psu.edu/repos/galaxyp/msstatstmt/msstatstmt/2.0.0+galaxy1) %} with the following parameters:
>    - *"Input Source"*: `MaxQuant`
>        - {% icon param-file %} *"evidence.txt - feature-level data"*: `output` (Input dataset)
>        - {% icon param-file %} *"proteinGroups.txt"*: `out_file1` (output of **Select Microbial** {% icon tool %})
>        - {% icon param-file %} *"annotation.txt"*: `output` (Input dataset)
>    - In *"Plot Output Options"*:
>        - *"Select protein IDs to draw plots"*: `generate all plots for each protein`
>    - *"Compare Groups"*: `Yes`
>        - *"Use comparison matrix?"*: `Yes`
>            - {% icon param-file %} *"Comparison Matrix"*: `output` (Input dataset)
>        - *"Select outputs"*: `Group Comparison` `MSstats Volcano Plot` `MSstats Comparison Plot`
>        - In *"Comparison Plot Options"*:
>            - *"Select protein IDs to draw plots"*: `generate all plots for each protein`
>            - *"Select comparisons to draw plots"*: `Generate all plots for each comparison`
>        - *"Select outputs"*: `MSstatsTMT summarization log` `MSstatsTMT summarization MSstats` `Protein abundance`
>
>
{: .hands_on}


## Statistical Analysis of Human proteins with **MSstatsTMT**

> <hands-on-title> MSstatsTMT</hands-on-title>
>
> 1. {% tool [MSstatsTMT](toolshed.g2.bx.psu.edu/repos/galaxyp/msstatstmt/msstatstmt/2.0.0+galaxy1) %} with the following parameters:
>    - *"Input Source"*: `MaxQuant`
>        - {% icon param-file %} *"evidence.txt - feature-level data"*: `output` (Input dataset)
>        - {% icon param-file %} *"proteinGroups.txt"*: `out_file1` (output of **Select HUMAN** {% icon tool %})
>        - {% icon param-file %} *"annotation.txt"*: `output` (Input dataset)
>    - In *"Plot Output Options"*:
>        - *"Select protein IDs to draw plots"*: `generate all plots for each protein`
>    - *"Compare Groups"*: `Yes`
>        - *"Use comparison matrix?"*: `Yes`
>            - {% icon param-file %} *"Comparison Matrix"*: `output` (Input dataset)
>        - *"Select outputs"*: `Group Comparison` `MSstats Volcano Plot` `MSstats Comparison Plot`
>        - In *"Comparison Plot Options"*:
>            - *"Select protein IDs to draw plots"*: `generate all plots for each protein`
>            - *"Select comparisons to draw plots"*: `Generate all plots for each comparison`
>        - *"Select outputs"*: `MSstatsTMT summarization log` `MSstatsTMT summarization MSstats` `Protein abundance`
>
>
{: .hands_on}

The MSstats output typically includes essential information such as estimated fold changes, p-values, and other statistical measures that help identify differentially expressed proteins across experimental conditions or sample groups. It provides a clear picture of the variations in protein expression levels, aiding in the prioritization of biologically relevant targets. MSstats output also often includes visualizations and quality control metrics, making it a valuable resource for researchers in their quest to extract meaningful insights from complex proteomic datasets and understand the underlying biology of their experiments.
Example of our data interpretation:

![Data-Interpretation results with MSstats]({% link topics/proteomics/images/clinical-mp/clinical-mp-data-interpretation-figure3.jpg %})


# Conclusion
With the completion of this tutorial, you have successfully completed the clinical metaproteomics tutorials.

In conclusion, clinical metaproteomics tutorials represent an essential gateway to harnessing the power of advanced proteomic techniques in the realm of clinical research and applications. These bioinformatics tutorials serve as valuable guides for understanding the intricacies of metaproteomic workflows, from  data analysis to interpretation. By providing comprehensive knowledge and practical insights, they equip researchers and clinicians with the tools necessary to explore the rich diversity of the microbiome and its impact on health and disease or environment. As metaproteomic techniques continue to evolve and integrate with clinical practice, we hope these tutorials will be instrumental in shaping clinical research.

tools_json:
["Unipept", "toolshed.g2.bx.psu.edu/repos/galaxyp/unipept/unipept/4.5.1", "Select", "Grep1", "MSstatsTMT", "toolshed.g2.bx.psu.edu/repos/galaxyp/msstatstmt/msstatstmt/2.0.0+galaxy1"]


topic: microbiome
tutorial: metaplasmidome_query
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: []
content_hash: ab73e40a8fac3b7c91f4019ccf9b95aa
last_modified: 2026-05-27T09:01:49.358499
zenodo_link: https://zenodo.org/records/14501567
workflows_json: []
title:
Query an annotated mobile genetic element database to identify and annotate genetic elements (e.g. plasmids) in metagenomics data

url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/metaplasmidome_query/tutorial.html

content:
Identifying and annotating mobile genetic elements (MGE) in metagenomic data can be tricky. To facilitate the process, we can use existing and custom annotated mobile genetic element database.

The *metaplasmidome* refers to the collection of all plasmids present within a given environment, typically identified through metagenomic sequencing. Plasmids are extrachromosomal genetic elements that often carry genes associated with antibiotic resistance, virulence, or metabolic functions, making them crucial for microbial adaptability. In the context of metagenomics, plasmids are identified alongside chromosomal DNA.

A common step in metaplasmidome analysis is searching sequencing reads against known plasmid databases to detect plasmid sequences within a metagenome, allowing researchers to map the diversity and abundance of plasmids in various environments. 

In this tutorial, we use a metaplasmidome database built from public air metagenomes and query it with assembled air metagenome data. A similar approach can be used for other mobile genetic elements.

> <details-title>How was built the air metaplasmidome database?</details-title>
>
> The air metaplasmidome is available from Zenodo ({% cite debroas_2024_11124657 %}) and was built from metagenomic data selected in Web of Science (Clarivate) on October 2022 using keywords: `txid655179[Organism:noexp] AND metagenome [Filter]; AIR Metagenome; Air microbiome; Troposphere; Aerosol; Atmosphere`. Data were manually curated to remove sequencing originated from metabarcoding data (i.e., 16S). The assembled data supplied by MetaSUB consortium ({% cite danko2021global %}) when available was used for air metagenome in the built environments. 
>
> Plasmid contents were predicted using the assembled data. Metagenomes sequencing by Illumina (paired-illumina reads) were assembled by using MEGAHIT 1.2.9 with metalarge option ({% cite li2015 %}) after cleaning data with bbduk2 (qtrim=rl trimq=28 minlen=25 maq=20 ktrim=r k=25 mink=11 and a list of adaptators to remove) from [bbtools suite](https://jgi.doe.gov/data-and-tools/software-tools/bbtools/)  
>
> Plasmids were predicted for each assembling by using PlasSuite scripts describing in-depth in Hilpert et al. {% cite hilpert2021reconstruction %} {% cite hennequin2022plasmidome %} and available on [GitHub](https://github.com/meb-team/PlasSuite/). Briefly, contigs were analyzed using both reference-based and reference-free approaches. The databases employed included those for chromosomes (archaea and bacteria) and plasmids from NCBI, as well as the MOB-suite tool ({% cite robertson2018mob %}), SILVA ({% cite quast2012silva %}) and phylogenetic markers harbored by chromosomes ({% cite wu2013systematic %}). Two reference-free methods were applied to contigs that were not affiliated with chromosomes (discarded) or plasmids (retained in the first step): PlasFlow ({% cite krawczyk2018plasflow%}) and PlasClass ({% cite pellow2020plasclass %}). Viruses were removed by using [viralVerify](https://github.com/ablab/viralVerify) ({% cite antipov2020metaviral %}) that provides in parallel provide plasmid/non-plasmid classification.
> 
> Eukaryotes contaminants were removed by aligning the sequences against NT databases and human chromosomes (GRCh38) with minimap2 with -x asm5 option ({% cite li2018minimap2 %}). Contigs mapping with an identity of 95% and a coverage of 80% were removed.  the final plasmidome set was clustered by mmseqs ({% cite mirdita2019mmseqs2 %}) with 80% of coverage and 90% of identity (--min-seq-id 0.90 -c 0.8 --cov-mode 1 --cluster-mode 2 --alignment-mode 3 --kmer-per-seq-scale 0.2). 
{: .details}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Galaxy and data preparation

Any analysis should get its own Galaxy history. So let's start by creating a new history and import the data (plasmidome database and query metagenomes) into it.

> <hands-on-title>Prepare Galaxy and data</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 3. Import the metaplasmidome reference database from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library
>
>    ```
>    https://zenodo.org/records/14501567/files/air_metaplasmidome.fasta.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 4. Rename `Air plasmidome database`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 5. Import the reads to query against the reference database from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library
>
>    ```
>    https://zenodo.org/records/14501567/files/air_metagenome_assemblies.fasta.gz
>    ```
>
> 6. Rename `Air metagenomes`
{: .hands_on}


> <question-title></question-title>
>
> 1. How many reads are in the query dataset?
> 2. How many plasmid sequences are in the metaplasmidome database?
>
> > <solution-title></solution-title>
> >
> > 1. 16,637 sequences
> > 2. 674,495 sequences
> {: .solution}
{: .question}


# Read mapping against the metaplasmidome database

> <hands-on-title>Map reads against metaplasmidome database</hands-on-title>
>
> 1. {% tool [Map with minimap2](toolshed.g2.bx.psu.edu/repos/iuc/minimap2/minimap2/2.28+galaxy0) %} with the following parameters:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>        - {% icon param-file %} *"Use the following dataset as the reference sequence"*: `metaplasmidome database`
>    - *"Single or Paired-end reads"*: `Single`
>        - {% icon param-file %} *"Select fastq dataset"*: `query`
>        - *"Select a profile of preset options"*: `Long assembly to reference mapping (-k19 -w19 -A1 -B19 -O39,81 -E3,1 -s200 -z200 --min-occ-floor=100). Typically, the alignment will not extend to regions with 5% or higher sequence divergence. Only use this preset if the average divergence is far below 5%. (asm5)`
>    - In *"Set advanced output options"*:
>        - *"Select an output format"*: `PAF`
>
{: .hands_on}

PAF is the default output format of minimap2. It is TAB-delimited with each line consisting of the following predefined fields:
1. Query sequence name
2. Query sequence length
3. Query start coordinate (0-based)
4. Query end coordinate (0-based)
5. `+` if query/target on the same strand and `-` if opposite
6. Target sequence name
7. Target sequence length
8. Target start coordinate on the original strand
9. Target end coordinate on the original strand
10.	Number of matching bases in the mapping
11.	Number bases, including gaps, in the mapping
12.	Mapping quality

> <question-title></question-title>
>
> 1. How many lines are in the file?
> 2. Can a query sequence be found several times in the target?
> 3. Are all alignments of good quality?
>
> > <solution-title></solution-title>
> >
> > 1. 29,796 lines
> > 2. There are 16,637 query sequences. Several query sequences are found several times in the file: SRR17300492_75807 is found 16 times. It means they have been mapped to several locations or sequences in the target.
> > 3. The first alignment (SRR17300492_75807 mapping on SRR17300667-707) has a score of 60, the highest Phred score. The third alignment has a score of 0, so not good.
> >
> {: .solution}
>
{: .question}

# Exploration of the mapping results

Let's look at the distribution of the mapping score (column 12)

> <hands-on-title>Plot score distribution</hands-on-title>
>
> 1. Change datatype to tabular
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
> 
> 2. {% tool [Cut columns from a table](Cut1) %} with:
>    - *"Cut columns"*: `c12`
>    - *"Delimited by"*: `tab`
>    - {% icon param-file %} *"From"*: Output of **Map with minimap2** {% icon tool %}
>
> 2. {% tool [Histogram with ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_histogram/ggplot2_histogram/3.4.0+galaxy0) %} with the following parameters:
>    - *"Input should have column headers - these will be the columns that are plotted * "*: Output of **Cut** {% icon tool %}
>    - *"Label for x axis"*: `Score`
>    - *"Label for y axis"*: `Distribution`
>    - In *"Advanced Options"*:
>      - *"Legend options"*: `Hide legend`
>
{: .hands_on}

![Distribution of mapping quality score (on x-axis, value between 0 and 60) with density (y-axis)](./images/mapping_score_histogram.png)

> <question-title></question-title>
>
> Are all alignments of good quality?
>
> > <solution-title></solution-title>
> >
> > There is a pic at 60 but many alignments have a score below 40.
> >
> {: .solution}
>
{: .question}

We should remove alignments with a score below 40. Before that, let's look at the plasmid coverage. We first need to compute it, i.e. the ratio between the number of matching bases in the mapping (Column 10) and plasmid length (target sequence length - Column 7).

> <hands-on-title>Compute plasmid coverage</hands-on-title>
> 1. {% tool [Cut columns from a table](Cut1) %} with:
>    - *"Cut columns"*: `c1-c12`
>    - *"Delimited by"*: `tab`
>    - {% icon param-file %} *"From"*: Output of **Map with minimap2** {% icon tool %}
>
> 2. {% tool [Compute on rows](toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1) %} with the following parameters:
>    - *"Input file"*: Output of **Cut** {% icon tool %}
>    - In *"Expressions"*:
>      - *"Add expression"*: `float(c10)/float(c7)`
>
> 3. Rename `Mapping stats + plasmid coverage`
{: .hands_on}

A new column has been added (column 13) with the plasmid coverage. Let's now plot its distribution.

> <hands-on-title>Plot plasmid coverage distribution</hands-on-title>
> 1. {% tool [Cut columns from a table](Cut1) %} with:
>    - *"Cut columns"*: `c13`
>    - *"Delimited by"*: `tab`
>    - {% icon param-file %} *"From"*: `Mapping stats + plasmid coverage`
>
> 2. {% tool [ Histogram with ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_histogram/ggplot2_histogram/3.4.0+galaxy0) %} with the following parameters:
>    - *"Input should have column headers - these will be the columns that are plotted * "*: Output of **Cut** {% icon tool %}
>    - *"Label for x axis"*: `Plasmid coverage`
>    - *"Label for y axis"*: `Distribution`
>    - *"Bin width for plotting"*: `0.1`
>    - In *"Advanced Options"*:
>      - *"Legend options"*: `Hide legend`
>
{: .hands_on}

![Distribution of plasmid coverage (on x-axis, value between 0 and 1) with density (y-axis)](./images/plasmid_coverage_histogram.png)

> <question-title></question-title>
>
> 1. What is the distribution of the plasmid coverage?
> 2. What could be a good threshold to filter?
>
> > <solution-title></solution-title>
> >
> > 1. There are 2 peaks: one around 0 (i.e. no plasmid coverage) that slowly decreases until 0.8 and a pic at 1 (full plasmid coverage)
> > 2. 0.8 seems to be a breaking point and could be a good value to filter.
> >
> {: .solution}
>
{: .question}

# Filtering

We will now filter the alignment to keep only the ones with a plasmid coverage (column 13) of at least 0.8, i.e. a read mapping to a plasmid covering at least 80% of the plasmid.

> <hands-on-title>Filter alignments based on plasmid coverage</hands-on-title>
>
> 1. {% tool [Filter data on any column using simple expressions](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `Mapping stats + plasmid coverage`
>    - *"With following condition"*: `c13>=0.8`
>
> 2. Rename `Alignments with plasmid coverage >= 0.8`
{: .hands_on}

> <question-title></question-title>
>
> 1. How many lines have been kept?
> 2. Which percentage of lines does that correspond to?
> 3. What does the distribution of the mapping score look like for these alignments?
> 4. Is there any extra filter we should do on the data?
>
> > <solution-title></solution-title>
> >
> > 1. 5,577 (over the 29,796)
> > 2. 18.73%
> > 3. To plot the distribution of the mapping score for filtered alignments, we need to run the series of tools as done earlier:
> >    
> >    1. {% tool [Cut columns from a table](Cut1) %} with:
> >        - *"Cut columns"*: `c12`
> >        - *"Delimited by"*: `tab`
> >        - {% icon param-file %} *"From"*: `Alignments with plasmid coverage >= 0.8`
> >
> >     2. {% tool [Histogram with ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_histogram/ggplot2_histogram/3.4.0+galaxy0) %} with the following parameters:
> >        - *"Input should have column headers - these will be the columns that are plotted * "*: Output of **Cut** {% icon tool %}
> >        - *"Label for x axis"*: `Score`
> >        - *"Label for y axis"*: `Distribution`
> >        - In *"Advanced Options"*:
> >          - *"Legend options"*: `Hide legend`
> >    
> >     ![Distribution of mapping quality score (on x-axis, value between 0 and 60) with density (y-axis)](./images/mapping_score_histogram_after_coverage_filtering.png)
> >
> >     Most of the alignments are of good quality (pic at 60). There is also a pic at 0, meaning that some alignments are of bad quality
> >    
> > 4. It might be good to add a filter on the score.
> {: .solution}
>
{: .question}

After filtering on the plasmid coverage, we also add a filter on the score to be sure that we keep only the best alignments (i.e. a score - column 12 - of at least 40).

> <hands-on-title>Filter alignments based on score</hands-on-title>
>
> 1. {% tool [Filter data on any column using simple expressions](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `Alignments with plasmid coverage >= 0.8`
>    - *"With following condition"*: `c12>=40`
>
> 2. Rename `Alignments with plasmid coverage >= 0.8 and score >= 40`
{: .hands_on}

> <question-title></question-title>
>
> 1. How many lines have been kept?
> 2. Which percentage of lines does that correspond to?
>
> > <solution-title></solution-title>
> >
> > 1. 5,249 (over the 5,577)
> > 2. 94.12%
> {: .solution}
>
{: .question}

# Extract sequences mapping to plasmids

Let's now extract the sequences mapping to plasmids (coverage higher than 80%).

First, we need to extract the names of the reads (in `air metagenomes`) mapping to plasmids in the metaplasmidome database. For that, we need to extract column 1 (query names, i.e. names of reads in `Air metagenomes`) and 6 (reference names, i.e. names of sequences in `Air plasmidome database`).

> <hands-on-title>Get sequences matching to plasmids</hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1,c6`
>    - {% icon param-file %} *"From"*: `Alignments with plasmid coverage >= 0.8 and score >= 40`
>
> 2. Rename to `Names for alignments with plasmid coverage >= 0.8 and score >= 40`
>
> 2. {% tool [Sort](sort1) %} with the following parameters:
>    - {% icon param-file %} *"Sort Dataset"*: output of **Cut** {% icon tool %}
>    - *"on column"*: `c2`
{: .hands_on}

Second, we need to convert the `Air metagenomes` FASTA file to a tabular so we can join it with `Names for alignments with plasmid coverage >= 0.8 and score >= 40`

> <hands-on-title>Get query sequence as a table</hands-on-title>
>
> 1. {% tool [Convert FASTA to Tabular](CONVERTER_fasta_to_tabular) %} with the following parameters:
>    - {% icon param-file %} *"Fasta file"*: `Air metagenomes`
>
> 2. Rename to `Air metagenome as table`
{: .hands_on}

Let's now join the 2 datasets on the names of the reads in `air metagenomes` so column 1 in `Names for alignments with plasmid coverage >= 0.8 and score >= 40` and column 1 in `Air metagenome as table`.

> <hands-on-title>Get sequences matching to plasmids</hands-on-title>
>
> 1. {% tool [Join two Datasets](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: output of **Sort** {% icon tool %}
>    - *"using column"*: `Column: 1`
>    - {% icon param-file %} *"with"*: output of **Convert FASTA to Tabular** {% icon tool %}
>    - *"and column"*: `Column: 1`
>    - *"Fill empty columns"*: `No`
>
{: .hands_on}

We have now a table with 4 columns: names of the reads in `air metagenomes`, names of sequences in the reference database, names of the reads in `air metagenomes`, sequence of reads in `air metagenomes`. We will now create 2 outputs:
- A table with metaplasmidome database sequence name, metagenomic sequence names and the metagenomic sequences
- A FASTA file with metagenomic sequences

## Create the table with air metagenomes identified as plasmids

Let's remove the duplicated column (column 3) and reorganize the columns

> <hands-on-title>Keep non duplicated columns</hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c2,c1,c4`
>    - {% icon param-file %} *"From"*: output of **Join two Datasets** {% icon tool %}
{: .hands_on}

Let's now add column names to the generated table.

> <hands-on-title>Add column names</hands-on-title>
>
> 1. {% tool [Add Header](toolshed.g2.bx.psu.edu/repos/estrain/add_column_headers/add_column_headers/0.1.3) %} with the following parameters:
>    - *"List of Column headers"*: `Metaplasmidome database sequence name,Metagenomic sequence name,Metagenomic sequence`
>    - {% icon param-file %} *"Data File (tab-delimted)"*: output of **Replace Text** {% icon tool %}
>
>
> 3. Rename the output to `Air metagenomes identified as plasmids`
{: .hands_on}

## Create the FASTA file with air metagenomes identified as plasmids

We can now generate a FASTA file with the identified sequences.

From the output of **Join two Datasets** {% icon tool %}, let's keep only metagenomic sequence names and the metagenomic sequences and remove duplicated sequences

> <hands-on-title>Keep unique metagenomic sequences</hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1,c4`
>    - {% icon param-file %} *"From"*: output of **Join two Datasets** {% icon tool %}
> 2. {% tool [Sort](sort1) %} with the following parameters:
>    - {% icon param-file %} *"Sort Dataset"*: output of **Cut** {% icon tool %}
>    - *"on column"*: `Column: 1`
>    - *"with flavor"*: `Alphabetical sort`
> 3. {% tool [Unique occurrences of each record](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sorted_uniq/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to scan for unique values"*: output of **Sort** {% icon tool %}
{: .hands_on}

> <question-title></question-title>
>
> How many lines have been kept?
>
> > <solution-title></solution-title>
> >
> > 4,055 (over 5,249)
> >
> {: .solution}
>
{: .question}

We have now a table with unique metagenomic sequences. Let's transform it into a FASTA file.

> <hands-on-title>Convert to a FASTA file</hands-on-title>
>
> 1. {% tool [Tabular-to-FASTA](toolshed.g2.bx.psu.edu/repos/devteam/tabular_to_fasta/tab2fasta/1.1.1) %} with the following parameters:
>    - {% icon param-file %} *"Tab-delimited file"*: output of **Unique occurrences of each record** {% icon tool %}
>    - *"Title column(s)"*: `Column: 1`
>    - *"Sequence column"*: `Column: 2`
>
> 2. Rename the output to `Air metagenome sequences identified as plasmids`
{: .hands_on}

# Annotate features on the identified plasmids
 
Let's now annotate features on the identified plasmids. For that, we will use the annotation of the air metaplasmidome sequences that have been done with Prokka (citation) and assume that the annotations are similar for the identified plasmids.

We will import the GFF generated by Prokka.

> <hands-on-title>Import the GFF with metaplasmidome reference database annotation</hands-on-title>
>
> 1. Import the metaplasmidome reference database annotation from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>
>    ```
>    https://zenodo.org/records/14501567/files/air_metaplasmidome_annotations.gff
>    ```
>
> 2. Inspect it.
{: .hands_on}

This file is a GFF: it describes genes and other features of DNA, RNA and protein sequences. It is a tab-delimited file with 9 fields per line:

1. **seqid**: The name of the sequence where the feature is located. 
2. **source**: The algorithm or procedure that generated the feature. 
3. **type**: The feature type name, like "gene" or "exon". 
4. **start**: Genomic start of the feature, with a 1-base offset. 
5. **end**: Genomic end of the feature, with a 1-base offset. 
6. **score**: Numeric value that generally indicates the confidence of the source in the annotated feature. 
7. **strand**: Single character that indicates the strand of the feature.
8. **phase**: Phase of CDS features.
9. **attributes**: A list of tag-value pairs separated by a semicolon with additional information about the feature. 

The **seqid** corresponds here to the ID of the sequences in the metaplasmidome reference database. So to filter, we need to compare `Metaplasmidome database sequence name` in `Air metagenomes identified as plasmids`
to **seqid** by joining the 2 datasets on column 1.

The file has with ~85 Million lines but many are comments (lines starting with `##`).

> <question-title></question-title>
>
> How many features are in the GFF file?
>
> > <solution-title></solution-title>
> >
> > 2,951,015 features
> >
> > To get this number, we run {% tool [Count GFF Features](toolshed.g2.bx.psu.edu/repos/devteam/count_gff_features/count_gff_features/0.2) %} with the following parameters:
> >  - {% icon param-file %} *"GFF Dataset to Filter"*: imported GFF
> >
> {: .solution}
>
{: .question}

Let's now filter the GFF to keep only information related to the sequences matching to plasmids. 
For that, we join the GFF on the SeqID column (column 1) with the `Air metagenomes identified as plasmids` file on the ` Metaplasmidome database sequence name` (Column 1)

> <hands-on-title>Extract information about the sequences matching to plasmids</hands-on-title>
>
> 1. {% tool [Join two Datasets](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: imported GFF
>    - *"using column"*: `Column: 1`
>    - {% icon param-file %} *"with"*: `Air metagenomes identified as plasmids`
>    - *"and column"*: `Column: 1`
>    - *"Fill empty columns"*: `No`
>
> 2. Inspect the generated file
{: .hands_on}

> <question-title></question-title>
>
> 1. How many lines have been kept?
> 2. Why is there more lines than in the `Air metagenomes identified as plasmids` file?
> 2. What are the columns? 
> 3. Which columns should we keep if we want to keep the Metagenomic sequence name, the feature and the attributes?
>
> > <solution-title></solution-title>
> >
> > 1. 26k+ lines
> > 2. Some sequences (e.g. SRR17300493-2380) have several lines: several features (CDS, rRNA, etc) annotated on it
> > 3. The 9 columns of the GFF file + the 3 columns of the `Air metagenomes identified as plasmids` file
> > 4. Columns to keep:
> >    1. Metagenomic sequence name (Column 11)
> >    2. Feature (Column 3)
> >    3. Attributes (Column 9)
> {: .solution}
>
{: .question}

Let's now cut the columns.

> <hands-on-title>Cut and filter </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c11,c3,c9`
>    - {% icon param-file %} *"From"*: output of **Join two Datasets** {% icon tool %}
>
> 2. Inspect the generated file
{: .hands_on}

> <question-title></question-title>
>
> What are the different identified features?
>
> > <solution-title></solution-title>
> >
> > Using {% tool [Group data by a column](Grouping1) %} to group and count on 2nd column, we find 26,329 CDS and 307 tRNA.
> >
> {: .solution}
>
{: .question}

## Extract the CDS

Let's filter to keep only the CDS and extract the gene names.

> <hands-on-title>Keep CDS</hands-on-title>
>
> 1. {% tool [Filter](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: output of **Cut** {% icon tool %}
>    - *"With following condition"*: `c2=='CDS'`
>
> 2. Inspect the generated file
{: .hands_on}

We have now a file with 26,329 lines. The 2nd column is not useful so we will remove it. Column 3 (**atributes** in GFF) lists tag-value pairs separated by a semicolon with additional information about the feature.
The first lines seems to be hypothetical proteins. If we scroll down, we can find some annotated genes (with `gene` keyword).

It would be good to create a tabular file with:
1. Metagenomic sequence name
2. Gene ID
3. Gene name
4. Gene product

As not all genes are annotated (with `gene` keyword), we first need to split the file between hypothetical proteins and annotated genes and process the two generated files independently.

> <hands-on-title>Keep CDS</hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1,c3`
>    - {% icon param-file %} *"From"*: output of **Filter** {% icon tool %}
>
> 2. {% tool [Filter by keywords and/or numerical value](toolshed.g2.bx.psu.edu/repos/proteore/proteore_filter_keywords_values/MQoutputfilter/2021.04.19.1) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: output of last **Cut** {% icon tool %}
>    - *"Does file contain header? "*: `No`
>    - In *Filter by keywords*:
>      - *"Column number on which to apply the filter"*: `c2`
>      - In `Enter keywords`:
>        - `copy/paste`
>        - *"Copy/paste keywords to find (keep or discard)"*: `gene=`
>
> 2. Inspect the generated files
{: .hands_on}

> <question-title></question-title>
>
> 1. How many lines have been kept (non-hypothetical proteins) and how many have been discarded (hypothetical proteins)?
> 2. Which information do we have for the 2nd identified gene?
>
> > <solution-title></solution-title>
> >
> > 1. There are:
> > - 25,001 hypothetical proteins CDS (lines in the discarded lines file)
> > - 1,329 annotated CDS (lines in the other file)
> > 
> > 2. For the 2nd gene, the column 3 is `ID=BAFOEJEB_01977;eC_number=1.8.5.7;Name=yqjG_1;dbxref=COG:COG0435;gene=yqjG_1;inference=ab initio prediction:Prodigal:2.6,similar to AA sequence:UniProtKB:P42620;locus_tag=BAFOEJEB_01977;product=Glutathionyl-hydroquinone reductase YqjG`, which mean:
> >    - Gene name is `yqjG_1`
> >    - The product is `Glutathionyl-hydroquinone reductase YqjG`
> >
> {: .solution}
>
{: .question}

Let's extract gene ID, gene name and the product in different columns from the annotated gene output and add a header to the file.

> <hands-on-title>Prepare the annotated CDS file</hands-on-title>
>
> 1. Change annotated CDS output of **Filter by keywords** {% icon tool %} datatype to tabular
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
> 2. {% tool [Replace Text in a specific column](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: output of **Filter by keywords** {% icon tool %}
>    - In *Replacement*:
>      - *"in column"*: `Column: 2`
>      - *"Find pattern"*: `ID=([^;]*);.*;gene=([^;]*).*;product=([^;]*).*`
>      - *"Replace with"*: `\\1\t\\2\t\\3`
>
> 3. {% tool [Add Header](toolshed.g2.bx.psu.edu/repos/estrain/add_column_headers/add_column_headers/0.1.3) %} with the following parameters:
>    - *"List of Column headers"*: `Metagenomic sequence name,Gene ID,Gene name,Gene product`
>    - {% icon param-file %} *"Data File (tab-delimited)"*: output of **Replace Text** {% icon tool %}
>
{: .hands_on}

Let's now prepare the hypothetical protein CDS file

> <hands-on-title>Prepare the hypothetical CDS gene file</hands-on-title>
>
> 1. Change hypothetical CDS gene output of **Filter by keywords** {% icon tool %} datatype to tabular
>
> 2. {% tool [Replace Text in a specific column](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: output of **Filter by keywords** {% icon tool %}
>    - In *Replacement*:
>      - *"in column"*: `Column: 2`
>      - *"Find pattern"*: `ID=([^;]*);.*`
>      - *"Replace with"*: `\\1\t\t`
>
{: .hands_on}

We can now merge both files.

> <hands-on-title>Concatenate the annotated and hypothetical protein gene files</hands-on-title>
>
> 1. {% tool [Concatenate multiple datasets or collections](cat1) %} with the following parameters:
>    - {% icon param-file %} *"Concatenate Dataset"*: output of **Add Header** {% icon tool %}
>    - In *Dataset*:
>      - {% icon param-file %} *"Select"*: output of last **Replace Text** {% icon tool %}
>
> 2. Rename `CDS in metagenomes identified as plasmids`
{: .hands_on}

## Add KO and PFAM annotation

Let's expand annotation with **[KO](https://www.genome.jp/kegg/ko.html) (KEGG Orthology)** ({% cite kanehisa2016kegg%}), a database of molecular functions
represented in terms of functional orthologs, and **[PFAM](http://pfam.xfam.org/)** ({% cite mistry2021pfam %}), a large collection of protein families. The files are 

> <hands-on-title>Import KO and PFAM annotations</hands-on-title>
>
> 1. Import the KO and PFAM annotations from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>
>    ```
>    https://zenodo.org/records/14501567/files/KOsignificatifNR.tsv
>    https://zenodo.org/records/14501567/files/pfamsignificatifNR.tsv
>    ```
>
> 2. Inspect both files
{: .hands_on}

Both are tabular files with several columns including the gene ID, KO/PFAM ID, and some extra annotation.
Let's now join the files with `Annotated genes in air metagenomes sequences identified as plasmids` to extend the annotations using gene ID.

> <hands-on-title>Join with KO and PFAM annotation files</hands-on-title>
>
> 1. {% tool [Join two Datasets](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: `Annotated genes in air metagenomes sequences identified as plasmids`
>    - *"using column"*: `Column: 2`
>    - {% icon param-file %} *"with"*: imported KO file
>    - *"and column"*: `Column: 2`
>    - *"Keep lines of first input that do not join with second input"*: `Yes`
>
> 2. Inspect the generated file
>
> 3. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1-c4,c7,c11`
>    - {% icon param-file %} *"From"*: output of **Join** {% icon tool %}
>
> 4. {% tool [Join two Datasets](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: output of **Cut** {% icon tool %}
>    - *"using column"*: `Column: 2`
>    - {% icon param-file %} *"with"*: imported PFAM file
>    - *"and column"*: `Column: 1`
>    - *"Keep lines of first input that do not join with second input"*: `Yes`
>
> 5. Inspect the generated file
>
> 6. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1-c6,c9,c10,c25`
>    - {% icon param-file %} *"From"*: output of **Join** {% icon tool %}
>
> 7. {% tool [Select last lines from a dataset (tail)](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_tail_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Text file"*: output of **Cut** {% icon tool %}
>    - *"Operation"*: `Keep everything from lines on`
>    - *"Number of lines"*: `2`
>
> 8. {% tool [Add Header](toolshed.g2.bx.psu.edu/repos/estrain/add_column_headers/add_column_headers/0.1.3) %} with the following parameters:
>    - *"List of Column headers"*: `Metagenomic sequence name,Gene ID,Gene name,Gene product,KO ID,KO annotation,PFAM name,PFAM ID,PFAM annotation`
>    - {% icon param-file %} *"Data File (tab-delimited)"*: output of **Cut** {% icon tool %}
>
> 9. Rename `CDS in metagenomes identified as plasmids + KO + PFAM`
{: .hands_on}

Let's look at the KO and PFAM statistics.

> <question-title></question-title>
>
> 1. How many genes have been extended with KO information?
> 2. How many different KO are found?
> 5. Which KO is the most found?
> 3. How many genes have been extended with PFAM information?
> 4. How many different PFAM are found?
> 5. Which PFAM is the most found?
>
> > <solution-title></solution-title>
> >
> > 1. Using {% tool [Filter](Filter1) %} with `c5!='.'`, we find that 426 CDS (1.62%) have been extended with KO information.
> > 2. 277 KO ({% tool [Group data by a column](Grouping1) %} to group and count on 5th column)
> > 3. K14572 (Ribosome biogenesis in eukaryotes) is found 9 times ({% tool [Sort](sort1) %} by descending order on the **Group data by a column** output on column 2)
> > 3. 4,817 CDS (18.29%) have been extended with PFAM information ({% tool [Filter](Filter1) %} with `c8!='.'` on `CDS + KO + PFAM`)
> > 4. 410 PFAM ({% tool [Group data by a column](Grouping1) %} to group and count on 8th column)
> > 5. PF00961.22 (Cytochrome c oxidase subunit 1) is found 89 times ({% tool [Sort](sort1) %} by descending order)
> >
> {: .solution}
>
{: .question}

Let's extract an overview of the annotations per metagenomic sequence by grouping on the 1st column and counting the number of distinct values on gene ID, gene name, KO ID, PFAM ID.

> <hands-on-title>Annotation per metagenomic sequences</hands-on-title>
>
> 1. {% tool [Group data by a column](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `CDS + KO + PFAM`
>    - *"Group by column"*: `Column: 1`
>    - In *"Operation"*:
>       - In *"1: Operation"*:
>         - *"Type"*: `Count distinct`
>         - *"On column"*: `Column: 2`
>       - In *"2: Operation"*:
>         - *"Type"*: `Count distinct`
>         - *"On column"*: `Column: 3`
>       - In *"3: Operation"*:
>         - *"Type"*: `Count distinct`
>         - *"On column"*: `Column: 5`
>       - In *"4: Operation"*:
>         - *"Type"*: `Count distinct`
>         - *"On column"*: `Column: 8`
> 
> 2. {% tool [Select last lines from a dataset (tail)](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_tail_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Text file"*: output of **Cut** {% icon tool %}
>    - *"Operation"*: `Keep everything from lines on`
>    - *"Number of lines"*: `2`
>
> 3. {% tool [Add Header](toolshed.g2.bx.psu.edu/repos/estrain/add_column_headers/add_column_headers/0.1.3) %} with the following parameters:
>    - *"List of Column headers"*: `Metagenomic sequence name,Number of CDS,Number of annotated CDS,Number of associated KO,Number of associated PFAM`
>    - {% icon param-file %} *"Data File (tab-delimited)"*: output of **Select last** {% icon tool %}
>
> 4. Rename `CDS annotation overview per metagenomic sequences`
{: .hands_on}

# Conclusion

tools_json:
["Map with minimap2", "toolshed.g2.bx.psu.edu/repos/iuc/minimap2/minimap2/2.28+galaxy0", "Cut columns from a table", "Cut1", "Histogram with ggplot2", "toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_histogram/ggplot2_histogram/3.4.0+galaxy0", "Compute on rows", "toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1", "Filter data on any column using simple expressions", "Filter1", "Cut", "Sort", "sort1", "Convert FASTA to Tabular", "CONVERTER_fasta_to_tabular", "Join two Datasets", "join1", "Add Header", "toolshed.g2.bx.psu.edu/repos/estrain/add_column_headers/add_column_headers/0.1.3", "Unique occurrences of each record", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sorted_uniq/9.3+galaxy1", "Tabular-to-FASTA", "toolshed.g2.bx.psu.edu/repos/devteam/tabular_to_fasta/tab2fasta/1.1.1", "Count GFF Features", "toolshed.g2.bx.psu.edu/repos/devteam/count_gff_features/count_gff_features/0.2", "Group data by a column", "Grouping1", "Filter", "Filter by keywords and/or numerical value", "toolshed.g2.bx.psu.edu/repos/proteore/proteore_filter_keywords_values/MQoutputfilter/2021.04.19.1", "Replace Text in a specific column", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1", "Concatenate multiple datasets or collections", "cat1", "Select last lines from a dataset (tail)", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_tail_tool/9.3+galaxy1"]


topic: microbiome
tutorial: qiime2-cancer-microbiome-intervention
title: QIIME 2 Cancer Microbiome Intervention
difficulty: intermediate
time_estimation: 10H
tools_json: []
requirements_json: []
tags_json: ["metabarcoding", "16S", "taxonomic profiling", "diversity", "microgalaxy"]
content_hash: d41d8cd98f00b204e9800998ecf8427e
last_modified: 2026-05-15T22:28:14.189383
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/qiime2-cancer-microbiome-intervention/tutorial.html


topic: microbiome
tutorial: metatranscriptomics-short
title: Metatranscriptomics analysis using microbiome RNA-seq data (short)
difficulty: introductory
time_estimation: 3H
content: {% include topics/microbiome/tutorials/metatranscriptomics/content.md short=true %}
tools_json: []
requirements_json: []
tags_json: ["metatranscriptomics", "microgalaxy"]
content_hash: beed5d3e14894c43ff6cb10e5d043b66
last_modified: 2026-05-27T09:01:49.364135
zenodo_link: https://zenodo.org/record/4776250
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/metatranscriptomics-short/tutorial.html


topic: microbiome
tutorial: metagenomics-binning
title: Binning of metagenomic sequencing data
difficulty: intermediate
time_estimation: 2H
requirements_json: ["type: internal", "metagenomics-assembly"]
tags_json: ["binning", "metagenomics", "microgalaxy"]
content_hash: e47d6664e54c71648cc9994523827847
last_modified: 2026-05-27T09:01:49.357081
zenodo_link: https://zenodo.org/records/17661262
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/metagenomics-binning/tutorial.html

content:
Metagenomics is the study of genetic material recovered directly from environmental samples, such as soil, water, or gut contents, without the need for isolating or cultivating individual organisms. Metagenomics binning is a process used to classify DNA sequences obtained from metagenomic sequencing into discrete groups, or bins, based on their similarity to each other.

The goal of metagenomics binning is to assign DNA sequences to the organisms or taxonomic groups from which they originate, allowing for a better understanding of the diversity and functions of the microbial communities present in the sample. This is typically achieved through computational methods that include sequence similarity, composition, and other features to group the sequences into bins.

> <comment-title></comment-title>
> Before starting this tutorial, it is recommended to do the [**Metagenomics Assembly Tutorial**]({% link topics/microbiome/tutorials/metagenomics-assembly/tutorial.md %})
{: .comment}

## Binning approaches

There are several approaches to metagenomics binning, including:

- **Sequence composition-based binning**: This method is based on the observation that different genomes have distinct sequence composition patterns, such as GC content or codon usage bias. By analyzing these patterns in metagenomic data, sequence fragments can be assigned to individual genomes or groups of genomes.

- **Coverage-based binning**: This method uses the depth of coverage of sequencing reads to group them into bins. Sequencing reads that originate from the same genome are expected to have similar coverage, and this information can be used to identify groups of reads that represent individual genomes or genome clusters.

- **Hybrid binning**: This method combines sequence composition-based and coverage-based binning to increase the accuracy of binning results. By using multiple sources of information, hybrid binning can better distinguish closely related genomes that may have similar sequence composition patterns.

- **Clustering-based binning**: This method groups sequence fragments into clusters based on sequence similarity, and then assigns each cluster to a genome or genome cluster based on its sequence composition and coverage. This method is particularly useful for metagenomic data sets with high levels of genomic diversity.

- **Supervised machine learning-based binning**: This method uses machine learning algorithms trained on annotated reference genomes to classify metagenomic data into bins. This approach can achieve high accuracy but requires a large number of annotated genomes for training.

## Binning challenges

Metagenomic binning is a complex process that involves many steps and can be challenging due to several problems that can occur during the process. Some of the most common problems encountered in metagenomic binning include:

- **High complexity**: Metagenomic samples contain DNA from multiple organisms, which can lead to high complexity in the data.
- **Fragmented sequences**: Metagenomic sequencing often generates fragmented sequences, which can make it difficult to assign reads to the correct bin.
- **Uneven coverage**: Some organisms in a metagenomic sample may be more abundant than others, leading to uneven coverage of different genomes.
- **Incomplete or partial genomes**: Metagenomic sequencing may not capture the entire genome of a given organism, which can make it difficult to accurately bin sequences from that organism.
- **Horizontal gene transfer**: Horizontal gene transfer (HGT) can complicate metagenomic binning, as it can introduce genetic material from one organism into another.
- **Chimeric sequences**: Sequences that are the result of sequencing errors or contamination can lead to chimeric sequences, which can make it difficult to accurately bin reads.
- **Strain variation**: Organisms within a species can exhibit significant genetic variation, which can make it difficult to distinguish between different strains in a metagenomic sample.

## Common binners

There are different algorithms and tools performing metagenomic binning. The most widely used include:

* **MaxBin** ({% cite maxbin2015 %}): A popular de novo binning algorithm that uses a combination of sequence features and marker genes to cluster contigs into genome bins.
* **MetaBAT** ({% cite Kang2019 %}): Another widely used de novo binning algorithm that employs a hierarchical clustering approach based on tetranucleotide frequency and coverage information.
* **CONCOCT** ({% cite Alneberg2014 %}): A de novo binning tool that uses a clustering algorithm based on sequence composition and coverage information to group contigs into genome bins.
* **MyCC** ({% cite Lin2016 %}): A reference-based binning tool that uses sequence alignment to identify contigs belonging to the same genome or taxonomic group.
* **GroopM** ({% cite Imelfort2014 %}): A hybrid binning tool that combines reference-based and de novo approaches to achieve high binning accuracy.
* **SemiBin** ({% cite Pan2022 %}): A command-line tool for metagenomic binning with deep learning, handling both short and long reads.
* **Vamb** ({% cite nissen2021improved %}): An algorithm that uses variational autoencoders (VAEs) to encode sequence composition and coverage information.
* **ComeBin** ({% cite Wang2024COMEBin %}): A metagenomic binning tool that integrates both composition and abundance features with machine learning-based clustering to improve binning accuracy across complex microbial communities.

## So many options, what binner to use ?

Each of these binning methods has its own strengths and limitations, and the choice of a binning tool often depends on the characteristics of the metagenomic dataset and the research question. Practical guidance on which binner to use for specific datasets and environments can be drawn from benchmark studies such as {%cite NatureBinner2025%}.

![The image displays four heatmaps (labeled g, h, i, j) comparing the performance of various binning methods across different datasets related to human gut and activated sludge microbiomes. Each heatmap shows numerical values representing metrics for methods such as CONCOCT, MaxBin 2, MetaBAT 2, VAMB, CLMB, MetaDecoder, BinnY, MetaBinner, SemiBin 2, and COMEBin. The columns correspond to different sequence types: short co-assembly, short single, short multi-sample, long single, long multi-sample, hybrid single, and hybrid multi-sample. The color gradient from light to dark red indicates the magnitude of the values, with darker shades representing higher values. The heatmaps provide a visual comparison of how each method performs across these different sequence types and datasets.](./images/Binning_Benchmark.png "Benchmark of multiple binners on activated sludge and human gut microbiome ({%cite NatureBinner2025%})"){:width="60%"}

Another useful approach to investigate the performance of binners is to use simulated datasets. Therefore, the CAMI challenges ({%cite Meyer2022%}) were introduced. CAMI, which stands for Critical Assessment of Metagenome Interpretation, is an international community-driven initiative that organizes benchmarking challenges to objectively evaluate the performance of metagenomic tools. This includes the benchmarking of binners based on standardized, realistically simulated microbial communities that vary in complexity, strain diversity, and abundance.

![This image visually compares the performance of various metagenomic binning tools across multiple environments, including Marine GSA, Marine MA, Strain-madness GSA, Strain-madness MA, Plant-associated GSA, and Plant-associated MA. The top row (a) uses box plots to show the distribution of genome counts recovered from each environment, with sample sizes (n) indicated for each category. The middle section (b) features horizontal bar charts illustrating the number of genomes recovered by different binning tools—such as MetaBinner, UltraBinner, CONCOCT, MetaWRAP, Vamb, MaxBin, MetaBAT, and Autometa—specifically for the Marine GSA environment. Similarly, the bottom section (c) presents bar charts for the Marine MA environment, highlighting the number of genomes recovered by each tool. The image effectively summarizes the efficiency and effectiveness of these tools in different metagenomic contexts.](./images/CAMI_Binners.png "Benchmark of multiple binners on the CAMI datasets ({%cite Meyer2022%})"){:width="60%"}

A general approach is to perform binning using multiple binners that have shown good performance for the specific dataset, followed by bin refinement to generate an improved bin set that retains the best bins from the analysis.

Does using more binners always improve results? In practice, one must also consider computational resources and time constraints. Running many binners can be very time-consuming and resource-intensive, especially for large studies. In some cases, adding extra binners does not lead to a meaningful increase in bin quality, so the choice of binners should be made carefully. Overall, identifying the optimal combination of binners remains an active area of research, and clear, widely accepted guidelines are still being established.

## Bin refinement

There are also bin refinement tools, which can evaluate, combine, and improve the raw bins produced by primary binners such as MetaBAT2, CONCOCT, MaxBin2, or SemiBin. These tools help remove contamination, merge complementary bins, and recover higher-quality MAGs.

* **MetaWRAP** ({% cite Uritskiy2018 %}): A comprehensive metagenomic analysis pipeline that includes modules for quality control, assembly, binning (wrapping multiple binners), refinement, reassembly, and annotation. It provides an easy-to-use framework for producing high-quality MAGs from raw reads.

* **DAS Tool** ({% cite Sieber2018DASTool %}): A bin-refinement tool that combines results from multiple binners (e.g., MetaBAT2, MaxBin2, CONCOCT, SemiBin) into a consensus set of optimized, non-redundant bins. DAS Tool improves overall bin quality by integrating the strengths from several algorithms.

* **Binnette** ({% cite Mainguy2024Binette %}): A fast and accurate bin refinement tool that constructs high-quality MAGs from the outputs of multiple binning tools. It generates hybrid bins using set operations on overlapping contigs — intersection, difference, and union — and evaluates their quality with CheckM2 to select the best bins. Compared to metaWRAP, Binette is faster and can process an unlimited number of input bin sets, making it highly scalable for large and complex metagenomic datasets.

**Anvi’o** ({% cite Eren2015 %}) is a platform for **interactive visualization and manual refinement** of metagenomic bins. While it can run automated binning (defaulting to **CONCOCT**), its main strength lies in allowing users to:

* Inspect contig-level coverage, GC content, and single-copy gene presence
* Visualize connections between contigs in a network view
* Manually merge, split, or reassign contigs to improve bin completeness and reduce contamination
* Annotate bins and link them to taxonomic or functional information

This interactive approach is particularly useful when automated binning produces ambiguous or low-quality bins, enabling **high-confidence MAG reconstruction**.

## Mock binning dataset for this training

Read mapping and binning real metagenomic datasets is a computationally demanding and time-consuming task. To demonstrate the basics of binning in this tutorial, we generated a small mock dataset that is just large enough to produce bins for all binners in this tutorial. The same binners can be applied for any real-life datasets, but as said, plan in some time, up to weeks in some cases.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare analysis history and data

Metagenomic binners typically take two types of data as input: 

- A fasta file containing the assembled contigs, which can be generated from raw metagenomic sequencing reads using an assembler such as MEGAHIT, SPAdes, or IDBA-UD.

- A BAM file containing the read coverage information for each contig, which can be generated from the same sequencing reads using mapping software such as Bowtie2 or BWA.

> <comment-title>Can Bins be generated without coverage information?</comment-title>
>
> Not all binners require coverage information. Some, like MetaBAT2, can operate using only genomic composition (e.g., tetranucleotide frequencies) when coverage files are not available. This is especially useful for single-sample datasets or legacy data where coverage cannot easily be calculated.
>
> Other tools that support composition-only binning include:
> - **MaxBin 2**, which can run with composition alone, but performs better with depth.
> - **SolidBin**, which supports single-sample binning based on sequence features.
> - **VAMB**, which primarily uses deep learning on composition, coverage optional.
>
> That said, including coverage information generally increases binning accuracy, especially for:
> - Differentiating closely related strains
> - Datasets with uneven abundance
> - Multi-sample metagenomics workflows (e.g., differential coverage binning)
>
> In summary: yes, it’s possible to bin without coverage, but coverage-aware workflows are recommended when available, as they reduce contamination and improve completeness.
>
{: .comment}


To run binning, we first need to get the data into Galaxy. Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Prepare the Galaxy history</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

We need to get the data into our history.

In case of a not very large dataset it's more convenient to upload data directly from your computer to Galaxy.

> <hands-on-title>Upload data into Galaxy</hands-on-title>
>
> 1. Import the contig file from [Zenodo]({{ page.zenodo_link }}) or a data library:
>
>    ```text
>    {{ page.zenodo_link }}/files/MEGAHIT_contigs.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    > <comment-title></comment-title>
>    > In case of large dataset, we can use FTP server or the [Galaxy Rule-based Uploader]({% link topics/galaxy-interface/tutorials/upload-rules/tutorial.md %}).
>    {: .comment}
>
> 2. Create a collection named `Contigs`
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
> 3. Import the raw reads in fastq format from [Zenodo]({{ page.zenodo_link }}) or a data library:
>
>    ```text
>    {{ page.zenodo_link }}/files/reads_forward.fastqsanger.gz
>    {{ page.zenodo_link }}/files/reads_reverse.fastqsanger.gz
>    ```
>
> 4. Create a paired collection named `Reads`
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
{: .hands_on}

> <comment-title>Why do we use collections here?</comment-title>
> In this tutorial, collections are not strictly necessary because we are working with only one contig file and its paired-end reads. However, in real metagenomic studies, it is common to process many samples—sometimes hundreds or even thousands—and in those cases, collections become essential for managing data efficiently.  
>  
> It is generally good practice to first test a workflow on a small subset of the data (for example, a collection containing only a single sample) to ensure that the tools run correctly and the parameters are appropriate before launching thousands of jobs on Galaxy.
{: .comment}

# Preparation for binning

As explained before, we need coverage information in a BAM file as a requirement for all binners. Some binners need a specific format for the coverage information, but this will be covered in the version specific to the desired binner. For now, we will map the quality-controlled reads to the contigs to get a BAM file with the coverage information. This BAM file also needs to be sorted for the downstream binners.

> <comment-title></comment-title>
> Make sure the reads are quality-controlled, e.g., by following the [**Quality Control** tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %})
{: .comment}

> <hands-on-title> Map reads to contigs </hands-on-title>
>
> 1. {% tool [Bowtie2](toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.4+galaxy0) %} with the following parameters:
>    - *"Is this single or paired library"*: `Paired-end`
>        - {% icon param-collection %} *"FASTQ Paired Dataset"*: `Reads` (Input dataset collection)
>        - *"Do you want to set paired-end options?"*: `No`
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from the history and build index`
>        - {% icon param-collection %} *"Select reference genome"*: `Contigs` (Input dataset collection)
>    - *"Set read groups information?"*: `Do not set`
>    - *"Select analysis mode"*: `1: Default setting only`
>    - *"Do you want to tweak SAM/BAM Options?"*: `No`
>    - *"Save the bowtie2 mapping statistics to the history"*: `Yes`
>
{: .hands_on}

Let's now sort the BAM files.

> <hands-on-title> Sort BAM files </hands-on-title>
>
> 1. {% tool [Samtools sort](toolshed.g2.bx.psu.edu/repos/devteam/samtools_sort/samtools_sort/2.0.7) %} with the following parameters:
>    - {% icon param-file %} *"BAM File"*: output of **Bowtie2** {% icon tool %}
>    - *"Primary sort key"*: `coordinate`
>
{: .hands_on}

The sorted BAM file can be used as input for any of the binning tools.

# Binning

As explained before, there are many challenges to metagenomics binning. The most common of them are listed below:

- High complexity.
- Fragmented sequences.
- Uneven coverage.
- Incomplete or partial genomes.
- Horizontal gene transfer.
- Chimeric sequences.
- Strain variation.

![The image illustrates the concept of metagenomic binning, where sequences called contigs are grouped into distinct bins based on their similarities. On the left side, a set of colored contigs (represented as horizontal lines in red, yellow, and green) is shown. An arrow labeled "Binning" points to the right, where these contigs are organized into three separate ovals labeled Bin 1, Bin 2, and Bin 3. Each bin contains contigs of similar colors, indicating that they have been grouped together based on shared characteristics, such as origin or genetic similarity. This process helps in categorizing and analyzing genetic material from complex metagenomic datasets.](./images/binning.png "Metagenomic binning involves grouping contigs into 'bins' based on sequence composition, coverage, or other properties."){:width="60%"}

In this tutorial, we offer several versions, each highlighting a different binners:

{% include _includes/cyoa-choices.html option1="MetaBAT2" option2="MaxBin2" option3="SemiBin" option4="CONCOCT" option5="COMEBin" default="MetaBAT2" %}

<div class="MetaBAT2" markdown="1">
{% include topics/microbiome/tutorials/metagenomics-binning/metabet2_version.md %}
</div>
<div class="MaxBin2" markdown="1">
{% include topics/microbiome/tutorials/metagenomics-binning/maxbin2_version.md %}
</div>
<div class="SemiBin" markdown="1">
{% include topics/microbiome/tutorials/metagenomics-binning/semibin_version.md %}
</div>
<div class="CONCOCT" markdown="1">
{% include topics/microbiome/tutorials/metagenomics-binning/concoct_version.md %}
</div>
<div class="COMEBin" markdown="1">
{% include topics/microbiome/tutorials/metagenomics-binning/comebin_version.md %}
</div>

# Bin refinement

Now that we have produced bins with our favorite binner, we can refine the recovered bins. 
Therefore, we need to 

1. Convert the bins into a contig-to-bin mapping table,
2. Combine the tables from each binner into one collection, and
3. Use **Binette** ({% cite Mainguy2024Binette %}) to create consensus bins.

   An alternative tool would be {% tool [DAS Tool](toolshed.g2.bx.psu.edu/repos/iuc/das_tool/das_tool/1.1.7+galaxy1) %}  ({% cite Sieber2018DASTool %}) which is also available in Galaxy.

For the refinement, we will use the bins created by all the binners used before. If you did not run them all by yourself, we have provided the results here as well. 

> <hands-on-title>Get result bins</hands-on-title>
>
> 1. Import the bin files from [Zenodo]({{ page.zenodo_link }}) or a data library:
>
>    ```text
>    {{ page.zenodo_link }}/files/semibin_0.fasta
>    {{ page.zenodo_link }}/files/maxbin_0.fasta
>    {{ page.zenodo_link }}/files/maxbin_1.fasta
>    {{ page.zenodo_link }}/files/metabat_0.fasta
>    {{ page.zenodo_link }}/files/concoct_1.fasta
>    {{ page.zenodo_link }}/files/concoct_2.fasta
>    {{ page.zenodo_link }}/files/concoct_3.fasta
>    {{ page.zenodo_link }}/files/concoct_4.fasta
>    {{ page.zenodo_link }}/files/concoct_5.fasta
>    {{ page.zenodo_link }}/files/concoct_6.fasta
>    {{ page.zenodo_link }}/files/concoct_7.fasta
>    {{ page.zenodo_link }}/files/concoct_8.fasta
>    {{ page.zenodo_link }}/files/concoct_9.fasta
>    ```
>
> 2. Create a collection for each binner by selecting only the bins created by this binner (e.g., `maxbin_0.fasta` and `maxbin_1.fasta` together) and creating a collection:
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
{: .hands_on}

Once each bin set is converted into one collection, it can be converted into a contig-to-bin mapping table. We need to perform this step for every bin set.

> <hands-on-title> Convert the bins into a contig-to-bin mapping table </hands-on-title>
>
> 1. {% tool [Converts genome bins in fasta format](toolshed.g2.bx.psu.edu/repos/iuc/fasta_to_contig2bin/Fasta_to_Contig2Bin/1.1.7+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Bin sequences"*: `bins` (output of any of the binners {% icon tool %})
>
{: .hands_on}


We can now build a list of the tables

> <hands-on-title> Build a list of the binning tables </hands-on-title>
>
> 1. {% tool [Build list](__BUILD_LIST__) %} with the following parameters:
>    - In *"Dataset"*:
>        - {% icon param-repeat %} *"Insert Dataset"*
>            - {% icon param-file %} *"Input Dataset"*: `contigs2bin` (output of **Converts genome bins in fasta format** of SemiBin {% icon tool %})
>            - *"Label to use"*: `Index`
>        - {% icon param-repeat %} *"Insert Dataset"*
>            - {% icon param-file %} *"Input Dataset"*: `contigs2bin` (output of **Converts genome bins in fasta format** of MetaBAT2 {% icon tool %})
>            - *"Label to use"*: `Index`
>        - {% icon param-repeat %} *"Insert Dataset"*
>            - {% icon param-file %} *"Input Dataset"*: `contigs2bin` (output of **Converts genome bins in fasta format** of MaxBin2 {% icon tool %})
>            - *"Label to use"*: `Index`
>        - {% icon param-repeat %} *"Insert Dataset"*
>            - {% icon param-file %} *"Input Dataset"*: `contigs2bin` (output of **Converts genome bins in fasta format** of CONCOCT {% icon tool %})
>            - *"Label to use"*: `Index`
>
{: .hands_on}

We can now refine the bins with Binette.

> <hands-on-title> Refine with Binette </hands-on-title>
>
> 1. {% tool [Binette](toolshed.g2.bx.psu.edu/repos/iuc/binette/binette/1.2.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input contig table"*: `output` (output of **Build list** {% icon tool %})
>    - {% icon param-collection %} *"Input contig file"*: `output` (Input dataset collection)
>    - *"Select if database should be used either via file or cached database"*: `cached database`
>    - *"Set minimus completeness"*: `0`
>
{: .hands_on}

> <question-title>Bin refinement</question-title>
>
> 1. How many bins are left after refinement ?
>
> > <solution-title></solution-title>
> >
> > 1. Two bins are left. Most contigs from different bins where combined into one bin. There is still one single contig bin left.
> >
> {: .solution}
>
{: .question}

# Checking the quality of the bins

Once binning is done, it is important to check its quality.

Binning results can be evaluated with **CheckM** ({% cite Parks2015 %}). CheckM is a software tool used in metagenomics binning to assess the completeness and contamination of genome bins. 

CheckM compares the genome bins to a set of universal single-copy marker genes that are present in nearly all bacterial and archaeal genomes. By identifying the presence or absence of these marker genes in the bins, CheckM can estimate the completeness of each genome bin (i.e., the percentage of the total set of universal single-copy marker genes that are present in the bin) and the degree of contamination (i.e., the percentage of marker genes that are found in more than one bin).

This information can be used to evaluate the quality of genome bins and to select high-quality bins for further analysis, such as genome annotation and comparative genomics. CheckM is widely used in metagenomics research and has been shown to be an effective tool for assessing the quality of genome bins. Some of the key functionalities of CheckM are:

- *Estimation of genome completeness*: CheckM uses a set of universal single-copy marker genes to estimate the completeness of genome bins. The completeness score indicates the proportion of these marker genes that are present in the bin, providing an estimate of how much of the genome has been recovered.

- *Estimation of genome contamination*: CheckM uses the same set of marker genes to estimate the degree of contamination in genome bins. The contamination score indicates the proportion of marker genes that are present in multiple bins, suggesting that the genome bin may contain DNA from more than one organism.

- *Identification of potential misassemblies*: CheckM can identify potential misassemblies in genome bins based on the distribution of marker genes across the genome.

- *Visualization of results*: CheckM can generate various plots and tables to visualize the completeness, contamination, and other quality metrics for genome bins, making it easier to interpret the results.

- *Taxonomic classification*: CheckM can also be used to classify genome bins taxonomically based on the presence of specific marker genes associated with different taxonomic groups.

Based on the previous analysis we will use **CheckM lineage_wf**: *Assessing the completeness and contamination of genome bins using lineage-specific marker sets*

`CheckM lineage_wf` is a specific workflow within the CheckM software tool that is used for taxonomic classification of genome bins based on their marker gene content. This workflow uses a reference database of marker genes and taxonomic information to classify the genome bins at different taxonomic levels, from domain to species.

Now we can investigate the completeness and contamination of any of your previously generated genome bins, as well as the refined set.

> <hands-on-title>Assessing the completeness and contamination of genome bins using lineage-specific marker sets with `CheckM lineage_wf`</hands-on-title>
> 1.  {% tool [CheckM lineage_wf](toolshed.g2.bx.psu.edu/repos/iuc/checkm_lineage_wf/checkm_lineage_wf/1.2.0+galaxy0) %} with parameters:
>     - *"Bins"*: `Folder containing the produced bins`
{: .hands_on}


The output of "CheckM lineage_wf" includes several files and tables that provide information about the taxonomic classification and quality assessment of genome bins. Here are some of the key outputs:

- **CheckM Lineage Workflow Output Report (Bin statistics)**: This report provides a summary of the quality assessment performed by CheckM. It includes statistics such as the number of genomes analyzed, their completeness, contamination, and other quality metrics.

- **Lineage-specific Quality Assessment**: CheckM generates lineage-specific quality assessment files for each analyzed genome. These files contain detailed information about the completeness and contamination of the genome based on its taxonomic lineage.

- **Marker Set Analysis**: CheckM uses a set of marker genes to estimate genome completeness and contamination. The tool produces marker-specific analysis files that provide details on the presence, absence, and copy number of each marker gene in the analyzed genomes.

- **Visualizations**: CheckM generates various visualizations to aid in the interpretation of the results. These include plots such as the lineage-specific completeness and contamination plots, scatter plots, and other visual representations of the data.

- **Tables and Data Files**: CheckM generates tabular data files that contain detailed information about the analyzed genomes, including their names, taxonomic assignments, completeness scores, contamination scores, and other relevant metrics. These files are useful for further downstream analysis or data manipulation.

It should be noted that "CheckM lineage_wf" offers a range of optional outputs that can be generated to provide additional information to the user.

To keep it simple, we will check the bin statistics to investigate the performance of our previous binning efforts.

> <question-title>Binning performance</question-title>
>
> 1. Which binner created the best bins?
> 2. How well did the bin refinement work?
>
> > <solution-title></solution-title>
> >
> > 1. Let's combine the stats:
> >
> > | Binner        | Bin ID     | Marker lineage        | # genomes | # markers | # marker sets | 0  | 1  | 2 | 3 | 4 | 5+ | Completeness | Contamination | Strain heterogeneity |
> > |---------------|------------|------------------------|-----------|------------|----------------|----|----|----|----|----|----|--------------|---------------|-----------------------|
> > | **MetaBAT2**  | 1          | k__Bacteria (UID203)   | 5449      | 103        | 58             | 89 | 14 | 0  | 0  | 0  | 0  | 15.67        | 0.00          | 0.00                  |
> > | **SemiBin**   | SemiBin_0  | k__Bacteria (UID203)   | 5449      | 103        | 58             | 89 | 14 | 0  | 0  | 0  | 0  | 15.67        | 0.00          | 0.00                  |
> > | **MaxBin**    | 001        | k__Bacteria (UID203)   | 5449      | 103        | 58             | 92 | 11 | 0  | 0  | 0  | 0  | 10.50        | 0.00          | 0.00                  |
> > | **MaxBin**    | 002        | k__Bacteria (UID203)   | 5449      | 103        | 58             | 99 | 4  | 0  | 0  | 0  | 0  | 6.90         | 0.00          | 0.00                  |
> > | **CONCOCT**   | 0          | root (UID1)            | 5656      | 56         | 24             | 56 | 0  | 0  | 0  | 0  | 0  | 0.00         | 0.00          | 0.00                  |
> > | **CONCOCT**   | 1          | root (UID1)            | 5656      | 56         | 24             | 56 | 0  | 0  | 0  | 0  | 0  | 0.00         | 0.00          | 0.00                  |
> > | **CONCOCT**   | 2          | root (UID1)            | 5656      | 56         | 24             | 56 | 0  | 0  | 0  | 0  | 0  | 0.00         | 0.00          | 0.00                  |
> > | **CONCOCT**   | 3          | root (UID1)            | 5656      | 56         | 24             | 56 | 0  | 0  | 0  | 0  | 0  | 0.00         | 0.00          | 0.00                  |
> > | **CONCOCT**   | 4          | root (UID1)            | 5656      | 56         | 24             | 56 | 0  | 0  | 0  | 0  | 0  | 0.00         | 0.00          | 0.00                  |
> > | **CONCOCT**   | 5          | root (UID1)            | 5656      | 56         | 24             | 56 | 0  | 0  | 0  | 0  | 0  | 0.00         | 0.00          | 0.00                  |
> > | **CONCOCT**   | 6          | root (UID1)            | 5656      | 56         | 24             | 56 | 0  | 0  | 0  | 0  | 0  | 0.00         | 0.00          | 0.00                  |
> > | **CONCOCT**   | 7          | root (UID1)            | 5656      | 56         | 24             | 56 | 0  | 0  | 0  | 0  | 0  | 0.00         | 0.00          | 0.00                  |
> > | **CONCOCT**   | 8          | root (UID1)            | 5656      | 56         | 24             | 56 | 0  | 0  | 0  | 0  | 0  | 0.00         | 0.00          | 0.00                  |
> > | **CONCOCT**   | 9          | k__Bacteria (UID203)   | 5449      | 103        | 58             | 89 | 14 | 0  | 0  | 0  | 0  | 15.67        | 0.00          | 0.00                  |
> >
> > **MetaBAT2** and **SemiBin** generated the best-performing bins in this dataset, each recovering one bin with ~15.7% completeness and no detectable contamination.  
> >
> > **MaxBin** produced bins of lower completeness (10.5% and 6.9%), but still without contamination.  
> >
> > **CONCOCT** largely failed to recover meaningful bins in this dataset: most bins show 0% completeness. This often occurs with CONCOCT on small assemblies or uneven coverage, as its Gaussian clustering model struggles when the number of contigs is low or coverage variation is insufficient.
> >
> > 2. Let's look at **Binette** stats:
> >
> > | Bin ID        | Marker lineage        | # genomes | # markers | # marker sets | 0  | 1  | 2 | 3 | 4 | 5+ | Completeness | Contamination | Strain heterogeneity |
> > |---------------|------------------------|-----------|------------|----------------|----|----|----|----|----|----|--------------|---------------|-----------------------|
> > | binette_bin1  | k__Bacteria (UID203)   | 5449      | 103        | 58             | 89 | 14 | 0  | 0  | 0  | 0  | 15.67        | 0.00          | 0.00                  |
> > | binette_bin2  | root (UID1)            | 5656      | 56         | 24             | 56 | 0  | 0  | 0  | 0  | 0  | 0.00         | 0.00          | 0.00                  |
> >
> > **Binette** produced one bin essentially identical in quality to the MetaBAT2/SemiBin bins (~15.7% completeness), along with a low-quality root-level bin with no completeness. The latter would typically be filtered out if the *minimum completeness* parameter were set to a more realistic threshold (e.g. >75% for real biological datasets).
> {: .solution}
>
{: .question}

> <comment-title>CheckM2 vs CheckM</comment-title>
> CheckM2 ({%cite Chklovski2023CheckM2%}) is the successor of CheckM, but CheckM is still widely used, since its marker-based logic can be more interpretable in a biological sense. E.g., to date (2025-11-21), NCBI still allows submitting MAGs to GenBank if either checkM or checkM2 has a completeness of > 90% (see the [NCBI WGS/MAG submission guidelines](https://www.ncbi.nlm.nih.gov/genbank/wgsfaq/#metagen)).
>
> **Key differences compared to CheckM1**:
>
> * CheckM1 relies primarily on lineage-specific single-copy marker genes to estimate completeness and contamination of microbial genomes.
> * CheckM2 uses a machine-learning (gradient boost / ML) approach trained on simulated and experimental genomes, and does *not* strictly require a well-represented lineage in its marker database. 
> * CheckM2 is reported to be more accurate and faster for both bacterial and archaeal lineages, especially when dealing with novel or very reduced-genome lineages (e.g., candidate phyla, CPR/DPANN) where classical marker-gene methods may struggle. 
> * The database of CheckM2 can be updated more rapidly with new high-quality reference genomes, which supports scalability and improved performance over time.
>
> If you're working with MAGs from underrepresented taxa (novel lineages) or very small genomes (streamlined bacteria/archaea), CheckM2 tends to give more reliable estimates of completeness/contamination. For more “standard” microbial genomes from well-studied taxa, CheckM1 may still work well, but you may benefit from the improved performance with CheckM2.
>
{: .comment}

# De-replication

De-replication is the process of identifying sets of genomes that are the "same" in a list of genomes, and removing all but the “best” genome from each redundant set. How similar genomes need to be to be considered “same”, how to determine which genome is “best”, and other important decisions are discussed in [Important Concepts](https://drep.readthedocs.io/en/latest/choosing_parameters.html).

A common use for genome de-replication is the case of individual assembly of metagenomic data. If metagenomic samples are collected in a series, a common way to assemble the short reads is with a “co-assembly”. That is, combining the reads from all samples and assembling them. The problem with this is that assembling similar strains can severely fragment assemblies, precluding the recovery of a good genome bin. An alternative option is to assemble each sample separately, and then “de-replicate” the bins from each assembly to make a final genome set.

![Image shows the process of individual assembly on two strains and five samples, after individual assembly of samples two samples are chosen for de-replication process. In parallel, co-assembly on all five samples is performed](./individual-assembly.png "Individual assembly followed by de-replication vs co-assembly"){:width="80%"}

Several tools have been designed for the process of de-replication. **dRep** ({% cite olm2017drep %}) is a software tool designed for the dereplication of genomes in metagenomic datasets. The goal is to retain a representative set of genomes to improve downstream analyses, such as taxonomic profiling and functional annotation.

An typical workflow of how dRep works for dereplication in metagenomics includes:

- *Genome Comparison*: dRep uses a pairwise genome comparison approach to assess the similarity between genomes in a given metagenomic dataset.

- *Clustering*: Based on the genome similarities, dRep performs clustering to group similar genomes into "genome clusters." Each cluster represents a group of closely related genomes.

- *Genome Quality Assessment*: dRep evaluates the quality of each genome within a cluster. It considers factors such as completeness, contamination, and strain heterogeneity.

- *Genome Selection*: Within each genome cluster, dRep selects a representative genome based on user-defined criteria. This representative genome is considered the "dereplicated" version of the cluster.

- *Dereplication Output*: The output of dRep includes information about the dereplicated genomes, including their identity, completeness, and contamination. The user can choose a threshold for genome similarity to control the level of dereplication.

> <hands-on-title>General list of actions for de-replication</hands-on-title>
> 1. Create new history
> 2. Assemble each sample separately using your favorite assembler
> 3. Perform a co-assembly to catch low-abundance microbes
> 4. Bin each assembly separately using your favorite binner
> 5. Bin co-assembly using your favorite binner
> 6. Pull the bins from all assemblies together
> 7. Run {% tool [dRep dereplication](toolshed.g2.bx.psu.edu/repos/iuc/drep_dereplicate/drep_dereplicate/3.6.2+galaxy1) %} on them
> 8. Perform downstream analysis on the de-replicated genome list
>
{: .hands_on}

# Conclusions

In summary, this tutorial shows a step-by-step guide on how to bin metagenomic contigs using various binners, including bin refinement.

It is critical to select the appropriate binning tool for a specific metagenomics study, as different binning methods may have different strengths and limitations depending on the type of metagenomic data being analyzed. By comparing the outcomes of several binning techniques, researchers can increase the precision and accuracy of genome binning.

There are various binning methods available for metagenomic data, including reference-based, clustering-based, hybrid approaches, and machine learning. Each method has its advantages and disadvantages, and the selection of the appropriate method depends on the research question and the characteristics of the data.

Comparing the outcomes of multiple binning methods can help to identify the most accurate and reliable method for a specific study. This can be done by evaluating the quality of the resulting bins in terms of completeness, contamination, and strain heterogeneity, as well as by comparing the composition and functional profiles of the identified genomes.

Overall, by carefully selecting and comparing binning methods, researchers can improve the quality and reliability of genome bins, which can ultimately lead to a better understanding of the functional and ecological roles of microbial communities in various environments.

tools_json:
["Bowtie2", "toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.4+galaxy0", "Samtools sort", "toolshed.g2.bx.psu.edu/repos/devteam/samtools_sort/samtools_sort/2.0.7", "DAS Tool", "toolshed.g2.bx.psu.edu/repos/iuc/das_tool/das_tool/1.1.7+galaxy1", "Converts genome bins in fasta format", "toolshed.g2.bx.psu.edu/repos/iuc/fasta_to_contig2bin/Fasta_to_Contig2Bin/1.1.7+galaxy1", "Build list", "__BUILD_LIST__", "Binette", "toolshed.g2.bx.psu.edu/repos/iuc/binette/binette/1.2.0+galaxy0", "CheckM lineage_wf", "toolshed.g2.bx.psu.edu/repos/iuc/checkm_lineage_wf/checkm_lineage_wf/1.2.0+galaxy0", "dRep dereplication", "toolshed.g2.bx.psu.edu/repos/iuc/drep_dereplicate/drep_dereplicate/3.6.2+galaxy1"]


topic: microbiome
tutorial: host-removal
title: Remove contamination and host reads
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: ["host-removal", "preprocessing", "metagenomics", "microgalaxy"]
content_hash: e5f1059a1b8dbabe4d6512ed61334186
last_modified: 2026-05-27T09:01:49.316997
zenodo_link: https://zenodo.org/records/17829290
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/host-removal/tutorial.html

content:
Metagenomic sequencing captures all DNA present in a sample, including the **microbial community**, **host DNA**, and potential **environmental or external contaminants** (such as human DNA introduced during sample handling or sequencing).  
Before performing taxonomic or functional analysis, it is essential to remove these host-drived or contaminant reads to avoid misleading downstream interpretations.

In this tutorial, we will learn how to identify and remove host or contaminant reads using Galaxy.
We will:
- Map raw reads to a **host reference genome** using Bowtie2 and extract unmapped reads.
- Repeat the process with unmapped reads against a **human reference genome** to remove potential human contamination.
- Generate a final set of **clean, non-host reads** suitable for downstream analyses such as assembly, binning, or profiling.

To test and illustrate the process, we use bee gut metagenome samples and focus on filtering out both host (bee) sequences and potential human contamination from the reads. 


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## Prepare Galaxy and data
Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

Now, we need to import the data

> <hands-on-title>Import datasets</hands-on-title>
>
> 1. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library:
>
>    ```text
>    {{ page.zenodo_link }}/files/SRR24759598_1.fastq.gz
>    {{ page.zenodo_link }}/files/SRR24759598_2.fastq.gz
>    {{ page.zenodo_link }}/files/SRR24759616_1.fastq.gz
>    {{ page.zenodo_link }}/files/SRR24759616_2.fastq.gz
>    ```
>    
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Create a paired collection named 'Raw reads' that includes both pairs.
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
{: .hands_on}

## Map reads to a host genome with Bowtie2
To remove host contamination, we start by mapping the reads to the host genome using Bowtie2 to detect and remove host-derived sequences.

> <hands-on-title>Remove host reads</hands-on-title>
>
> 1. {% tool [Bowtie2](toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.3+galaxy1) %} with the following parameters:
>    - *"Is this single or paired library"*: `Paired-end Dataset Collection`
>        - {% icon param-collection %} *"FASTQ Paired Dataset"*: `Input reads`
>        - *"Write unaligned reads (in fastq format) to separate file(s)"*: `Yes`
>        - *"Do you want to set paired-end options?"*: `Yes`
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>        - *"Select reference genome"*: `A. mellifera genome (apiMel3, Baylor HGSC Amel_3.0)`
>    - *"Set read groups information?"*: `Do not set`
>    - *"Select analysis mode"*: `1: Default setting only`
>    - *"Do you want to tweak SAM/BAM Options?"*: `No`
>    - *"Save the bowtie2 mapping statistics to the history"*: `Yes`
>
> 2. Run the tool. The outputs will include:
>    - Mapping statistics report (`bowtie2.log`)
>    - Unaligned (unmapped) forward and reverse reads
>
> 3. These unmapped reads represent sequences **not belonging to the host** and will be used in the next step. 
>
> > <comment-title>Tip</comment-title>
> > Host reference genomes vary depending on the study organism. You can upload a FASTA file of your host genome if it is not available as a built-in index.
> {: .comment}
>
{: .hands_on}

## Re-pair unmapped reads
We now combine the unmapped forward and reverse reads into a new paired-end dataset for further processing.

> <hands-on-title> Combine unmapped forward and reverse reads into a paired collection </hands-on-title>
>
> 1. {% tool [Zip collections](__ZIP_COLLECTION__) %} with the following parameters:
>    - {% icon param-file %} *"Input 1"*: `output_unaligned_reads_l` 
>    - {% icon param-file %} *"Input 2"*: `output_unaligned_reads_r` 
>
>    2. This step creates a new paired-end collection that represents all reads **not aligned to the host genome**. Rename this output collection to 'Reads without bee reads'
>
>    > <comment-title> Note </comment-title>
>    >
>    > Zipping restores the normal paired-end structure, which is required for downstream tools or for rerunning the workflow on another reference.
>    {: .comment}
>
{: .hands_on}

## Summarize mapping statistics
Once the host mapping is complete, we use MultiQC to summarize and visualize the mapping statistics, helping us assess how many reads were removed and how many remain.

> <hands-on-title> Evaluate host read removal results </hands-on-title>
>
> 1. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3) %} with the following parameters:
>    - In *"Results"*:
>        - {% icon param-repeat %} *"Insert Results"*
>            - *"Which tool was used generate logs?"*: `Bowtie 2`
>                - {% icon param-file %} *"Output of Bowtie 2"*: `mapping_stats` (output of **Bowtie2** {% icon tool %})
>    - *"Report title"*: `Host Removal`
>
> 2. Run the tool and open the generated HTML report.
> 3. Review the mapping percentage, number of reads aligned, and number of unmapped reads.
>
>    > <comment-title>Tip</comment-title>
>    >
>    > Low mapping percentages in the report confirm that most host reads were successfully removed.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What percentage of reads mapped to the bee reference genome?  
> 2. What percentage of reads were removed?
>
> > <solution-title></solution-title>
> >
> > 1. 2.8% for SRR24759598 and 1.1% for SRR24759616. 
> > 2. 2.8% for SRR24759598 and 1.1% for SRR24759616.
> >
> {: .solution}
>
{: .question}

## Remove potential human contamination
After removing host reads, we can run the **same workflow again** to eliminate possible **human contamination** that may remain in the dataset.

> <hands-on-title> Rerun the workflow using the human genome as reference</hands-on-title>
>
> 1. Use the unmapped reads **Reads without bee reads** as the input for this second run.
> 2. In the **Bowtie2** step:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>    - *"Select reference genome"*: `Human (GRCh38)`
>    - Keep all other parameters the same as in the first run.
>
> 3. Continue through the **Zip collections** and **MultiQC** steps as before.
> 4. The output of this second run represents your **final cleaned reads**, free from both host and human sequences. Rename this collection to 'Clean metagenomic reads'
>
>    > <comment-title>Note</comment-title>
>    > Rerunning the same workflow maintains reproducibility. 
>    > Only the reference genome and the input data change between the two runs.
>    {: .comment}
>
{: .hands_on}

> <question-title>Verify your final dataset</question-title>
>
> 1. How does the mapping percentage differ between the host and human filtering runs?
> 2. How many reads have been removed?
>
> > <solution-title></solution-title>
> >
> > 1. The second run (against the human genome) usually removes only a small number of additional reads.
> > 2. 0.1% in both samples.
> >
> {: .solution}
>
{: .question}


# Conclusion

In this tutorial, you learned how to:

- Identify and remove reads originating from host or contaminant genomes using **Bowtie2**.  
- Combine unmapped forward and reverse reads into a paired collection for reuse.  
- Summarize mapping statistics and verify host-read removal using **MultiQC**.  
- Rerun the same workflow with a human reference genome to remove residual human contamination.  

The resulting **clean reads** are now ready for downstream metagenomic analyses such as:
- **Assembly**   
- **Binning**   
- **Functional or taxonomic profiling** 

These preprocessing steps are essential to ensure accurate microbial community reconstruction without interference from host DNA.

tools_json:
["Bowtie2", "toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.3+galaxy1", "Zip collections", "__ZIP_COLLECTION__", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3"]


topic: microbiome
tutorial: diversity
title: Calculating α and β diversity from microbiome taxonomic data
difficulty: introductory
time_estimation: 20M
requirements_json: []
tags_json: ["metagenomics", "diversity"]
content_hash: 2064f938924987a9304fa49105f19251
last_modified: 2026-05-27T09:01:49.316430
zenodo_link: https://zenodo.org/records/13150694
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/diversity/tutorial.html

content:
A **diversity index** is a quantitative measure that is used to assess the level of diversity or variety within a particular system, such as a biological community, a population, or a workplace. It provides a way to capture and quantify the distribution of different types or categories within a system.

In various fields, diversity indexes are employed to understand and compare the composition and richness of various elements. Apart from ecology, fields such as social and cultural science are interested in the diversity within a population or workplace. In these cases, the indexes may consider factors like age, gender, ethnicity, or other relevant characteristics to assess the diversity and inclusiveness of a group or organization.

To study microbiome data, indirect methods like **metagenomics** can be used. Metagenomic samples contain DNA from different organisms at a specific site, where the sample was collected. Metagenomic data can be used to find out which organisms coexist in that niche and which genes are present in the different organisms.

Once we know which taxons are present in a metagenomic sample ([Tutorial on Taxonomic Profiling and Visualization of Metagenomic Data]({% link topics/microbiome/tutorials/taxonomic-profiling/tutorial.md %})), we can do diversity analyses.

Related to ecology, the term **diversity** describes the number of different species present in one particular area and their relative abundance. More specifically, several different metrics of diversity can be calculated. The most common ones are $$\alpha$$, $$\beta$$ and $$\gamma$$ diversity:

- **$$\alpha$$ diversity** describes the diversity within a community

    It considers the number of different species in an environment (also referred to as species **richness**). Additionally, it can take the abundance of each species into account to measure how evenly individuals are distributed across the sample (also referred to as species **evenness**). 

- **$$\beta$$ diversity** compare the diversity between different communities by measuring their distance

- **$$\gamma$$ diversity** is a measure of the overall diversity for the different ecosystems within a region.

![The image illustrates the concepts of alpha (α), beta (β), and gamma (γ) diversity using two landscapes, each containing different sites populated with various colored fish. Landscape AB includes Site A and Site B, while Landscape CD includes Site C and Site D. Alpha diversity refers to the variety of species within a single site, as indicated by the different colored fish within each site. Beta diversity measures the difference in species composition between sites within a landscape, shown by comparing the fish species between Site A and Site B, or Site C and Site D. Gamma diversity encompasses the overall diversity across all sites within a landscape, highlighting the total variety of species present in Landscape AB or Landscape CD. The image visually explains how these three levels of diversity are interconnected, emphasizing the importance of considering different scales when studying biodiversity.](./images/diversity_differences.png)

In this analysis we will use Galaxy for calculating different alpha diversity indexes and the Bray-Curtis dissimilarity index for β diversity. 

# Background on data

The dataset we will use for this tutorial comes from an oasis in the Mexican desert called Cuatro Ciénegas ({% cite Okie.2020 %}). The researchers were interested in genomic traits that affect the rates and costs of biochemical information processing within cells. They performed a whole-ecosystem experiment, thus fertilizing the pond to achieve nutrient enriched conditions.

Here we will use 2 datasets:
- `JP4D`: a microbiome sample collected from the Lagunita Fertilized Pond
- `JC1A`: a **control** samples from a control mesocosm.

The datasets differ in size, but according to the authors this doesn't matter for their analysis of genomic traits. Also, they underline that differences between the two samples reflect trait-mediated ecological dynamics instead of microevolutionary changes as the duration of the experiment was only 32 days. This means that depending on available nutrients, specific lineages within the pond grow more successfully than others because of their genomic traits. The samples have been analysed as explained in the [Taxonomic profiling tutorial]({% link topics/microbiome/tutorials/taxonomic-profiling/tutorial.md %}).

In a nutshell, taxonomic labels have been assigned to the metagenomics data using [Kraken2](https://ccb.jhu.edu/software/kraken2/) to find out which species are present in the samples. Finally, species abundance was estimated using [Bracken](https://ccb.jhu.edu/software/bracken/). For this tutorial, we will use the output file of Bracken.

To get an overview, you can find a Krona chart visualizing the different species present in the two samples.

<iframe id="krona" src="krona-kraken.html" frameBorder="0" width="100%" height="900px"> ![Krona chart with multi-layered pie chart representing the community profile with in the center the higher taxonomy levels (i.e. domain) and on the exterior the more detailed ones (i.e. species)](./images/krona-kraken.png) </iframe>

The dataset we will work with in this tutorial is the output file of Bracken, which estimates species abundance.

```
name	taxonomy_id	taxonomy_lvl	kraken_assigned_reads	added_reads	new_est_reads	fraction_total_reads
Paracoccus sp. MC1862	2760307	S	98	4	102	0.00169
Paracoccus sp. AK26	2589076	S	85	8	93	0.00154
Paracoccus sp. Arc7-R13	2500532	S	67	13	80	0.00133
Paracoccus sp. BM15	1529068	S	27	1	28	0.00046
Paracoccus sanguinis	1545044	S	142	37	179	0.00297
Paracoccus contaminans	1945662	S	87	18	105	0.00174
Paracoccus aminovorans	34004	S	86	26	112	0.00186
```

> <question-title></question-title>
>
> What information do the different columns contain?
>
> > <solution-title></solution-title>
> >
> >  1. species name
> >  2. taxonomy ID
> >  3. taxonomic level: K_kingdom, P_phylum, C_class, O_order, F_family, G_genus, and S_species
> >  4. reads assigned by Kraken
> >  5. additional reads added by Bracken: In order to estimate species abundance, Bracken reestimates the reads assigned by Kraken using bayesian reestimation. For details on the procedure, have a look into the [Bracken publication](https://peerj.com/articles/cs-104/).
> >  6.  sum of column 4 and column 5
> >  7. fraction of the reads assigned to the particular species and the total reads
> >
> {: .solution}
{: .question}


> <details-title>More details on using input other than Bracken</details-title>
>
> It is possible to calculate $$\alpha$$ and $$\beta$$ diversity also on other datasets than the Bracken output. Any tool that outputs taxonomy abundances can be used prior to the diversity analysis. To use **Krakentools** (as described here), the respective output file needs to be converted into the correct table format and filtered for the taxonomic rank "species". This step is not necessary when using Bracken output as already only the species level is listed.
>
{: .details}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare Galaxy and data

Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

We need now to import the data

> <hands-on-title>Import datasets</hands-on-title>
>
> 1. Import the following samples via link from [Zenodo]({{ page.zenodo_link }}) or Galaxy shared data libraries:
>
>    ```text
>    {{ page.zenodo_link }}/files/JC1A_Estimate_Abundance_at_Species_Level.tsv
>    {{ page.zenodo_link }}/files/JP4D_Estimate_Abundance_at_Species_Level.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Create a collection.
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
{: .hands_on}


# Calculating $$\alpha$$ diversity

## Theory of $$\alpha$$ diversity

**$$\alpha$$ diversity** describes the diversity within a community. There are several different indexes used to calculate $$\alpha$$ diversity because different indexes capture different aspects of diversity and have varying sensitivities to different factors. These indexes have been developed to address specific research questions, account for different ecological or population characteristics, or highlight certain aspects of diversity. 

![The image explains the concept of alpha diversity, which refers to the variety of species within a specific community or ecosystem. It compares three communities: Community A, Community B, and Community C. Both Community A and Community B contain five different shapes, each representing a different microbial species, indicating that they have the same level of alpha diversity. Community C, however, consists of only three different shapes, suggesting it has lower alpha diversity compared to Communities A and B. The image uses arrows and labels to highlight that Communities A and B have higher alpha diversity than Community C, with a note clarifying that each shape symbolizes a distinct microbial species.](./images/alphadiversity_metrics.png)(Source: [Pedro J Torres](https://medium.com/pjtorres-high-gut-alpha-diversity-and-health/high-alpha-diversity-and-health-65e5eca7fa36))

Metrics of $$\alpha$$ diversity can be grouped into different classes:

![The image is a conceptual diagram illustrating the concepts of richness and evenness in ecological communities using colored dots within four quadrants. The vertical axis represents richness, increasing upwards, while the horizontal axis represents evenness, increasing to the right. The top left quadrant shows moderate richness and evenness with a mix of colored dots. The top right quadrant, with a high number of evenly distributed colored dots, indicates high richness and evenness. The bottom left quadrant, having fewer and unevenly distributed dots, suggests lower richness and evenness. The bottom right quadrant displays moderate richness but lower evenness, with a somewhat uneven distribution of dots. This diagram visually explains how different communities can vary in species diversity and abundance.](./images/alpha_diversity_richness_evenness.png)

- **Richness** estimates the quantity of distinct species within a sample

   - **Margalef’s richness** indicates the estimated species richness, accounting for the community size. This metric takes into account that a larger community size can support a greater number of species ({% cite Margalef.1969 %})
                                          
      $$ D = \frac{(S - 1)}{\log(n)} $$

      With
      - $$S$$ the total number of species,
      - $$n$$ the total number of individuals in the sample                                                                                                                                                          
   - **Chao1** estimates the true species richness or diversity of a community, particularly when there might be rare or unobserved species. Chao1 estimates the number of unobserved species based on the number of singletons and doubletons. It assumes that there are additional rare species that are likely to exist but have not been observed. The estimation considers the number of unobserved singletons and doubletons and incorporates them into the observed species richness to provide an estimate of the true species richness ({% cite Chao.1992 %}).
   
      $$ 
      S_{chao1} = S_{obs} + \frac{n_{1}(n_{1} - 1)}{2(n_2 + 1)} 
      $$
   
      With:
      - \\(S_{obs}\\) the observed species richness, 
      - \\(n_{1}\\) the number of species represented by a single individual (singletons), 
      - \\(n_{2}\\) the number of species represented by two individuals (doubletons).
   
   - **ACE** (Abundance-based Coverage Estimator) takes into account the abundance distribution of observed species and incorporates the presence of rare or unobserved species. ACE estimates the number of unobserved species based on the abundance distribution and incorporates it into the observed species richness. It takes into account the relative rarity of observed species and uses this information to estimate the true species richness.    
                                                                              

- **Evenness** evaluates the relative abundances of species rather than their total count
  
   - **Pielou’s evenness** quantifies how close the community's diversity is to the maximum possible diversity. This index is calculated by taking the Shannon Diversity Index (which measures the overall diversity of the community) and dividing it by the maximum possible diversity given the observed species richness ({%cite Pielou.1966 %}).  

      $$ J = \frac{H'}{ln(S)} $$
                                                                                 
      With:
      - $$H'$$ the Shannon Weiner diversity Index
      - $$S$$ the total number of species in a sample, across all samples in dataset.
 

- **Diversity** incorporates both the relative abundances and total count of distinct species
  
   - **Shannons** index calculates the uncertainty in predicting the species identity of an individual that is selected from a community ({% cite Shannon.1948 %}).

      $$ 
      H' = - \sum_{i=1}^{S} p_i \ln (p_i) 
      $$
      
      With \\(p_i\\) the proportion of individuals of species $$i$$

      The index ranges from 0 to a maximum value that depends on the number of species and their relative abundances. The higher the Shannon index value, the greater the diversity within the community. 
                                                                                                                                    
   - **Berger-Parker** index expresses the proportional importance of the most abundant type. This index is highly biased by sample size and richness ({% cite Berger.1970 %} ).

      $$ 
      D = \frac{n_{max}}{N} 
      $$
   
      With:
      - \\(n_{max}\\) the abundance of the most dominant species
      - $$N$$ the total number of individuals (sum of all abundances)

      In contrast to the Shannon index, which considers both species richness and evenness, the Berger-Parker index emphasizes the dominance of a particular species.

   - **Simpsons** index calculates the probability that two individuals selected from a community will be of the same species. Small values are observed in datasets of high diversity and large values in datasets of low diversity ({% cite SIMPSON.1949 %}).

      $$ 
      D = \sum_{i=1}^{S} \left( \frac{n_i}{N} \right)^2 
      $$
      
      With:
      - \\(n_i\\) the number of individuals in species $$i$$
      - $$N$$ total number of individuals of all species
      - \\(\frac{n_i}{N} = p_i\\) the proportion of individuals of species $$i$$
      - $$S$$ the species richness.

      The index ranges from 0 to 1, with 1 representing maximum diversity. A value of 0.6 indicates that two randomly selected individuals from the community have 60% probability to belong to different species.

   - **Inverse Simpons** index is the transformation of Simpsons index that increases with increasing diversity. A higher Inverse Simpson's index value signifies a community with a greater number of species and a more even distribution of individuals among those species. The index ranges from 1 to the total number of species in the community, with higher values indicating higher diversity. 

   - **Fishers alpha** index describes the relationship between the number of species and the number of individuals in those species. Parametric index of diversity that assumes that the abundance of species follows a logarithmic series distribution ({% cite Fisher.1943 %}).

      $$ 
      S = a \cdot \ln \left(1+\frac{n}{a}\right) 
      $$

      With:                                                      
      - $$S$$ the number of taxa
      - $$n$$ the number of individuals 
      - $$a$$ the the Fisher's alpha. 

      A higher Fisher's alpha value indicates greater species diversity within the sampled area.

## Computing α diversity using Galaxy                                                  

**KrakenTools** is a suite of scripts designed to help Kraken users with downstream analysis of Kraken results. The Krakentool **Calculate alpha diversity** offers the possibility to calculate five different $$\alpha$$ diversity indexes:
1. Shannon's $$\alpha$$ diversity
2. Berger Parker's $$\alpha$$
3. Simpson's diversity
4. Inverse Simpson's diversity
5. Fisher's index

> <hands-on-title>Calculate α diversity with Krakentools</hands-on-title>
> 1. {% tool [Krakentools: Calculate alpha diversity](toolshed.g2.bx.psu.edu/repos/iuc/krakentools_alpha_diversity/krakentools_alpha_diversity/1.2+galaxy1) %} with the following parameters:
>    - {% icon param-collection %} *"Abundance file"*: uploaded Bracken output files
>    - *"Specify alpha diversity type"*: `Shannon's alpha diversity`
>
> 2. Run {% tool [Krakentools: Calculate alpha diversity](toolshed.g2.bx.psu.edu/repos/iuc/krakentools_alpha_diversity/krakentools_alpha_diversity/1.2+galaxy1) %} with the 4 other indexes
{: .hands_on}

> <question-title></question-title>
>
> 1. What are the values of 5 different $$\alpha$$ indexes?
> 2. How should we interpret the different $$\alpha$$ indexes?
> 2. Are the results consistent among the different indexes?
> 3. What is the dominant species in the samples? What is the problem we can see?
>
> > <solution-title></solution-title>
> >
> > 1. The results of five $$\alpha$$ indexes for both the JC1A and JP4D sample as well as an explanation of the meaning of these values.
> >     
> >     |                 | JC1A      | JP4D      |
> >     | --------------- | --------- | --------- |
> >     | Shannon         | 2.06      | 3.74      |
> >     | Berger-Parker   | 0.46      | 0.08      |
> >     | Simpson         | 0.74      | 0.97      |
> >     | Inverse Simpson | 3.92      | 28.84     |
> >     | Fisher          | 209.73    | 456.61    |
> >      
> > 2. Intepretation of the different indexes:
> >    - **Shannon index**: the index ranges from 0 to a maximum value that depends on the number of species and their relative abundances. The higher the Shannon index value, the greater the diversity within the community. A value of ~4 indicates a **relatively high level of diversity** within the community. 
> >     - **Berger-Parker index**: In contrast to the Shannon index, which considers both species richness and evenness, the Berger-Parker index emphasizes the dominance of a particular species.
> >        - For JC1A, a value of 0.46 suggests that **a single species dominates the community**, as it represents **46%** of the total individuals in the community. This indicates a relatively low level of species evenness, meaning that the abundance of individuals is heavily skewed towards one dominant species. A value of 0.46 indicates that the community is heavily influenced by one species, while the other species in the community are less abundant.
> >        - In the case of JP4D, the dominant species accounts for only **8 %** of the total individuals, which implies a **more balanced distribution of individuals** among different species compared to a higher Berger-Parker index value.
> >     - **Simpson's index**: The index ranges from 0 to 1, with 1 representing maximum diversity. Therefore, value of 0.97 indicates a **high level of species diversity and evenness** within the community: community is highly diverse, with a relatively even distribution of individuals among different species. In other words, the value of 0.97 indicates that two randomly selected individuals from the community have 97% probability to belong to different species. This implies a rich and balanced community where multiple species coexist in relatively equal abundance.
> >     - **Inverse Simpson's index**: The Inverse Simpson's index is the reciprocal of the Simpson's index, which quantifies species diversity and evenness within a community. A higher Inverse Simpson's index value signifies a community with a greater number of species and a more even distribution of individuals among those species. The index ranges from 1 to the total number of species in the community, with higher values indicating higher diversity. A value of **3.92** suggests a **relatively low level of species diversity** within the community and a value of 28.84 suggests a relatively high level of species diversity within the community.
> > 
> >     Index | JC1A | JP4D |
> >     --- | --- | ---
> >     Shannon | Less diversity as in JP4D | Relatively high level of diversity
> >     Berger-Parker | A single species dominates the community | mMre balanced distribution of individuals among different species |
> >     Simpson | Lower level of species diversity and evenness| High level of species diversity and evenness |
> >     Inverse Simpson | Relatively low level of species diversity | Relatively high level of species diversity |
> >     Fisher | Lower species diversity | Greater species diversity |
> >
> >
> > 3. The results are consistent as all indexes show JP4D to be the more diverse sample compared to JC1A.                                                                   
> > 4. When looking at the Bracken files, we see that the dominant species in this samples is `Homo sapiens` - so us! This should not be part of the samples and is probably due to contamination. This finding should be used to reanalyze the samples and remove the human contamination. For the sake of simplicity we will continue with the samples as is and assume the `Homo sapiens` to be a species in out samples.                                                     
> {: .solution}
>
{: .question}


> <comment-title></comment-title>
>
> Apart from Krakentools, there are at least two more tools available in Galaxy that can be used to calculate diversity indexes, 
> 
> - **QIIME 2** ({% cite Bolyen.2019 %})
>
>    QIIME 2 (Quantitative Insights Into Microbial Ecology 2) is a powerful open-source bioinformatics software package that provides a comprehensive suite of tools and methods for processing, analyzing, and visualizing microbiome data. It offers a modular approach to microbiome analysis, allowing researchers to build flexible analysis pipelines tailored to their specific research goals. The software supports a wide range of data types, including 16S rRNA gene sequencing, metagenomics, metatranscriptomics, and others.
> 
>    Some of the key features and functionalities of QIIME 2 include:
>    1. Diversity Analysis: QIIME 2 allows users to explore and quantify microbial diversity within and between samples. It provides metrics for alpha diversity (within-sample diversity) and beta diversity (between-sample diversity).
>    2. Data Import and Preprocessing
>    3. Taxonomic Assignments
>    4. Community Analysis
>    5. Phylogenetic Analysis
>    6. Statistical Analysiss.
>    7. Visualization
>
> - **[Vegan](https://github.com/vegandevs/vegan)**
> 
>    The vegan package is a community ecology package in the R programming language. It provides a wide range of tools and methods for analyzing and interpreting ecological data, particularly in the context of community ecology. The package is designed to handle multivariate data and offers various statistical techniques for studying species composition, diversity, and community dynamics.
>
>    The vegan package encompasses several functionalities, including:
>
>    1. Diversity Analysis: vegan offers numerous diversity indices, such as species richness, Shannon diversity index, Simpson index, and many others. These indices allow researchers to quantify the diversity of species within a community and compare diversity between different samples or groups.
>    2. Ordination Techniques
>    3. Community Classification:
>    4. Ecological Network Analysis
>    5. Ecological Indices
>    6. Plotting and Visualization
>
{: .comment}

# Calculating $$\beta$$ diversity 

## Theory of $$\beta$$ diversity

**$$\beta$$ diversity** measures the distance between two or more separate entities. It therefore describes the difference between two communities or ecosystems. 

There are **multiple indexes** used to calculate $$\beta$$ diversity because different indexes emphasize different aspects of compositional dissimilarity between communities or sites.

These indexes have been developed to address specific research questions, accommodate different data types, or provide insights into different dimensions of $$\beta$$ diversity.

- **Jaccard Index** measures the proportion of shared species between two samples ({% cite Jaccard.1912 %}).
  
   $$ 
   J(X, Y) =  \frac{\| X ∩ Y\|}{\| X ∪ Y\|} 
   $$
  
   With:
   - $$X ∩ Y$$ the intersection of sets $$X$$ and $$Y$$, i.e. elements common to both sets
   - $$X ∪ Y$$ the union of sets $$X$$ and $$Y$$, i.e. all unique elements from both sets combined
  
- **Sørensen Index** is similar to Jaccard Index, but accounts for species abundance ({% cite Srensen.1948 %}).

   $$
   DSC = \frac{2\| X ∩ Y\|}{\| X\|} + \| Y\| 
   $$

   With:
   - $$X ∩ Y$$ the intersection of sets $$X$$ and $$Y$$, i.e.elements common to both sets
   - $$ \| X\|$$ and $$\| Y \|$$ the cardinalities of the two sets, i.e. the number of elements in each set.
  
- **Bray-Curtis Dissimilarity** measures the dissimilarity of species abundances between two samples ({% cite Bray.1957 %}).

   $$ 
   BC_{ij} = 1 - \frac{2C_{ij}}{S_{i} + S_{j}}
   $$

   With:
   - \\(C_{ij}\\) the sum of the absolute differences in abundances between corresponding species in samples $$i$$ and $$j$$
   - \\(S_{i}\\) the total abundance or sum of species abundances in sample $$i$$
   - \\(S_{j}\\) the total abundance or sum of species abundances in sample $$j$$

   The higher the dissimilarity value, the greater the difference in species composition or abundances.
  
- **Kulczynski Dissimilarity** masures the dissimilarity in the proportional abundances of shared species.

   $$ 
   D = 1 - \frac{S_{AB}}{S_{A} + S_{B} - 2S_{AB}} 
   $$

   With:
   - \\(S_{AB}\\) the number of shared species between communities $$A$$ and $$B$$
   - \\(S_{A}\\) the number of species in community $$A$$
   - \\(S_{B}\\) the number of species in community $$B$$
  
- **UniFrac** incorporates information on phylogenetic distances between observed species in the computation. Can be calculated either weighted (accounts for abundances) or unweighted (accounts only for richness).

   ![The image consists of three phylogenetic trees, each representing different levels of community relatedness based on a parameter denoted as DD. The first tree on the left illustrates "Identical communities" with D=0.0D=0.0, showing branches that are closely related and color-coded similarly, indicating high similarity or identical species composition among the communities. The middle tree represents "Related communities" with D=0.5D=0.5, displaying a mix of colors on the branches, which suggests some degree of divergence and shared ancestry, indicating that these communities are related but not identical. The tree on the right depicts "Unrelated communities" with D=1.0D=1.0, characterized by distinctly different colors on separate branches, signifying a high level of dissimilarity and no shared recent ancestry among the communities. This visual representation highlights how community composition and relatedness can vary from being identical to completely unrelated.](./images/unifrac.png)

## Computing $$\beta$$ diversity using Galaxy   

> <hands-on-title>Calculate β diversity with Krakentools</hands-on-title>
> 1. {% tool [Krakentools: Calculate beta diversity (Bray-Curtis dissimilarity)](toolshed.g2.bx.psu.edu/repos/iuc/krakentools_beta_diversity/krakentools_beta_diversity/1.2+galaxy1) %} with the following parameters:
>   - {% icon param-collection %} *"Taxonomy file"*: uploaded Bracken output files
>   - *"Specify type of input file"*: `Bracken species abundance file`
>    
{: .hands_on}


> <question-title></question-title>
>
> 1. What is the Bray-Curtis dissimilarity calculated for the two samples?
> 2. How can we interpret Bray-Curtis dissimilarity between the two samples?
>
> > <solution-title></solution-title>
> >
> > 1. The output file gives you a table comparing sample 0 and 1, respectively. Consequently, comparing 0 to 0 and 1 to 1 results in a dissimilarity of 0, as those are exactly the same. Comparing sample 0 to sample 1 shows a Bray-Curtis dissimilarity of 0.701.
> > 2. The Bray-Curtis dissimilarity measures the dissimiliraty of two samples. Consequently, an output of 0 represents two samples that are exactly the same, while an output of 1 means they are maximally divergent. In our case, a Bray-Curtis dissimilarity of 0.7 suggests that there is a substantial difference in the species composition or abundances between the two communities being compared. The higher the dissimilarity value, the greater the difference in species composition or abundances.
> {: .solution}
>
{: .question}

# Evaluation of different diversity metrics

Bonilla-Rosso et al. did **performance evaluation** of different diversity metrics using simulated data sets {% cite BonillaRosso.2012 %}. However, it is important to note that none of the estimated metrics showed statistical similarity to their corresponding parameters in the source communities. Moreover, the results obtained were inconsistent across the samples.

This **inconsistency** can be attributed to the fact that individual metrics only provide a **specific** perspective on diversity and are prone to bias in their estimation, leading to incorrect ranking of the samples.

In summary, **relying solely on single-diversity metrics may not be enough** to accurately compare the diversity between two communities. Instead, we recommend utilizing multi-dimensional metrics to capture diverse rankings across different scales of diversity, which can be affected differently in manipulative studies.

**Multidimensional diversity metrics**, also known as multivariate diversity metrics, are quantitative measures that capture multiple dimensions or aspects of diversity simultaneously. These metrics go beyond single-diversity metrics, such as species richness or Shannon entropy, which provide a one-dimensional representation of diversity.

In contrast, multidimensional diversity metrics **take into account various attributes or characteristics** of species or communities to provide a more comprehensive understanding of diversity. These attributes can include species abundances, functional traits, phylogenetic relationships, or spatial distributions.

The choice and composition of dimensions in multidimensional diversity metrics depend on the research context and the specific objectives of the study. Some aspects, multidimensional diversity metrics take into account, include:
1. **Functional Diversity**: This metric considers the range and variation of functional traits among species within a community. It assesses the diversity of ecological roles and functional strategies present, contributing to ecosystem functioning and resilience.
2. **Phylogenetic Diversity**: This metric incorporates the evolutionary relationships among species within a community. It quantifies the diversity based on the length and topology of the phylogenetic tree, highlighting the evolutionary history and relatedness of species.
3. **Spatial Diversity**: This metric incorporates spatial patterns and distributions of species within a landscape or ecosystem. It considers the heterogeneity of habitats, connectivity, and the arrangement of species populations across space.

Multidimensional diversity metrics offer a **more nuanced and holistic perspective** on biodiversity, capturing different facets and dimensions of ecological variation. They provide insights into the ecological processes shaping communities and ecosystems and can be valuable in conservation planning, ecosystem management, and understanding the functional implications of biodiversity patterns.

> <details-title>More details on Multidimensional diversity metrics and Parameter q</details-title>
>
> Expressing the compositional complexity of an assemblage cannot be accomplished with a single numerical value. Traditional measures like diversities (Hill numbers) and entropies (Rényi entropies) vary in their **order $$q$$**, which determines **the extent to which rare or common species are emphasized**. The ranking and comparison of assemblages rely on the chosen value of $$q$$. 
>
> Instead of selecting a few measures to describe an assemblage, it is preferable to **present a continuous profile** that depicts diversity or entropy as a function of $$q$$ (where $$q \sep 0$$). This approach enables a visual comparison of the compositional complexities among multiple assemblages and facilitates the assessment of the evenness in the relative abundance distributions of the assemblages. In practice, the profile is typically plotted for values of $$q$$ ranging from 0 to $$q =$$ 3 or 4, beyond which there is usually little change.
>
> 
> - **Hill numbers**
>
>    ![The image presents a graph and accompanying text that illustrate the diversity profiles of two ecological communities, labeled A and B, each containing the same number of species (100 species) and individuals (1,000 individuals), but differing in evenness. The graph plots Hill numbers, which represent the effective number of species, against a parameter qq, which determines the sensitivity of the measure to species relative abundance. Profile A, depicted by a blue curve, shows a steep decline as qq increases, indicating a few dominant species and many rare species, reflecting lower evenness. Profile B, represented by a red horizontal line, maintains a constant Hill number across all values of qq, indicating that all species have a similar number of individuals, reflecting higher evenness. The text explains that Hill numbers provide a way to compare diversity profiles graphically and quantitatively, with differences in the distribution of individuals among species affecting the diversity measures. Community A, with greater imbalance and less evenness, exhibits lower order 1 and order 2 diversity compared to the more even Community B.](./images/hill_numbers.png "<a href="https://www.redalyc.org/journal/5117/511766773011/html/">Source</a>")
> 
>    Hill numbers, also known as diversity indices or diversity measures, are mathematical metrics used to quantify the diversity or richness of a biological community. They were developed by ecologist Robert H. Whittaker and are widely used in ecology and biodiversity studies.
>
>    Hill numbers provide a way to summarize and compare the diversity of different communities based on the abundance or occurrence of different species within those communities. These numbers take into account both the number of species present and their relative abundances. The higher the Hill number, the greater the diversity or richness of the community.
>
>    Hill numbers are often represented by the symbol $$D$$, followed by a subscript that indicates the order of diversity. The order of diversity determines the weight given to rare versus common species. Commonly used Hill numbers include:
>
>    1. Species richness ($$D_0$$): This is the simplest Hill number and represents the total number of species in a community, without considering their abundances. It provides a basic measure of biodiversity based on species count.
>    2. Shannon diversity index ($$D_1$$): This index incorporates both species richness and evenness. It takes into account both the number of species and their relative abundances, providing a more comprehensive measure of diversity.
>    3. Simpson diversity index ($$D_2$$): This index focuses on the dominance or concentration of species within a community. It considers both species richness and the probability that two individuals randomly selected from the community belong to the same species.
>
> - **Rényi entropy**
>
>    Rényi entropy is a concept in information theory and statistical physics introduced by Alfréd Rényi, a Hungarian mathematician. It is a generalization of the Shannon entropy, which measures the uncertainty or information content of a random variable or probability distribution.
>    The Rényi entropy of a discrete probability distribution is defined by the parameter $$\alpha$$, which determines the order of the entropy. The formula for calculating Rényi entropy is:
>
>    $$ 
>    H \alpha (P) = \frac{1}{1 - α} \dot log_{2}(\sum{i=1}^{N} pi^{\alpha}) 
>    $$
>
>    where $$P = {p_1, p_2, ..., p_N}$$ is the probability distribution of $$N$$ discrete events or states, and $$p_i$$ represents the probability of the ith event.
>
>    The value of $$\alpha$$ determines the properties of Rényi entropy. When $$\alpha = 1$$, Rényi entropy reduces to Shannon entropy, providing a measure of average uncertainty or information content. As $$\alpha$$ approaches 0, Rényi entropy converges to the minimum value, representing the most certain or least diverse distribution. Conversely, as $$\alpha$$ approaches infinity, Rényi entropy approaches the maximum value, indicating a uniform or maximally diverse distribution.
>
>    Rényi entropy has applications in various fields, including information theory, statistical physics, and data analysis. It offers a way to quantify the diversity or randomness of a system beyond the traditional Shannon entropy, allowing for a more nuanced understanding of information content and structure.
>
{: .details}


For further information on how to choose the best diversity metric also check:

1. [Measuring the diversity of the human microbiota with targeted next-generation sequencing](https://academic.oup.com/bib/article/19/4/679/2871295) {% cite BonillaRosso.2012 %}
2. [Understanding microbial community diversity metrics derived from metagenomes: performance evaluation using simulated data sets](https://academic.oup.com/femsec/article/82/1/37/567182) {% cite Finotello.2018 %}

# Conclusion

In this tutorial, we look how to calculate $$\alpha$$ and $$\beta$$ diversity from microbiome data. We apply **Krakentools** to calculate the $$\alpha$$ and $$\beta$$ diversity of two microbiome sample datasets.

tools_json:
["Krakentools: Calculate alpha diversity", "toolshed.g2.bx.psu.edu/repos/iuc/krakentools_alpha_diversity/krakentools_alpha_diversity/1.2+galaxy1", "Krakentools: Calculate beta diversity (Bray-Curtis dissimilarity)", "toolshed.g2.bx.psu.edu/repos/iuc/krakentools_beta_diversity/krakentools_beta_diversity/1.2+galaxy1"]


topic: microbiome
tutorial: nanopore-16S-metagenomics
title: 16S Microbial analysis with Nanopore data
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: ["metabarcoding", "16S", "nanopore", "microgalaxy", "plants"]
content_hash: 73914667de12a306d88ac460ef5fd109
last_modified: 2026-05-15T22:28:14.020556
zenodo_link: https://zenodo.org/record/4274812
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/nanopore-16S-metagenomics/tutorial.html

content:
__Healthy soils__ are an essential element in maintaining the planet’s ecological balance. That is why their protection must be considered a priority in order to guarantee the well-being of humanity. The alteration of microbial populations often precedes changes in the physical and chemical properties of soils, so monitoring their condition can serve to predict their future evolution, allowing to develop strategies to __mitigate ecosystem damage__.

Advances in sequencing technologies have opened up the possibility of using the study of taxa present in bacterial communities as indicators of soil condition. For such an approach to be possible, variations in microbial populations need to be less affected by spatial factors than by human-derived alterations, which has been confirmed by various investigations ({% cite Hermans2016 %}, {% cite Fierer2006 %}).

Among the environmental variables that have shown the greatest correlation with alterations in the composition of bacterial communities are soil pH, the carbon-nitrogen ratio (C:N), and the co-concentrations of Olsen P (a measure of plant available phosphorus), aluminium and copper (Fig. 1).


![Bacterial bioindicators](../../images/metagenomics-nanopore/bioindicators.jpg "Relationship between bacterial community composition and soil parameters. The radius of each circle represents the taxon abundance that was accounted for by each soil variable, based on adjusted R-squared values from distance-based multivariate multiple regression analyses. The univariate relationship between the abundance of each taxon and soil variables, calculated using Pearson’s correlation coefficient, is represented by the color of the circle. Phyla or classes are ordered according to overall abundance in all the samples, from most abundant (top) to least abundant (bottom). Image published originally in {% cite Hermans2016 %}.")


On the other hand, many species of microorganisms establish complex __symbiotic relationships__ with plant organisms. The fraction of the substrate that is directly influenced by root secretions and associated microorganism is known as __rhizosphere__. Thus, for example, bacteria of the genus _Bacillus_, _Pseudomonas_ or _Burkholderia_ appear associated with the plant roots, protecting them from pathogenic microorganisms.

![Bacterial communities](../../images/metagenomics-nanopore/bacteria_community.png "Representation of the relationships between root microbiomes and plant organisms.")

In this tutorial, we will use sequencing data obtained through the __MinION sequencer__ (Oxford Nanopore Technologies) with two objectives: 1) evaluate the health status of soil samples and 2) study how microbial populations are modified by their interaction with plant roots. This example was inspired by {% cite Brown2017 %}.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Background on data

Next, we will introduce some details about the datasets that we are going to use to perform the analysis.

## The datasets

In this example, we will use a dataset originally hosted in the __NCBI SRA database__, with the accession number [SRP194577](https://www.ncbi.nlm.nih.gov/sra?term=SRP194577). To obtain the samples, the DNA was extracted using the Zymo Research Kit, followed by __PCR amplification of 16S rRNA genes__. Sequencing was carried out on a Nanopore Minion device. We are going to use four datasets, corresponding to two experimental conditions:

- Soil:
	- Surface sample (0-5 cm): `bulk_top.fastq.gz`
	- Deep sample (10-15 cm): `bulk_bottom.fastq.gz`
- Rhizosphere:
	- Surface sample (0-5 cm): `rhizosphere_top.fastq.gz`
	- Deep sample (10-15 cm): `rhizosphere_bottom.fastq.gz`

> <question-title></question-title>
>
>  Why do we sequence the 16S rRNA genes for analyzing microbial communities?
>
> > <solution-title></solution-title>
> > There are several reasons to use these genes as taxonomic markers. First, they are constituent parts of ribosomes, a piece of highly-conserved biological machinery, which makes them universally distributed. On the other hand, these genes present regions with varying degrees of sequence variability, which allows them to be used as a species-specific signature.
> {: .solution}
{: .question}


# Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from Zenodo:
>
>    - Open the file {% icon galaxy-upload %} __upload__ menu
>    - Click on __Collection__ tab
>    - Click of the **Paste/Fetch** button
>    - Paste the Zenodo links and press <kbd>Start</kbd> and <kbd>Build</kbd>
>
>      ```
>      https://zenodo.org/record/4274812/files/bulk_bottom.fastq.gz
>      https://zenodo.org/record/4274812/files/bulk_top.fastq.gz
>      https://zenodo.org/record/4274812/files/rhizosphere_bottom.fastq.gz
>      https://zenodo.org/record/4274812/files/rhizosphere_top.fastq.gz
>      ```
>
>    - Assign a name to the new collection: `soil collection`
>
> {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
{: .hands_on}

# Assess datasets quality

Before starting to work on our datasets, it is necessary to assess their quality. This is an essential step if we aim to obtain a __meaningful downstream analysis__.


> <details-title>FASTQ sequence quality format</details-title>
>
> FASTQ format is a text-based format for storing both a biological sequence and its corresponding quality scores. Both the sequence letter and the quality score are encoded with a single ASCII character for brevity. You can find more info in the [Wikipedia article](https://en.wikipedia.org/wiki/FASTQ_format).
>
{: .details}

## Quality control using FastQC and MultiQC

__FastQC__ is one of the most widely used tools to __check the quality__ of the samples generated by High Throughput Sequencing (HTS) technologies.

> <hands-on-title>Quality check</hands-on-title>
>
> 1. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72+galaxy1) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: `soil collection`
>
> 2. Rename the outputs as `FastQC unprocessed: Raw` and `FastQC unprocessed: Web`
> 3. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy1) %} with the following parameters:
>    - In *"Results"*:
>      - *"Which tool was used generate logs?"*: `FastQC`
>      - {% icon param-collection %} *"Dataset collection"*:
>
>		- `FastQC unprocessed: Raw`
> 4. Click on the {% icon galaxy-eye %} (eye) icon and inspect the generated HTML file
>
>
{: .hands_on}

__FastQC__ provides information on various parameters, such as the range of quality values across all bases at each position. __MultiQC__ allows summarizing the output of different outputs from __FastQC__.

![FastQC plots length](../../images/metagenomics-nanopore/fastqc_sequence_length_distribution_plot.png "Sequence length distribution")

In the first place, we are going to analyse the sequence length distribution of the different datasets.

> <question-title></question-title>
>
> 1. Can you explain the sequence length distribution plot?
>
> > <solution-title></solution-title>
> >
> > The main peak, around 1700 bp, corresponds approximately to the length of the gene coding for 16S rRNA. There is also a secondary peak around 200 bp, which may be due to __truncated amplifications__, or as a result of __non-specific hybridization of primers__ used for PCR.
> >
> {: .solution}
{: .question}


![FastQC plots quality](../../images/metagenomics-nanopore/fastqc_per_base_sequence_quality_plot.png "Per base sequence quality")



If we examine figure 3, we can see that up to 3000 bp the quality of our sequencing data is around a __Phred score__ of 12, which is a relatively low value compared to other sequencing technologies. This is explained because Nanopore reads poses __high error rates in the basecalled reads__ (10% as compared to 1% for Illumina). However, Nanopore sequencing generates __very long reads__ (in theory only limited by the mechanisms of extraction of the genetic material), enabling the sequencing of the complete 16S rRNA gene, which makes it possible to identify bacterial taxa at higher resolution.

![FasTQC plots CG](../../images/metagenomics-nanopore/fastqc_per_sequence_gc_content_plot.png "Per sequence GC content")

Let's have a look at figure 4, which shows the content of the GC by sequence.

> <question-title></question-title>
>
> What can cause GC content to show a bimodal peak?
>
> > <solution-title></solution-title>
> >
> > There are several possible causes for the presence of twin peaks in the GC content. One possible explanation is the presence of adapters in the sequence. Another possible cause is some kind of contamination, such as chimeras.
> >
> {: .solution}
>
{: .question}

> <comment-title></comment-title>
> For more information on the topic, please see [the quality control tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).
{: .comment}


# Improve the dataset quality

In order to improve the quality of our data, we will use two tools presented above, __porechop__ and __fastp__.

## Adapter and chimera removal with porechop

Nanopore sequencing technology requires to ligate adapters to both ends of genomic material to facilitate the strand capture and loading of a processive enzyme at the 5'end, boosting the effectiveness of the sequencing process. Adapter sequences should be removed because they __can interfere with aligment of reads__ to 16S rRNA gene reference database, for which we will use the __porechop__ tool ({% cite rrwick2017 %}).

> <comment-title></comment-title>
> If you are interested in Nanopore sequecing technology, you can find more information in {% cite Jain2016 %}.
>
{: .comment}


On the other hand, __chimeric sequences__ are considered a contaminant and should be removed because they can result in artificial inflation of the microbial diversity. With __Porechop__ you can eliminate them.

> <hands-on-title>Remove adapters with porechop</hands-on-title>
>
> 1. {% tool [Porechop](toolshed.g2.bx.psu.edu/repos/iuc/porechop/porechop/0.2.3) %} with the following parameters:
>    - *"Input FASTA/FASTQ"*: {% icon param-collection %} `soil collection`
>    - *"Output format for the reads"*: `fastq`
> 2. Rename the output as `soil collection trimmed`
>
{: .hands_on}

## Filter sequences with fastp

To increase the specificity of the analysis, we will select the reads with lengths between 1000 bp and 2000 bp, which are more informative from a taxonomic point of view, because they include both preserved and hypervariable regions of the 16S rRNA gene. In addition, sequences will be filtered on a minimum average read quality score of 9, according to the recommendations from {% cite Nygaard2020 %}. This stage will be carried out through the use of __fastp__ ({% cite Chen_2018 %}), an open-source tool designed to process FASTQ files.

> <hands-on-title>Filter sequence with fastp</hands-on-title>
>
> 1. {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.20.1+galaxy0) %} with the following parameters:
>    - *"Single-end or paired reads"*: `Single-end`
>    - {% icon param-collection %} *"Dataset collection"*: `soil collection trimmed`
>        - In *"Adapter Trimming Options"*:
>            - *"Disable adapter trimming"*: `Yes`
>    - In *"Filter Options"*:
>        - In *"Quality filtering options"*:
>            - *"Qualified quality phred"*: `9`
>        - In *"Length filtering options"*:
>            - *"Length required"*: `1000`
>            - *"Maximum length"*: `2000`
>    - In *"Read Modification Options"*:
>        - *"PolyG tail trimming"*: `Disable polyG tail trimming`
>
> 2. Rename the output as `soil collection processed`
>
>
{: .hands_on}

# Re-evaluate datasets quality

After processing the sequences, we are going to analyze them again using __FastQC__ and __MultiQC__ to see if we have managed to correct the anomalies that we had detected.

> <hands-on-title>Quality check</hands-on-title>
>
> 1. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72+galaxy1) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: `soil collection processed`
>
> 4. Rename the outputs as `FastQC processed: Raw` and `FastQC processed: Web`
> 3. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy1) %} with the following parameters:
>    - In *"Results"*:
>      - *"Which tool was used generate logs?"*: `FastQC`
>      - {% icon param-collection %} *"Dataset collection"*:
>
>		- `FastQC processed: Raw`
> 4. Click on the {% icon galaxy-eye %} (eye) icon and inspect the generated HTML file
>
>
{: .hands_on}

We can verify that after processing the samples, the GC content presents a unimodal distribution, which indicates that the anomalies in the sequences have been successfully eliminated (Figure 6).

![FasTQC plots GC](../../images/metagenomics-nanopore/fasqc_per_sequence_content_plot_clean.png "Per sequence GC content in processed samples")

# Assign taxonomic classifications

One of the key steps in metagenomic data analysis is to identify the taxon to which the individual reads belong. Taxonomic classification tools are based on microbial genome databases to identify the origin of each sequence.

## Taxonomic classification with Kraken2

To perform the taxonomic classification we will use __Kraken2__ ({% cite Wood2019 %}). This tool uses the minimizer method to sample the k-mers (all the read's subsequences of length _k_) in a deterministic fashion in order to reduce memory constumption and processing time. In addition, it masks low-complexity sequences from reference sequences by using __dustmasker__.

{% snippet topics/microbiome/faqs/kraken.md %}

For this tutorial, we will use the __SILVA database__ ({% cite Quast2012 %}). It includes over 3.2 million 16S rRNA sequences from the _Bacteria_, _Archaea_ and _Eukaryota_ domains.

> <hands-on-title>Assign taxonomic labels with Kraken2</hands-on-title>
>
> 1. {% tool [Kraken2](toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.0.8_beta+galaxy0) %} with the following parameters:
>    - *"Single or paired reads"*: `Single`
>    - {% icon param-collection %} *"Dataset collection"*: `soil collection processed`
>    - *"Print scientific names instead of just taxids"*: `Yes`
>    - *"Confidence"*: `0.1`
>    - In *"Create Report"*:
>        - *"Print a report with aggregrate counts/clade to file"*: `Yes`
>        - *"Format report output like Kraken 1's kraken-mpa-report"*: `Yes`
>    - *"Select a Kraken2 database"*: `Silva (Created: 2020-06-24T164526Z, kmer-len=35, minimizer-len=31, minimizer-spaces=6)`
>
>    > <comment-title></comment-title>
>    >
>    > A confidence score of 0.1 means that at least 10% of the k-mers should match entries in the database. This value can be reduced if a less restrictive taxonomic assignation is desired.
>    {: .comment}
>
{: .hands_on}



# Analyze taxonomic assigment

Once we have assigned the corresponding taxa to each sequence, the next step is to properly visualize the data, for which we will use the __Krona pie chart__ tool ({% cite Ondov_2011 %}). But before that, we need to adjust the format of the data output from __Kraken2__.

> <hands-on-title>Adjust dataset format</hands-on-title>
>
> 1. {% tool [Reverse](toolshed.g2.bx.psu.edu/repos/iuc/datamash_reverse/datamash_reverse/1.1.0) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: `Report: Kraken2 on collection`
>
> 2. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: `Reverse on collection`
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"Find pattern"*: `\|`
>            - *"Replace with"*: `\t`
>
> 3. {% tool [Remove beginning](Remove beginning1) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: `Replace Text on collection`
>
{: .hands_on}

## Visualize the taxonomical classification with Krona

__Krona__ allows hierarchical data to be explored with zooming, multi-layered pie charts. With this tool, we can easily visualize the composition of the bacterial communities and compare how the populations of microorganisms are modified according to the conditions of the environment.

> <hands-on-title>Visualize metagenomics analysis results</hands-on-title>
>
> 1. {% tool [Krona pie chart](toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1) %} with the following parameters:
>    - *"What is the type of your input data"*: `Tabular`
>    - {% icon param-collection %} *"Dataset collection"*: `Remove beginning on collection`
>    - *"Provide a name for the basal rank"*: `Bacteria`
>
{: .hands_on}

Let's take a look at the result. Using the search bar we can check if certain taxa are present.

<iframe id="krona" src="krona_all.html" frameBorder="0" width="100%" height="900px"> ![Krona](../../images/metagenomics-nanopore/krona.png) </iframe>

> <question-title></question-title>
>
> 1. What can we say about the health status of the soil samples?
> 2. Are there significant differences in the bacterial population structure between soil samples and rhizosphere samples?
>
> > <solution-title></solution-title>
> >
> > 1. The low presence of _Alphaproteobacteria_ (including members of the order _Rhizobiales_), as well as the abundance of _Bacteroidetes_ and _Gammaproteobacteria_, suggest that the soil is highly exposed to phosphorus. This mineral is strongly associated with the anthropogenic activity as it is an important component of agrochemicals.
> > 2. We can observe that there are important differences in the structure of the bacterial populations between the soil and rhizosphere samples. Particularly significant is the increase in phylum _Planctomycetes_, which are usually abundant in the rhizosphere.
> >
> {: .solution}
>
{: .question}

# Conclusion

In this tutorial we used MinION Nanopore sequencing data to study the health status of soil samples and the structure of bacterial populations. The results indicate that the soil suffers some degree of erosion as a result of their exposure to agrochemicals. We have also been able to study how the composition of microbial communities is modified in presence of plant organisms.

tools_json:
["FastQC", "toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72+galaxy1", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy1", "Porechop", "toolshed.g2.bx.psu.edu/repos/iuc/porechop/porechop/0.2.3", "fastp", "toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.20.1+galaxy0", "Kraken2", "toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.0.8_beta+galaxy0", "Reverse", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_reverse/datamash_reverse/1.1.0", "Replace Text", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2", "Remove beginning", "Remove beginning1", "Krona pie chart", "toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1"]


topic: microbiome
tutorial: taxonomic-profiling
title: Taxonomic Profiling and Visualization of Metagenomic Data
difficulty: introductory
time_estimation: 2H
requirements_json: []
tags_json: ["metagenomics", "taxonomic profiling", "microgalaxy"]
content_hash: 967d8e23c2b15e9db879a8687fa634f7
last_modified: 2026-05-27T09:01:49.387742
zenodo_link: https://zenodo.org/record/7871630
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/taxonomic-profiling/tutorial.html

content:
The term **"microbiome"** describes "a characteristic microbial community occupying a reasonably well-defined habitat which has distinct physio-chemical properties. The term thus not only refers to the microorganisms involved but also encompasses their theatre of activity" ({% cite whipps1988mycoparasitism %}).

Microbiome data can be gathered from different environments such as soil, water or the human gut. The biological interest lies in general in the question how the microbiome present at a specific site influences this environment. To study a microbiome, we need to use indirect methods like metagenomics or metatranscriptomics.

**Metagenomic samples** contain DNA from different organisms at a specific site, where the sample was collected. Metagenomic data can be used to find out which organisms coexist in that niche and which genes are present in the different organisms.
**Metatranscriptomic samples** include the transcribed gene products, thus RNA, that therefore allow to not only study the presence of genes but additionally their expression in the given environment. The following tutorial will focus on metagenomics data, but the principle is the same for metatranscriptomics data.

The investigation of microorganisms present at a specific site and their relative abundance is also called **"microbial community profiling"**.
The main objective is to identify the microorganisms that are present within the given sample. This can be achieved for all known microbes, where the DNA sequence specific for a certain species is known.

For that we try to **identify the taxon** to which each individual read belongs.

{% snippet topics/microbiome/faqs/taxon.md %}

For metagenomic data analysis we start with sequences derived from DNA fragments that are isolated from the sample of interest. Ideally, the sequences from all microbes in the sample are present. The underlying idea of taxonomic assignment is to compare the DNA sequences found in the sample (reads) to DNA sequences of a database. When a read matches a database DNA sequence of a known microbe, we can derive a list with microbes present in the sample.

When talking about taxonomic assignment or taxonomic classification, most of the time we actually talk about two methods, that in practice are often used interchangeably:
- **taxonomic binning**: the clustering of individual sequence reads based on similarities criteria and assignation of clusters to reference taxa
- **taxonomic profiling**: classification of individual reads to reference taxa to extract the relative abundances of the different taxa

The reads can be obtained from **amplicon sequencing**  (e.g. 16S, 18S, ITS), where only specific gene or gene fragments are targeted (using specific primers) or **shotgun metagenomic sequencing**, where all the accessible DNA of a mixed community is amplified (using random primers). 
This tutorial focuses on reads obtained from shotgun metagenomic sequencing. 

## Taxonomic profiling

Tools for taxonomic profiling can be divided into three groups. Nevertheless, all of them require a pre-computed database based on previously sequenced microbial DNA or protein sequences.
1. **DNA-to-DNA**: comparison of sequencing reads with genomic databases of DNA sequences with tools like Kraken ({% cite Wood2014 %})
2. **DNA-to-Protein** : comparison of sequencing reads with protein databases (more computationally intensive because all six frames of potential DNA-to amino acid translations need to be analyzed) with tools like DIAMOND)
3. **Marker based**: searching for marker genes (e.g. 16S rRNA sequence) in reads, which is quick, but introduces bias, with tools like MetaPhlAn ({% cite blanco2023extending %})

The comparison of reads to database sequences can be done in different ways, leading to three different types of taxonomic assignment:

- **Genome based** approach

   Reads are aligned to reference genomes. Considering the coverage and breadth, genomes are used to measure genome abundance. Furthermore, genes can be analyzed in genomic context. Advantages of this method are the high detection accuracy, that the unclassified percentage is known, that all SNVs can be detected and that high-resolution genomic comparisons are possible. This method takes medium compute cost.

- **Gene based** approach

   Reads are aligned to reference genes. Next, marker genes are used to estimate species abundance. Furthermore, genes can be analyzed in isolation for presence or absence in a specific condition. The major advantage is the detection of the pangenome (entire set of genes within a species). Major disadvantages are the high compute cost, low detection accuracy and that the unclassified percentage is unknown. At least intragenic SNVs can be detected and low-resolution genomic comparison is possible.

- **k-mer based** approach

   Databases as well as the samples DNA are broken into strings of length $$k$$ for comparison. From all the genomes in the database, where a specific k-mer is found, a lowest common ancestor (LCA) tree is derived and the abundance of k-mers within the tree is counted. This is the basis for a root-to-leaf path calculation, where the path with the highest score is used for classification of the sample. By counting the abundance of k-mers, also an estimation of relative abundance of taxa is possible. The major advantage of k-mer based analysis is the low compute cost. Major disadvantages are the low detection accuracy, that the unclassified percentage is unknown and that there is no gene detection, no SNVs detection and no genomic comparison possible. An example for a k-mer based analysis tool is Kraken, which will be used in this tutorial.

After this theoretical introduction, let's now get hands on analyzing an actual dataset!

## Background on data

The dataset we will use for this tutorial comes from an oasis in the Mexican desert called Cuatro Ciénegas ({% cite Okie.2020 %}). The researchers were interested in genomic traits that affect the rates and costs of biochemical information processing within cells. They performed a whole-ecosystem experiment, thus fertilizing the pond to achieve nutrient enriched conditions.

Here we will use 2 datasets:
- `JP4D`: a microbiome sample collected from the Lagunita Fertilized Pond
- `JC1A`: a **control** samples from a control mesocosm.

The datasets differ in size, but according to the authors this doesn't matter for their analysis of genomic traits. Also, they underline that differences between the two samples reflect trait-mediated ecological dynamics instead of microevolutionary changes as the duration of the experiment was only 32 days. This means that depending on available nutrients, specific lineages within the pond grow more successfully than others because of their genomic traits.

The datafiles are named according to the first four characters of the filenames.
It is a collection of paired-end data with R1 being the forward reads and R2 being the reverse reads. Additionally, the reads have been trimmed using __cutadapt__ as explained in the [Quality control tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare Galaxy and data

Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

Now, we need to import the data

> <hands-on-title>Import datasets</hands-on-title>
>
> 1. Import the following samples via link from [Zenodo]({{ page.zenodo_link }}) or Galaxy shared data libraries:
>
>    ```text
>    {{ page.zenodo_link }}/files/JC1A_R1.fastqsanger.gz
>    {{ page.zenodo_link }}/files/JC1A_R2.fastqsanger.gz
>    {{ page.zenodo_link }}/files/JP4D_R1.fastqsanger.gz
>    {{ page.zenodo_link }}/files/JP4D_R2.fastqsanger.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Create a paired collection.
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
{: .hands_on}

# k-mer based taxonomic assignment with Kraken2

Our input data is the DNA reads of microbes present at Cuatro Ciénegas.

To find out which microorganisms are present, we will compare the reads of the sample to a reference database, i.e. sequences of known microorganisms stored in a database, using **Kraken2** ({% cite wood2019improved %}).

{% snippet topics/microbiome/faqs/kraken.md %}

For this tutorial, we will use the PlusPF database which contains the Standard (archaea, bacteria, viral, plasmid, human, UniVec_Core), protozoa and fungi data.

> <details-title>What is in the PlusPF database</details-title>
> 
> Database | Origin
> --- | ---
> Archaea | RefSeq complete archaeal genomes/proteins
> Bacteria | RefSeq complete bacterial genomes/proteins
> Plasmid | RefSeq plasmid nucleotide/protein sequences
> Viral | RefSeq complete viral genomes/proteins
> Human | GRCh38 human genome/proteins
> Fungi | RefSeq complete fungal genomes/proteins
> Plant | RefSeq complete plant genomes/proteins
> Protozoa | RefSeq complete protozoan genomes/proteins
> UniVec_Core | A subset of UniVec, NCBI-supplied database of vector, adapter, linker, and primer sequences that may be contaminating sequencing projects and/or assemblies, chosen to minimize false positive hits to the vector database
>
> The databases have been prepared for Kraken2 by Ben Leagmead and [details can found on his page](https://benlangmead.github.io/aws-indexes/k2).
{: .details}

> <hands-on-title>Assign taxonomic labels with Kraken2</hands-on-title>
>
> 1. {% tool [Kraken2](toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.1.3+galaxy1) %} with the following parameters:
>    - *"Single or paired reads"*: `Paired`
>      - {% icon param-collection %} *"Collection of paired reads"*: Input paired collection
>    - *"Confidence"*: `0.1`
>
>      A confidence score of 0.1 means that at least 10% of the k-mers should match entries in the database. This value can be reduced if a less restrictive taxonomic assignation is desired.
>
>    - In *"Create Report"*:
>        - *"Print a report with aggregrate counts/clade to file"*: `Yes`
>    - *"Select a Kraken2 database"*: most recent `Prebuilt Refseq indexes: PlusPF`
>
{: .hands_on}

**Kraken2** will create two outputs for each dataset

- **Classification**: tabular files with one line for each sequence classified by Kraken and 5 columns:

   1. `C`/`U`: a one letter indicating if the sequence classified or unclassified
   2. Sequence ID as in the input file
   3. NCBI taxonomy ID assigned to the sequence, or 0 if unclassified
   4. Length of sequence in bp (`read1|read2` for paired reads)
   5. A space-delimited list indicating the lowest common ancestor (LCA) mapping of each k-mer in the sequence     
      
      For example, `562:13 561:4 A:31 0:1 562:3` would indicate that:
      1. The first 13 k-mers mapped to taxonomy ID #562
      2. The next 4 k-mers mapped to taxonomy ID #561
      3. The next 31 k-mers contained an ambiguous nucleotide
      4. The next k-mer was not in the database
      5. The last 3 k-mers mapped to taxonomy ID #562     
      
      `|:|` indicates end of first read, start of second read for paired reads

   For JC1A:

   ```
   Column 1	Column 2	Column 3	Column 4	Column 5
   U 	MISEQ-LAB244-W7:91:000000000-A5C7L:1:1101:13417:1998 	0 	151|190 	A:17 0:15 2055:5 0:1 2220095:5 0:74 |:| 0:3 A:53 2:2 0:32 204455:1 2823043:5 0:60
   U 	MISEQ-LAB244-W7:91:000000000-A5C7L:1:1101:15782:2187 	0 	169|173 	0:101 37329:1 0:33 |:| 0:10 2751189:5 0:30 1883:2 0:39 2609255:5 0:48
   U 	MISEQ-LAB244-W7:91:000000000-A5C7L:1:1101:11745:2196 	0 	235|214 	0:173 2282523:5 2746321:2 0:21 |:| 0:65 2746321:2 2282523:5 0:108
   U 	MISEQ-LAB244-W7:91:000000000-A5C7L:1:1101:18358:2213 	0 	251|251 	0:35 281093:5 0:3 651822:5 0:145 106591:3 0:21 |:| 0:64 106591:3 0:145 651822:5
   U 	MISEQ-LAB244-W7:91:000000000-A5C7L:1:1101:14892:2226 	0 	68|59 	0:34 |:| 0:25
   U 	MISEQ-LAB244-W7:91:000000000-A5C7L:1:1101:18764:2247 	0 	146|146 	0:112 |:| 0:112
   C 	MISEQ-LAB244-W7:91:000000000-A5C7L:1:1101:12147:2252 	9606 	220|220 	9606:148 0:19 9606:19 |:| 9606:19 0:19 9606:148
   ```

   > <question-title></question-title>
   >
   > For JC1A sample
   > 1. Is the first sequence in the file classified or unclassified?
   > 2. What is the taxonomy ID assigned to the first classified sequence?
   > 3. What is the corresponding taxon?
   >
   > > <solution-title></solution-title>
   > > 1. unclassified
   > > 2. 9606, for the line 7
   > > 3. 9606 corresponds to Homo sapiens when looking at NCBI.
   > {: .solution}
   {: .question}

- **Report**: tabular files with one line per taxon and 6 columns or fields

   1. Percentage of fragments covered by the clade rooted at this taxon
   2. Number of fragments covered by the clade rooted at this taxon
   3. Number of fragments assigned directly to this taxon
   4. A rank code, indicating
      - (U)nclassified
      - (R)oot
      - (D)omain
      - (K)ingdom
      - (P)hylum
      - (C)lass
      - (O)rder
      - (F)amily
      - (G)enus, or
      - (S)pecies

      Taxa that are not at any of these 10 ranks have a rank code that is formed by using the rank code of the closest ancestor rank with a number indicating the distance from that rank. E.g., `G2` is a rank code indicating a taxon is between genus and species and the grandparent taxon is at the genus rank.

   5. NCBI taxonomic ID number
   6. Indented scientific name

   ```
   Column 1	Column 2	Column 3	Column 4	Column 5	Column 6
   76.85 	105397 	105397 	U 	0 	unclassified
   23.15 	31742 	1197 	R 	1 	root
   22.20 	30450 	312 	R1 	131567 	cellular organisms
   12.58 	17256 	3768 	D 	2 	Bacteria
   8.77 	12028 	2868 	P 	1224 	Proteobacteria
   4.94 	6779 	3494 	C 	28211 	Alphaproteobacteria
   1.30 	1782 	1085 	O 	204455 	Rhodobacterales
   0.43 	593 	461 	F 	31989 	Rhodobacteraceae
   0.05 	74 	53 	G 	265 	Paracoccus
   ```

   > <question-title></question-title>
   >
   > 1. What are the percentage on unclassified for JC1A and JP4D?
   > 2. What are the kindgoms found for JC1A and JP4D?
   > 3. Where might the eukaryotic DNA come from?
   > 4. How is the diversity of Proteobacteria in JC1A and JP4D?
   >
   > > <solution-title></solution-title>
   > >
   > > 1. 77% for JC1A and 90% for JP4D
   > > 2. Kindgoms:
   > >    - JC1A: 13% Bacteria, 9% Eukaryota, 0.03% Virus
   > >    - JP4D: 9% Bacteria, 0.7% Eukaryota
   > > 3. It seems to be human contamination
   > > 4. JC1A seems to have a big diversity of classes and species of Proteobacteria. JP4D seems more dominated by Aphaproteobacteria.
   > {: .solution}
   >
   {: .question}

Getting an overview of the assignation is not straightforward with the **Kraken2** outputs directly. We can use visualisation tools for that.

A "simple and worthwile addition to Kraken for better abundance estimates" ({% cite Ye.2019 %}) is called __Bracken__ (Bayesian Reestimation of Abundance after Classification with Kraken). Instead of only using proportions of classified reads, it takes a probabilistic approach to generate final abundance profiles. It works by re-distributing reads in the taxonomic tree: "Reads assigned to nodes above the species level are distributed down to the species nodes, while reads assigned at the strain level are re-distributed upward to their parent species" ({% cite Lu.2017 %}).

> <hands-on-title>Estimate species abundance with Bracken</hands-on-title>
>
> 1. {% tool [Bracken](toolshed.g2.bx.psu.edu/repos/iuc/bracken/est_abundance/3.1+galaxy0) %} with the following parameters:
>     - {% icon param-collection %} *"Kraken report file"*: **Report** output of **Kraken**
>     - *"Select a kmer distribution"*: `PlusPF`, same as for Kraken
>
>        It is important to choose the same database that you also chose for Kraken2
>
>     - *"Level"*: `Species`
>     - *"Produce Kraken-Style Bracken report"*: `yes`
>
{: .hands_on}

# Visualization of taxonomic assignment

Once we have assigned the corresponding taxa to each sequence, the next step is to properly visualize the data. There are several tools for that:
- __Krona__ ({% cite Ondov.2011 %})
- __Phinch__ ({% cite Bik.2014 %})
- __Pavian__ ({% cite Breitwieser.2020 %})

## Visualisation using Krona

__Krona__ creates an interactive HTML file allowing hierarchical data to be explored with zooming, multi-layered pie charts. With this tool, we can easily visualize the composition of the bacterial communities and compare how the populations of microorganisms are modified according to the conditions of the environment.

Kraken outputs can not be given directly to **Krona**, they first need to be converted.

__Krakentools__ ({% cite Lu.2017 %}) is a suite of tools to work on Kraken outputs. It include a tool designed to translate results of the Kraken metagenomic classifier to the full representation of NCBI taxonomy. The output of this tool can be directly visualized by the Krona tool.

> <hands-on-title>Convert Kraken report file</hands-on-title>
>
> 1. {% tool [Krakentools: Convert kraken report file](toolshed.g2.bx.psu.edu/repos/iuc/krakentools_kreport2krona/krakentools_kreport2krona/1.2+galaxy2) %} with the following parameters:
>    - {% icon param-collection %} *"Kraken report file"*: **Report** collection of **Kraken**
>
> 2. Inspect the generated output for JC1A
{: .hands_on}


> <question-title></question-title>
>
> ```
> 3869 	k__Bacteria
> 2868 	k__Bacteria 	p__Proteobacteria
> 3494 	k__Bacteria 	p__Proteobacteria 	c__Alphaproteobacteria
> 1085 	k__Bacteria 	p__Proteobacteria 	c__Alphaproteobacteria 	o__Rhodobacterales
> 461 	k__Bacteria 	p__Proteobacteria 	c__Alphaproteobacteria 	o__Rhodobacterales 	f__Rhodobacteraceae
> 53 	k__Bacteria 	p__Proteobacteria 	c__Alphaproteobacteria 	o__Rhodobacterales 	f__Rhodobacteraceae 	g__Paracoccus
> 10 	k__Bacteria 	p__Proteobacteria 	c__Alphaproteobacteria 	o__Rhodobacterales 	f__Rhodobacteraceae 	g__Paracoccus 	s__Paracoccus_pantotrophus
> 6 	k__Bacteria 	p__Proteobacteria 	c__Alphaproteobacteria 	o__Rhodobacterales 	f__Rhodobacteraceae 	g__Paracoccus 	s__Paracoccus_sanguinis
> 4 	k__Bacteria 	p__Proteobacteria 	c__Alphaproteobacteria 	o__Rhodobacterales 	f__Rhodobacteraceae 	g__Paracoccus 	s__Paracoccus_sp._AK26
> 1 	k__Bacteria 	p__Proteobacteria 	c__Alphaproteobacteria 	o__Rhodobacterales 	f__Rhodobacteraceae 	g__Paracoccus 	s__Paracoccus_sp._MA
> ```
>
> 1. What are the different columns?
> 2. What are the lines?
>
> > <solution-title></solution-title>
> >
> > 1. Column 1 seems to correspond to the number of fragments covered by a taxon, the columns after represent the different taxonomic level (from kingdom to species)
> > 2. A line is a taxon with its hierarchy and the number of reads assigned to it
> {: .solution}
>
{: .question}

Let's now run **Krona**

> <hands-on-title>Generate Krona visualisation</hands-on-title>
> 1. {% tool [Krona pie chart](toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1+galaxy0) %} with the following parameters:
>    - *"Type of input data"*: `Tabular`
>    - {% icon param-collection %} *"Input file"*: output of **Krakentools**
>
> 2. Inspect the generated file
{: .hands_on}

<iframe id="krona" src="krona-kraken.html" frameBorder="0" width="100%" height="900px"> ![Krona chart with multi-layered pie chart representing the community profile with in the center the higher taxonomy levels (i.e. domain) and on the exterior the more detailed ones (i.e. species)](./images/krona-kraken.png) </iframe>

> <question-title></question-title>
>
> 1. What are the percentage on unclassified for JC1A and JP4D?
> 2. What are the kindgoms found for JC1A and JP4D?
> 3. Where might the eukaryotic DNA come from?
> 4. How is the diversity of Proteobacteria in JC1A and JP4D?
>
> > <solution-title></solution-title>
> >
> > 1. 78% for JC1A and 90% for JP4D
> > 2. Kindgoms:
> >    - JC1A: 13% Bacteria, 10% Eukaryota, 0.03% Virus
> >    - JP4D: 10% Bacteria, 0.7% Eukaryota
> > 3. It seems to be human contamination
> > 4. JC1A seems to have a big diversity of classes and species of Proteobacteria. JP4D seems more dominated by Aphaproteobacteria.
> {: .solution}
>
{: .question}


## Visualization using Pavian

__Pavian__ (pathogen visualization and more) ({% cite Breitwieser.2020 %}) is an interactive visualization tool for metagenomic data. It was developed for the clinical metagenomic problem to find a disease-causing pathogen in a patient sample, but it is useful to analyze and visualize any kind of metagenomics data.

> <hands-on-title>Launch Pavian</hands-on-title>
> 1. {% tool [Pavian](interactive_tool_pavian) %} with the following paramters:
>     - {% icon param-collection %} *"Kraken and MetaPhlAn-style reports"*: **Report** collection of **Kraken**
>
{: .hands_on}

**Pavian** runs a Galaxy Interactive tool. You can access it when it become orange.

> <hands-on-title>Interact with Pavian</hands-on-title>
>
> 1. Open **Pavian**
>
>    {% snippet faqs/galaxy/interactive_tools_open.md %}
>
> 2. Import data
>    1. Click on `Use data on server`
>    2. Select both samples
>    3. Click on `Read selected directories`
>    4. Check you have a table in `Available sample sets` looks like
>
>       X | FormatOK | Include | Name | ReportFile | ReportFilePath
>       --- | --- | --- | --- | --- | ---
>       1 | X | X | JP4AD | JP4AD | /home/shiny//JP4AD
>       2 | X | X | JC1A | JC1A | /home/shiny//JC1A
>
>    5. Click on `Save table`
>
> 3. Click on `Results Overview` in the left panel
>
{: .hands_on}

This page shows the summary of the classifications in the selected sample set:

![Screenshot of a table with 2 rows (JP4A and JC1A) and columns: Name, Number of raw reads, Classified reads, Chordate reads, Artificial reads, Unclassified reads, Microbial reads, Bacterial reads, Viral reads, Fungal reads, Protozoan reads. The cells have a barchart that shows the relation of the cell value to other cell values in the same category](./images/pavian-kraken-results-overview.png)

> <question-title></question-title>
>
> 1. Does both sample have same size?
> 1. What are the percentage of classified reads for JC1A and JP4D?
> 2. Are the percentage of bacterial reads similar?
>
> > <solution-title></solution-title>
> >
> > 1. JP4D has much more reads than JC1A
> > 2. 10.2% for JP4D and 23.1% for JC1A
> > 3. 12.6% for JC1A and 9.22% for JP4D. So similar magnitude orders
> {: .solution}
>
{: .question}

Let's now inspect assignements to reads per sample.

> <hands-on-title>Inspect samples with Pavian</hands-on-title>
>
> 1. Click on `Sample` in the left panel
> 2. Select `JC1A` in the `Select sample` drop-down on the top
>
>    The first view gives a Sankey diagram for one sample:
>
>    ![Sankey plot with taxonomy hierarchy from domain on the left to species on the right](./images/pavian-kraken-sankey-JC1A.png)
>
>    > <question-title></question-title>
>    >
>    > 1. What is a Sankey diagram?
>    > 2. What are the different set of values represented as the horizontal axis?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. A sankey diagram is a visualization used to depict a flow from one set of values to another
>    > > 2. The taxonomy hierarchy from domain on the left to species on the right
>    > {: .solution}
>    >
>    {: .question}
>
> 3. Click on `Proteobacteria` in the Sankey plot
> 4. Inspect the created graph on the right
>
>    ![Bar plot with the number of reads across all samples for Proteobacteria. JP4A bar is much higher than JC1A one. The turqoise bar shows the number of reads that are identified at the specific taxon; the orange bar shows the number of reads identified at children of the specified taxon.](./images/pavian-kraken-proteobacteria.png)
>
>    > <question-title></question-title>
>    >
>    > 1. Are the number of reads assigned to Proteobacteria similar for both samples?
>    > 2. Why?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. JP4A has many more reads assigned to Proteobacteria than JC1A
>    > > 2. JP4A has many more reads initialls
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}

We would like now to compare both samples.

> <hands-on-title>Inspect samples with Pavian</hands-on-title>
>
> 1. Click on `Comparison` in the left panel
> 2. Select `%`  and unclick `Reads` in the blue area drop-down on the top
> 3. Click on `Domain` green button
>
>    ![Screenshot of a table with 4 rows (Bacteria, Eukaryota, Archaea, Viruses) and / columns (Name, Rank, TID, Max, JP4A, JC1A, Lineage)](./images/pavian-kraken-comparison-domain.png)
>
>    > <question-title></question-title>
>    >
>    > Is there similar proportion of Bacteria in both samples?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > JP4D has much higher proportion of Bacteria (> 93%>) than JC1A (57%), which contains quite a lot of Eukaryote
>    > {: .solution}
>    >
>    {: .question}
>
> 4. Select `Homo sapiens` in the `Filter taxa` box below the green buttons
>
>    > <question-title></question-title>
>    >
>    > Is there similar proportion of Bacteria in both samples?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > After human filtering, both samples have similar proportion of Bacteria
>    > {: .solution}
>    >
>    {: .question}
>
> 3. Click on `Class` green button
>
>    > <question-title></question-title>
>    >
>    > 1. How are the diversities of classes in both samples?
>    > 2. What could it biologically mean given that JC1A is a control and JP4D a sample from fertilized pond?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. JP4D seems highly dominated by a Alphaproteobacteria class. JC1A has also a majority of Alphaproteobacteria, but also significant proportions of Betaproteobacteria, Gammaproteobacteria, Flavobacteria, Actinomycetia
>    > > 2. Alphaproteobacteria seems to have a survival advantage in the new environment. According to the authors this correlates with specific genomic traits that enable them to cope better with high nutrient availability.
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}

Once you are done with Pavian, you should delete it in your history so the corresponding job is killed.

> <details-title>Visualize the taxonomical classification with Phinch</details-title>
>
> __Phinch__ ({% cite Bik.2014 %}) is another tools to visualize large biological datasets like our taxonomic classification. Taxonomy Bar Charts, Bubble Charts, Sankey Diagrams, Donut Partitions and Attributes Column Chart can be generated using this tool.
>
> As a first step, we need to convert the Kraken output file into a kraken-biom file to make it accessible for Phinch. For this, we need to add a metadata file, provided on [Zenodo]({{ page.zenodo_link }}/files/metadata.tabular).
> When generating a metadata file for your own data, you can take this as an example and apply the [general guidelines](http://qiime.org/documentation/file_formats.html).
>
> > <hands-on-title>Phinch</hands-on-title>
> > 1. Import the metadata tabular from [Zenodo]({{ page.zenodo_link }}) or Galaxy shared data libraries:
> >
> >    ```text
> >    {{ page.zenodo_link }}/files/metadata.tabular
> >    ```
> >
> > 1. Use {% tool [Kraken-biom](toolshed.g2.bx.psu.edu/repos/iuc/kraken_biom/kraken_biom/1.2.0+galaxy1) %} to convert Kraken2 report into the correct format for phinch with the following parameters.
> >    - {% icon param-collection %} *"Input"*: **Report** output of **Kraken2**
> >    - {% icon param-file %} *"Sample Metadata file"*: Metadata tabular
> >    - *"Output format"*: `JSON`
> >
> > 2. {% tool [Phinch Visualisation](interactive_tool_phinch) %} with the following paramters:
> >    - *"Input"*: `Kraken-biom output file`
> {: .hands_on}
>
> **Phinch** runs a Galaxy Interactive tool. You can access it when it become orange.
>
> > <hands-on-title>Interact with Phinch</hands-on-title>
> >
> > 1. Open **Phinch**
> >
> >    {% snippet faqs/galaxy/interactive_tools_open.md %}
> >
> >    The first pages shows an overview of your samples. Here, you have the possibility to further filter your data, for example by date or location, depending on which information you provided in your metadata file.
> > 
> >    > <question-title></question-title>
> >    >
> >    > 1. How many sequence reads do the samples contain?
> >    >
> >    > > <solution-title></solution-title>
> >    > >
> >    > > 1. JC1A (Phinch name: 0) contains 56008 reads, while JP4D (Phinch name: 1) contains 242438 reads
> >    > {: .solution}
> >    {: .question}
> >
> > 2. Click on `Proceed to gallery` to see an overview of all visualization options.
> {: .hands_on}
>
> Let's have a look at the **taxonomy bar chart**. Here, you can see the abundance of different taxa depicted in different colors in your samples. On top of the chart you can select which rank is supposed to be shown in the chart.  You can also change the display options to for example switch between value und percentage. 
>
> > <question-title></question-title>
> >
> > 1. What information can you get from hovering over a sample?
> > 2. How many percent of the sample reads are bacteria and how many are eukaryota?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. the taxon’s name and the taxonomy occurrence in the sample
> > > 2. choose kingdom and hover over the bars to find "taxonomy occurence in this sample":
> > >    - Sample 0: 75,65 % bacteria; 24,51 % eukaryota
> > >    - Sample 1: 92,70 % bacteria; 6,87 % eukaryota
> > {: .solution}
> {: .question}
>
> Let's go back to the gallery and choose the **bubble chart**. Here, you can find the distribution of taxa across the whole dataset at the rank that you can choose above the chart. When hovering over the bubbles, you get additional information concerning the taxon. 
> 
> > <question-title></question-title>
> >
> > 1. Which is the most abundant Class?
> > 2. How many reads are found in both samples?
> >
> > > <solution-title></solution-title>
> > >
> > > To order the bubbles according to their size you can choose the `list` option shown right next to the taxonomy level. Clicking on one bubble gives you the direct comparison of the distribution of this taxon in the different samples. 
> > > 1. The most abundant Class is Alphaproteobacteria
> > > 2. With 18.114 reads in Sample 0 and 153.230 reads in sample 1
> > {: .solution}
> {: .question}
>
>
> Another displaying option is the **Sankey diagram**, that is depicting the abundance of taxonomies as a flow chart. Again, you can choose the taxonomy level that you want to show in your diagram. When clicking on one bar of the diagram, this part is enlarged for better view.
>
> The **donut partition** summarizes the microbial community according to non-numerical attributes. In the drop-down menu at the top right corner, you can switch between the different attributes provided in the metadata file. In our case, you can for example choose the 'environmental medium' to see the difference between sediment and water (It doesn't really make a lot of sense in our very simple dataset, as this will show the same result as sorting them by sample 0 and 1, but if attributes change across different samples this might be an interesting visualization option). When clicking on one part of the donut you will also find the distribution of the taxon across the samples. On the right hand side you can additionally choose if you’d like to have dynamic y axis or prefer standard y axis to compare different donuts with each other.
>
> The **attributes column chart** summarizes the microbial community according to numerical attributes. In the drop-down menu at the top right corner, you can switch between the different attributes provided in the metadata file. In our case, you can for example choose the 'geographic location' to (again, it doesn't really make a lot of sense in our very simple dataset, as this will show the same result as sorting them by sample 0 and 1, but if attributes change across different samples this might be an interesting visualization option).
> 
> Once you are done with Phinch, you should delete it in your history so the corresponding job is killed.
{: .details}

# Choosing the right tool

When it comes to taxonomic assignment while analyzing metagenomic data, **Kraken2** is not the only tool available. Several papers do benchmarking of different tools ({% cite Meyer.2022 %},{% cite Sczyrba.2017 %},{% cite Ye.2019 %}).

> <details-title>Benchmarking taxonomic classification tools</details-title>
>
> The benchmarking papers present different methods for comparing the available tools:
> - The **CAMI challenge** is based on results of different labs that each used the CAMI dataset to perform their analysis on and send it back to the authors.
> - {% cite Ye.2019 %} performed all the analysis themselves.
>
> Additionally, the datasets used for both benchmarking approaches differ:
> - **CAMI**: only ~30%-40% of reads are simulated from known taxa while the rest of the reads are from novel taxa, plasmids or simulated evolved strains.
> - {% cite Ye.2019 %} used International Metagenomics and Microbiome Standards Alliance (IMMSA) datasets, wherein the taxa are described better.
>
> When benchmarking different classification tools, several metrics are used to compare their performance:
> 1. **Precision**: proportion of true positive species identified in the sample divided by number of total species identified by the method.
> 2. **Recall**: proportion of true positive species divided by the number of distinct species actually in the sample.
> 3. Precision-recall curve: each point represents the precision and recall scores at a specific abundance threshold, the **area under the precision-recall curve (AUPR)**
> 4. **L2 distance**: representation of abundance profiles → how accurately the abundance of each species or genera in the resulting classification reflects the abundance of each species in the original biological sample (“ground truth”)
>
> When it comes to taxonomic profiling, thus investigating the abundance of specific taxa, the biggest problem is the abundance bias. It is introduced during isolation of DNA (which might work for some organisms better then for others) and by PCR duplicates during PCR amplification.
{: .details}

> <details-title>Profiling tools</details-title>
>
> Profilers, which are tools that investigate relative abundances of taxa within a dataset, fall into three groups depending on their performance:
> 1. Profilers, that correctly predict relative abundances
> 2. Precise profilers (suitable, when many false positives would increase cost and effort in downstream analysis)
> 3. Profilers with high recall (suitable for pathogen detection, when the failure of detecting an organism can have severe negative consequences)
>
> However, some characteristics are common to all profilers:
> - Most profilers only perform well until the family level
> - Drastic decrease in performance between family and genus level, while little change between order and family level
> - Poorer performance of all profilers on CAMI datasets compared to International Metagenomics and Microbiome Standards Alliance (IMMSA)
> - Fidelity of abundance estimates decreases notably when viruses and plasmids were present
> - High numbers of false positive calls at low abundance
> - Taxonomic profilers vs profiles from taxonomic binning:
> Precision and recall of the taxonomic binners were comparable to that of the profilers;
> abundance estimation at higher ranks was more problematic for the binners
{: .details}

Tool | Approach | Available in Galaxy | CAMI challenge | {% cite Ye.2019 %}
--- | --- | --- | --- | ---
mOTUs |  | No | Most memory efficient |
MetaPhlAn | Marker genes | Yes | | Recommended for low computational requirements (< 2 Gb of memory)
DUDes | | No | |
FOCUS |  | No | Fast, most memory efficient |
Bracken | k-mer | Yes | Fast |
Kraken | k-mer | Yes | Fastest; most memory efficient | Good performance metrics; very fast on large numbers of samples; allow custom databases when high amounts of memory (>100 Gb) are available

# Marker gene based approach using MetaPhlAn 

In this tutorial, we follow second approach using **MetaPhlAn** ({% cite truong2015metaphlan2 %}). MetaPhlAn is a computational tool for profiling the composition of microbial communities (Bacteria, Archaea and Eukaryotes) from metagenomic shotgun sequencing data (i.e. not 16S) at species-level. MetaPhlAn 4 relies on ~5.1M unique clade-specific marker genes identified from ~1M microbial genomes (~236,600 references (bacterial, archeal, viral and eukaryotic) and 771,500 metagenomic assembled genomes) spanning 26,970 species-level genome bins.

It allows:
- unambiguous taxonomic assignments;
- accurate estimation of organismal relative abundance;
- species-level resolution for bacteria, archaea, eukaryotes and viruses;
- strain identification and tracking
- orders of magnitude speedups compared to existing methods.
- microbiota strain-level population genomics
- MetaPhlAn clade-abundance estimation

The basic usage of MetaPhlAn consists in the identification of the clades (from phyla to species and strains in particular cases) present in the microbiota obtained from a microbiome sample and their relative abundance.

**MetaPhlAn** in Galaxy can not directly take as input a paired collection but expect 2 collections: 1 with forward data and 1 with reverse. Before launching **MetaPhlAn**, we need to split our input paired collection.

> <hands-on-title>Assign taxonomic labels with MetaPhlAn</hands-on-title>
>
> 1. {% tool [Unzip collection](__UNZIP_COLLECTION__) %} with the following parameters:
>    - {% icon param-collection %} *"Paired input to unzip"*: Input paired collection
>
> 2. {% tool [MetaPhlAn](toolshed.g2.bx.psu.edu/repos/iuc/metaphlan/metaphlan/4.1.1+galaxy4) %} with the following parameters:
>    - In *"Inputs"*
>      - *"Input(s)"*: `Fasta/FastQ file(s) with microbiota reads`
>        - *"Fasta/FastQ file(s) with microbiota reads"*: `Paired-end files`
>          - {% icon param-collection %} *"Forward paired-end Fasta/FastQ file with microbiota reads"*: output of **Unzip collection** with **forward** in the name
>          - {% icon param-collection %} *"Reverse paired-end Fasta/FastQ file with microbiota reads"*: output of **Unzip collection** with **reverse** in the name
>    - In *"Outputs"*:
>        - *"Output for Krona?"*: `Yes`
{: .hands_on}


5 files and a collection are generated by **MetaPhlAn** {% icon tool %}:

- `Predicted taxon relative abundances`: tabular files with the **community profile**

   ```
   #SampleID 	Metaphlan_Analysis 		
   #clade_name 	NCBI_tax_id 	relative_abundance 	additional_species
   k__Bacteria 	2 	100.0 	
   k__Bacteria|p__Bacteroidetes 	2|976 	94.38814 	
   k__Bacteria|p__Proteobacteria 	2|1224 	5.61186 	
   k__Bacteria|p__Bacteroidetes|c__CFGB45935 	2|976| 	94.38814 	
   k__Bacteria|p__Proteobacteria|c__Alphaproteobacteria 	2|1224|28211 	5.61186 	
   k__Bacteria|p__Bacteroidetes|c__CFGB45935|o__OFGB45935 	2|976|| 	94.38814 	
   k__Bacteria|p__Proteobacteria|c__Alphaproteobacteria|o__Rhodobacterales 	2|1224|28211|204455 	5.61186 	
   k__Bacteria|p__Bacteroidetes|c__CFGB45935|o__OFGB45935|f__FGB45935 	2|976||| 	94.38814 	
   k__Bacteria|p__Proteobacteria|c__Alphaproteobacteria|o__Rhodobacterales|f__Rhodobacteraceae 	2|1224|28211|> 204455|    31989 	5.61186 	
   k__Bacteria|p__Bacteroidetes|c__CFGB45935|o__OFGB45935|f__FGB45935|g__GGB56609 	2|976|||| 	94.38814 	
   k__Bacteria|p__Proteobacteria|c__Alphaproteobacteria|o__Rhodobacterales|f__Rhodobacteraceae|g__Phycocomes 	2| 1224|    28211|204455|31989|2873978 	5.61186 	
   k__Bacteria|p__Bacteroidetes|c__CFGB45935|o__OFGB45935|f__FGB45935|g__GGB56609|s__GGB56609_SGB78025 	2| 976||||| 	94.    38814 	
   k__Bacteria|p__Proteobacteria|c__Alphaproteobacteria|o__Rhodobacterales|f__Rhodobacteraceae|g__Phycocomes|     s__Phycocomes_zhengii 	2|1224|28211|204455|31989|2873978|2056810 	5.61186 	
   k__Bacteria|p__Bacteroidetes|c__CFGB45935|o__OFGB45935|f__FGB45935|g__GGB56609|s__GGB56609_SGB78025| t__SGB78025 	2|    976|||||| 	94.38814 	
   k__Bacteria|p__Proteobacteria|c__Alphaproteobacteria|o__Rhodobacterales|f__Rhodobacteraceae|g__Phycocomes|     s__Phycocomes_zhengii|t__SGB31485 	2|1224|28211|204455|31989|2873978|2056810| 	5.61186 	
   ```

   Each line contains 4 columns:
   1. the lineage with different taxonomic levels, from high level taxa (kingdom: `k__`) to more precise taxa
   2. the NCBI taxon ids of the lineage taxonomic level
   3. the relative abundance found for our sample for the lineage
   4. any additional species

   > <question-title></question-title>
   >
   > 1. Which kindgoms have been identified for JC1A?
   > 2. For JP4D?
   > 3. How is the diversity for JP4D compared to **Kraken** results?
   >
   > > <solution-title></solution-title>
   > > 1. All reads have been unclassified so no kindgom identified
   > > 2. Bacteria, no eukaryotes
   > > 3. Diversity is really reduced for JP4D using **MetaPhlAn**, compared to the one identified with **Kraken**
   > {: .solution}
   {: .question}

- `Predicted taxon relative abundances for Krona`: same information as the previous files but formatted for > visualization using **Krona**

   ```
   Column 1	Column 2	Column 3	Column 4	Column 5	Column 6	Column 7	Column 8	Column 9	Column 10
   94.38814 		Bacteria 	Bacteroidetes 	CFGB45935 	OFGB45935 	FGB45935 	GGB56609 	GGB56609 SGB78025 	
   5.61186 		Bacteria 	Proteobacteria 	Alphaproteobacteria 	Rhodobacterales 	Rhodobacteraceae 	Phycocomes 	Phycocomes zhengii 	
   94.38814 		Bacteria 	Bacteroidetes 	CFGB45935 	OFGB45935 	FGB45935 	GGB56609 	GGB56609 SGB78025 	SGB78025
   5.61186 		Bacteria 	Proteobacteria 	Alphaproteobacteria 	Rhodobacterales 	Rhodobacteraceae 	Phycocomes 	Phycocomes zhengii 	SGB31485 
   ```

   Each line represent an identified taxons with 9 columns:
   - Column 1: The percentage of reads assigned the taxon
   - Column 2-9: The taxon description at the different taxonomic levels from Kindgom to more precise taxa

- A **collection** with the same information as in the tabular file but splitted into different files, one per taxonomic level
- `BIOM file` with **community profile** in BIOM format, a common and standard format in microbiomics and used as the input for tools like mothur or Qiime.

- `Bowtie2 output` and `SAM file` with the results of the sequence mapping on the reference database

Let's now run **Krona** to visualize the communities

> <hands-on-title>Generate Krona visualisation</hands-on-title>
> 1. {% tool [Krona pie chart](toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1) %} with the following parameters:
>    - *"Type of input data"*: `tabular`
>    - {% icon param-collection %} *"Input file"*: Krona output of **Metaphlan**
> 
> 2. Inspect the generated file
{: .hands_on}

As pointed before, the community looks a lot less diverse than with **Kraken**. This is probably due to the reference database, which is potentially not complete enough yet to identify all taxons. Or there are too few reads in the input data to cover enough marker genes. Indeed, no taxon has been identified for JC1A, which contains much less reads than JP4D. However, **Kraken** is also known to have high number of false positive. A benchmark dataset or mock community where the community DNA content is known would be required to correctly judge which tool provides the better taxonomic classification.

# Conclusion

In this tutorial, we look how to get the community profile from microbiome data. We apply **Kraken2** or **MetaPhlAn** to assign taxonomic labels to two microbiome sample datasets. We then visualize the results using **Krona**, **Pavian** and **Phinch** to analyze and compare the datasets. Finally, we discuss important facts when it comes to choosing the right tool for taxonomic assignment.å

tools_json:
["Kraken2", "toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.1.3+galaxy1", "Bracken", "toolshed.g2.bx.psu.edu/repos/iuc/bracken/est_abundance/3.1+galaxy0", "Krakentools: Convert kraken report file", "toolshed.g2.bx.psu.edu/repos/iuc/krakentools_kreport2krona/krakentools_kreport2krona/1.2+galaxy2", "Krona pie chart", "toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1+galaxy0", "Pavian", "interactive_tool_pavian", "Kraken-biom", "toolshed.g2.bx.psu.edu/repos/iuc/kraken_biom/kraken_biom/1.2.0+galaxy1", "Phinch Visualisation", "interactive_tool_phinch", "Unzip collection", "__UNZIP_COLLECTION__", "MetaPhlAn", "toolshed.g2.bx.psu.edu/repos/iuc/metaphlan/metaphlan/4.1.1+galaxy4", "toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1"]


topic: microbiome
tutorial: dada-16S
title: Building an amplicon sequence variant (ASV) table from 16S data using DADA2
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: ["metabarcoding", "16S", "microgalaxy"]
content_hash: 7eaf3f22b7dd51efd1021e5508dbc926
last_modified: 2026-05-27T09:01:49.311649
zenodo_link: https://doi.org/10.5281/zenodo.800651
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/dada-16S/tutorial.html

content:
The investigation of environmental microbial communities and microbiomes has been revolutionized by the 
development of high-throughput amplicon sequencing. In amplicon sequencing a particular genetic locus, 
for example the 16S rRNA gene (or a part of it) in bacteria, is amplified from DNA extracted from the community of interest, 
and then sequenced on a next-generation sequencing platform. This technique removes the need to culture 
microbes in order to detect their presence, and cost-effectively provides a deep census of a microbial community.

> <comment-title>Sources</comment-title>
>
> This tutorial is significantly based on [DADA2 Pipeline Tutorial](https://benjjneb.github.io/dada2/tutorial.html) and ["Introduction to dada2"](https://bioconductor.org/packages/devel/bioc/vignettes/dada2/inst/doc/dada2-intro.html)
>
{: .comment}

{% include topics/microbiome/tutorials/mothur-miseq-sop/background_16S.md %}

To process 16S amplicon sequencing data, several tools exist: **QIIME2** ({% cite Bolyen_2019 %}), **Mothur** ({% cite Schloss_2009 %}), **Lotus2** ({% cite _zkurt_2022 %}), and **DADA2** ({% cite Callahan_2016 %}). All these tools can be used in Galaxy. In this tutorial we will use **DADA2**.

> <comment-title>Processing 16S amplicon data in Galaxy using other tools than DADA2</comment-title>
>
> Tutorials are available for the other tools in Galaxy:
> - [QIIME 2 Cancer Microbiome Intervention](https://docs.qiime2.org/jupyterbooks/cancer-microbiome-intervention-tutorial/index.html#)
> - [QIIME 2 Moving Pictures](https://docs.qiime2.org/2024.2/tutorials/moving-pictures-usage)
> - [16S Microbial Analysis with mothur]({% link topics/microbiome/tutorials/mothur-miseq-sop/tutorial.md %})
>
{: .comment}

The output of the dada2 pipeline is a feature table of amplicon sequence variants (an ASV table): A matrix with columns corresponding to samples and rows to ASVs, in which the value of each entry is the number of times that ASV was observed in that sample. This table is analogous to the traditional OTU table, except that one gets the actual sequences of the "true" sequence variants instead of rather anonymous and abstract entities called OTU1, OTU2, etc. 

The process of amplicon sequencing introduces errors into the sequencing data, and these errors 
severely complicate the interpretation of the results. **DADA2** implements a novel algorithm that models the errors introduced during amplicon sequencing, and uses that error model to infer the true sample composition. 

**DADA2** replaces the traditional "OTU-picking" step in amplicon sequencing workflows, producing instead higher-resolution tables of amplicon sequence variants (ASVs).

{% include topics/microbiome/tutorials/mothur-miseq-sop/background_otus.md %}

As seen in the paper introducing DADA2 ({% cite Callahan_2016 %}) and in [further benchmarking](http://benjjneb.github.io/dada2/SotA.html), the **DADA2** method is more sensitive and specific than traditional OTU methods: DADA2 detects real biological variation missed by OTU methods while outputting fewer spurious sequences.

To illustrate the 16S amplicon data processing using **DADA2**, we will use dataset generated by the Schloss lab in their
[their mothur MiSeq SOP](https://www.mothur.org/wiki/MiSeq_SOP): FASTQ files generated by 2x250 Illumina Miseq amplicon sequencing of the V4 region of the 16S rRNA gene from gut samples collected longitudinally from a mouse post-weaning. They [describe the experiment](https://www.mothur.org/wiki/MiSeq_SOP#Logistics) as follows:

> *The Schloss lab is interested in understanding the effect of normal variation in the gut microbiome on host health. To that end,
> we collected fresh feces from mice on a daily basis for 365 days post weaning. During the first 150 days post weaning (dpw),
> nothing was done to our mice except allow them to eat, get fat, and be merry. We were curious whether the rapid change in
> weight observed during the first 10 dpw affected the stability microbiome compared to the microbiome observed between days
> 140 and 150.*
> 
{: .quote}

To speed up analysis for this tutorial, we will use only a subset of this data. We will look at a single mouse at 20 different time points (10 early, 10 late).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare Galaxy and data

Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>History creation</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

The starting point for the **DADA2** pipeline is a set of demultiplexed FASTQ files corresponding to the samples in your amplicon sequencing study. That is, **DADA2** expects there to be an individual FASTQ file for each sample (or two FASTQ files, one forward and one reverse, for each sample). Demultiplexing is often performed at the sequencing center, but if that has not been done there are a variety of tools do accomplish this.

In addition, **DADA2** expects that there are no non-biological bases in the sequencing data. This may require pre-processing with outside software if, as is relatively common, your PCR primers were included in the sequenced amplicon region.

For this tutorial, we have 20 pairs of files. For example, the following pair of files:

- `F3D0_S188_L001_R1_001.fastq`
- `F3D0_S188_L001_R2_001.fastq`

The first part of the file name indicates the sample: `F3D0` here signifies that this sample was obtained from Female 3 on Day 0.

The rest of the file name is identical, except for `_R1` and `_R2`, this is used to indicate the forward and reverse reads respectively.

> <hands-on-title>Import datasets</hands-on-title>
>
> 1. Import the following samples via link from [Zenodo]({{ page.zenodo_link }}) or Galaxy shared data libraries:
>
>    > <solution-title>List of Zenodo URLs</solution-title>
>    > ```
>    > https://zenodo.org/record/800651/files/F3D0_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D0_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D141_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D141_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D142_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D142_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D143_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D143_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D144_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D144_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D145_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D145_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D146_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D146_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D147_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D147_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D148_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D148_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D149_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D149_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D150_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D150_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D1_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D1_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D2_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D2_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D3_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D3_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D5_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D5_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D6_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D6_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D7_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D7_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D8_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D8_R2.fastq
>    > https://zenodo.org/record/800651/files/F3D9_R1.fastq
>    > https://zenodo.org/record/800651/files/F3D9_R2.fastq
>    > https://zenodo.org/record/800651/files/Mock_R1.fastq
>    > https://zenodo.org/record/800651/files/Mock_R2.fastq
>    > ```
>    {: .solution }
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

40 files are a lot of files to manage. Luckily Galaxy can make life a bit easier by allowing us to create **dataset collections**. This enables us to easily run tools on multiple datasets at once.

Since we have **paired-end** data, each sample consist of two separate FASTQ files:
- one containing the forward reads, and 
- one containing the reverse reads.

We can recognize the pairing from the file names, which will differ only by `_R1` or `_R2` in the filename. We can tell Galaxy about this paired naming convention, so that our tools will know which files belong together. We do this by building a **List of Dataset Pairs**

> <hands-on-title>Organizing our data into a sorted paired collection</hands-on-title>
>
> 1. Create a paired collection named `Raw Reads`, rename your pairs with the sample name
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
> 2. {% tool [Sort collection](__SORTLIST__) %} with the following parameters:
>    - {% icon param-collection %} *"Input Collection"*: `Raw Reads`
>
> 3. Rename output collection to `Sorted Raw Reads`
>
>    {% snippet faqs/galaxy/collections_rename.md %}
>
{: .hands_on}

We can now start processing the reads using **DADA2**.

{% include _includes/cyoa-choices.html option1="Step-by-step" option2="Workflow" default="Step-by-step" text="Do you want to go step-by-step or using a workflow?" disambiguation="workflow"%}

# Processing data using DADA2

Once demultiplexed FASTQ files without non-biological nucleotides are in hand, the **DADA2** pipeline proceeds as follows:

1. Prepare reads
   1. Inspect read quality profiles
   2. Filter and trim
   3. Learn the Error Rates
   4. Infer sample composition
   5. Merge forward/reverse reads

3. Construct sequence table
   1. Make sequence table
   2. Remove chimeras
  
4. Assign taxonomy
5. Evaluate accuracy

<div class="Workflow" markdown="1">

{% snippet faqs/galaxy/workflows_run_wfh.md title="dada2/main" wfhub_id="790" version="2"%}

> <hands-on-title>Define DADA2 workflow parameters</hands-on-title>
>
> 1. Select following parameters:
>    - {% icon param-collection %} *"Paired short read data"*: `Raw Reads`
>    - *"Read length forward read"*: `240`
>    - *"Read length reverse read"*: `160`
>    - *"Pool samples"*: `process samples individually`
>    - *"Cached reference database"*: `Silva version 132`
>
> 2. Run the workflow
{: .hands_on}

The workflow generates several outputs that we will inspect them given the step they correspond to.

</div>

# Prepare reads

The first step is to check and improve the quality of data.

## Inspect read quality profiles

We start by visualizing the quality profiles of the forward reads:

<div class="Step-by-step" markdown="1">

> <hands-on-title>Inspect read quality profiles</hands-on-title>
>
> 1. {% tool [dada2: plotQualityProfile](toolshed.g2.bx.psu.edu/repos/iuc/dada2_plotqualityprofile/dada2_plotQualityProfile/1.34.0+galaxy0) %} with the following parameters:
>    - *"Processing mode"*: `Joint`
>        - *"Paired reads"*: `paired - in a data set pair`
>            - {% icon param-collection %} *"Paired short read data"*: `Sorted Raw Reads`
>        - *"Aggregate data"*: `No`
>    - *"sample number"*: `10000000`
{: .hands_on}
</div>

**dada2: plotQualityProfile** generates 2 files with quality profiles per sample: 1 PDF for forward reads and 1 PDF for reverse reads.

> <hands-on-title>Inspect read quality profiles</hands-on-title>
>
> 1. Inspect **plotQualityProfile** forward read output
{: .hands_on}

![Quality Profiles for the forward reads of all samples with read length on the X-axis and Quality score on the Y-axis](./images/plotQualityProfile_forward.png)

In gray-scale is a heatmap of the frequency of each quality score at each base position. The mean quality score at each position is shown by the green line, and the quartiles of the quality score distribution by the orange lines. The red line shows the scaled proportion of reads that extend to at least that position (this is more useful for other sequencing technologies, as Illumina reads are typically all the same length, hence the flat red line).

> <question-title></question-title>
>
> 1. Are the forward reads good quality?
> 2. We generally advise trimming the last few nucleotides to avoid less well-controlled errors that can arise there. What would you recommend as threshold?
> 3. Are the reverse reads good quality?
> 4. What would you recommend as threshold for trimming?
>
> > <solution-title></solution-title>
> >
> > 1. The forward reads are good quality
> > 2. These quality profiles do not suggest that any additional trimming is needed. We will truncate the forward reads at position 240 (trimming the last 10 nucleotides).
> > 3. The reverse reads are of significantly worse quality, especially at the end, which is common in Illumina sequencing. This isn’t too worrisome, as DADA2 incorporates quality information into its error model which makes the algorithm robust to lower quality sequence, but trimming as the average qualities crash will improve the algorithm’s sensitivity to rare sequence variants.
> > 
> >    ![Quality Profiles for the reverse reads of all samples with read length on the X-axis and Quality score on the Y-axis](./images/plotQualityProfile_reverse.png)
> >    
> > 4. Based on the profiles, we will truncate the reverse reads at position 160 where the quality distribution crashes.
> > 
> {: .solution}
>
{: .question}

> <comment-title>Considerations for your own data</comment-title>
>
> Your reads must still overlap after truncation in order to merge them later! The tutorial is using 2x250 V4 (expected size, 270 bp–387 bp) sequence data, so the forward and reverse reads almost completely overlap and our trimming can be completely guided by the quality scores. If you are using a less-overlapping primer set, like V1-V2 or V3-V4, your truncated length must be large enough to maintain $$ 20 + biological.length.variation $$ nucleotides of overlap between them.
>
{: .comment}

## Filter and trim

<div class="Step-by-step" markdown="1">

Let's now trim and filter the reads:

> <hands-on-title>Filter and trim</hands-on-title>
>
> 1. {% tool [dada2: filterAndTrim](toolshed.g2.bx.psu.edu/repos/iuc/dada2_filterandtrim/dada2_filterAndTrim/1.34.0+galaxy0) %} with the following parameters:
>    - *"Paired reads"*: `paired - in a data set pair`
>        - {% icon param-collection %} *"Paired short read data"*: `Sorted Raw Reads`
>    - In *"Trimming parameters"*:
>        - *"Truncate read length"*: `240`
>    - In *"Filtering parameters"*:
>        - *"Remove reads by number expected errors"*: `2`
>
>          This parameter sets the maximum number of "expected errors" allowed in a read, which is [a better filter than simply averaging quality scores](https://academic.oup.com/bioinformatics/article/31/21/3476/194979).
>
>    - *"Separate filters and trimmers for reverse reads"*: `Yes`
>        - In *"Trimming parameters"*:
>            - *"Truncate read length"*: `160`
>        - In *"Filtering parameters"*:
>            - *"Remove reads by number expected errors"*: `2`
>
{: .hands_on}

> <comment-title>Speeding up downstream computation</comment-title>
> 
>  The standard filtering parameters are starting points, not set in stone. If you want to speed up downstream computation, consider tightening the *"Remove reads by number expected errors"* value. If too few reads are passing the filter, consider relaxing the value, perhaps especially on the reverse reads (e.g. *"Remove reads by number expected errors"* to `2` for forward and `5` for reverse), and reducing the *"Truncate read length"* to remove low quality tails. Remember though, when choosing *"Truncate read length"* for paired-end reads you must maintain overlap after truncation in order to merge them later. 
> 
{: .comment}

> <comment-title>ITS data</comment-title>
> 
> For ITS sequencing, it is usually undesirable to truncate reads to a fixed length due to the large length variation at that locus. That is OK, just leave out  *"Truncate read length"*. See the [DADA2 ITS workflow](https://benjjneb.github.io/dada2/ITS_workflow.html) for more information 
> 
{: .comment}

</div>

**dada2: filterAndTrim** generates 2 outputs:
- 1 paired-collection named `Paired reads` with the trimmed and filtered reads
- 1 collection with statistics for each sample

   > <question-title></question-title>
   >
   > 1. How many reads were in F3D0 sample before Filter and Trim?
   > 2. How many reads are in F3D0 sample after Filter and Trim?
   >
   > > <solution-title></solution-title>
   > >
   > > In F3D0 element in the collection, we see:
   > > 1. `reads.in`: 7793
   > > 2. `reads.out`: 7113
   > >
   > {: .solution}
   >
   {: .question}

To better see the impact of filtering and trimming, we can inspect the read quality profiles and compare to the profiles before.

<div class="Step-by-step" markdown="1">

> <hands-on-title>Inspect read quality profiles after Filter And Trim</hands-on-title>
>
> 1. {% tool [dada2: plotQualityProfile](toolshed.g2.bx.psu.edu/repos/iuc/dada2_plotqualityprofile/dada2_plotQualityProfile/1.34.0+galaxy0) %} with the following parameters:
>    - *"Processing mode"*: `Joint`
>        - *"Paired reads"*: `paired - in a data set pair`
>            - {% icon param-collection %} *"Paired short read data"*: `Paired reads` output of **dada2: filterAndTrim**
>        - *"Aggregate data"*: `No`
>    - *"sample number"*: `10000000`
>
{: .hands_on}

</div>

> <question-title></question-title>
>
> Looking at outputs of **plotQualityProfile** for reverse reads before and after **dada2: filterAndTrim**
> 
> {% snippet faqs/galaxy/features_scratchbook.md %}
>
> ![Quality Profiles for forward reads of all samples with read length on the X-axis and Quality score on the Y-axis](./images/plotQualityProfile_reverse.png "Quality profiles for reverse reads before trimming and filtering")
>
> ![Quality Profiles for reverse reads of all samples with read length on the X-axis and Quality score on the Y-axis](./images/plotQualityProfile_reverse_after_filter_trim.png "Quality profiles for reverse reads after trimming and filtering")
>
> How did trimming and filtering change the quality profiles for reverse reads?
>
> > <solution-title></solution-title>
> >
> > The reads have been trimmed making the profiles looking better. 
> >
> {: .solution}
>
{: .question}

## Learn the Error Rates

The DADA2 algorithm makes use of a parametric error model (`err`) and every amplicon dataset has a different set of error rates. The **dada2: learnErrors** method learns this error model from the data, by alternating estimation of the error rates and inference of sample composition until they converge on a jointly consistent solution. As in many machine-learning problems, the algorithm must begin with an initial guess, for which the maximum possible error rates in this data are used (the error rates if only the most abundant sequence is correct and all the rest are errors).

<div class="Step-by-step" markdown="1">

**dada2: learnErrors** tool expects as input one collection and not a paired collection. We then need to split to paired collection into 2 collections: one with forward reads and one with reverse reads.

> <hands-on-title> Unzip paired collection </hands-on-title>
>
> 1. {% tool [Unzip collection](__UNZIP_COLLECTION__) %} with the following parameters:
>    - {% icon param-file %} *"Paired input to unzip"*: `Paired reads` output of **dada2: filterAndTrim**
>
> 2. Tag the `(forward)` collection with `#forward`
>
>    {% snippet faqs/galaxy/collections_add_tag.md tag="#forward" %}
>
> 3. Tag the `(reverse)` collection with `#reverse`
{: .hands_on}

Let's now run **dada2: learnErrors** on both collections.

> <hands-on-title> Learn the Error Rate </hands-on-title>
>
> 1. {% tool [dada2: learnErrors](toolshed.g2.bx.psu.edu/repos/iuc/dada2_learnerrors/dada2_learnErrors/1.34.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Short read data"*: `forward` output of **Unzip collection**
>   
> 3. {% tool [dada2: learnErrors](toolshed.g2.bx.psu.edu/repos/iuc/dada2_learnerrors/dada2_learnErrors/1.34.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Short read data"*: `reverse` output of **Unzip collection**
>   
{: .hands_on}

</div>

![Error rates for each possible transition as a function of the associated quality score, the final estimated error rates, the initial input rates, and the expected error rates under the nominal definition of quality scores](./images/learnErrors_forward.png "Error rates for each possible transition for the forward reads")

The error rates for each possible transition (A->C, A->G, ...) are shown as a function of the associated quality score, the final estimated error rates (if they exist), the initial input rates, and the expected error rates under the nominal definition of quality scores. Points are the observed error rates for each consensus quality score. The black line shows the estimated error rates after convergence of the machine-learning algorithm. The red line shows the error rates expected under the nominal definition of the Q-score.

> <question-title></question-title>
>
> 1. Are the estimated error rates (black line) in a good fit to the observed rates (points) for the forward reads?
> 2. What about the reverse reads?
>
>    ![Error rates for each possible transition as a function of the associated quality score, the final estimated error rates, the initial input rates, and the expected error rates under the nominal definition of quality scores](./images/learnErrors_reverse.png "Error rates for each possible transition for the reverse reads")
>
> > <solution-title></solution-title>
> >
> > 1. The estimated error rates (black line) are a good fit to the observed rates (points), and the error rates drop with increased quality as expected. Everything looks reasonable and we proceed with confidence.
> > 2. Same
> >
> {: .solution}
>
{: .question}
 

## Infer Sample Composition 

In this step, the core sample inference algorithm ({% cite Callahan_2016 %}) is applied to the filtered and trimmed sequence data.

<div class="Step-by-step" markdown="1">

> <hands-on-title> Sample Inference </hands-on-title>
>
> 1. {% tool [dada2: dada](toolshed.g2.bx.psu.edu/repos/iuc/dada2_dada/dada2_dada/1.34.0+galaxy0) %} with the following parameters:
>    - *"Process samples in batches"*: `no`
>
>      > <comment-title>Process samples in batches or not</comment-title>
>      >
>      > Choosing `yes` for *"Process samples in batches"* gives identical results as `no` if samples are not pooled.
>      >
>      > - `no`: a single Galaxy job is started where the samples are processed sequentially
>      > - `yes`: a separate Galaxy job is started for each sample (given compute resources, this can be much faster)
>      >
>      > If `yes`, the sorting of the collection is not needed.
>      >
>      {: .comment}
>
>        - {% icon param-file %} *"Reads"*: output of **Unzip collection** with `#forward` tag
>        - *"Pool samples"*: `process samples individually`
>
>           > <comment-title>Sample pooling</comment-title>
>           >
>           > By default, the **dada2: dada** function processes each sample independently. However, pooling information across samples can increase sensitivity to sequence variants that may be present at very low frequencies in multiple samples. Two types of pooling are possible:
>           >
>           > - `pool sample`: standard pooled processing, in which all samples are pooled together for sample inference
>           > - `pseudo pooling between individually processed samples`: pseudo-pooling, in which samples are processed independently after sharing information between samples, approximating pooled sample inference in linear time.
>           >
>           {: .comment}
>
>    - {% icon param-file %} *"Error rates"*: output of **dada2: learnErrors** with `#forward` tag
>
> 3. {% tool [dada2: dada](toolshed.g2.bx.psu.edu/repos/iuc/dada2_dada/dada2_dada/1.34.0+galaxy0) %} with the following parameters:
>    - *"Process samples in batches"*: `no`
>        - {% icon param-file %} *"Reads"*: output of **Unzip collection** with `#reverse` tag
>        - *"Pool samples"*: `process samples individually`
>    - {% icon param-file %} *"Error rates"*: output of **dada2: learnErrors** with `#reverse` tag
{: .hands_on}

</div>

We can look at the number of true sequence variants from unique sequences have been inferred by **DADA2**.

> <hands-on-title> Inspect number of true sequence variants from unique sequences </hands-on-title>
>
> 1. Click one output collection of **dada2: dada**
> 2. Click on {% icon details %} *Show Details*
> 2. Expand *Tool Standard Output*
{: .hands_on}

> <question-title></question-title>
>
> 1. How many unique sequences are in forward reads for sample F3D0?
> 2. How many true sequence variants from unique sequences have been inferred by DADA2 for this dataset?
>
> > <solution-title></solution-title>
> >
> > 1. 7113 reads in 1979 unique sequences
> > 2. 128 sequence variants were inferred from 1979 input unique sequences.
> >
> {: .solution}
>
{: .question}

## Merge paired reads

We now merge the forward and reverse reads together to obtain the full denoised sequences. Merging is performed by aligning the denoised forward reads with the reverse-complement of the corresponding denoised reverse reads, and then constructing the merged "contig" sequences. By default, merged sequences are only output if the forward and reverse reads overlap by at least 12 bases, and are identical to each other in the overlap region (but these conditions can be changed via tool parameters).

<div class="Step-by-step" markdown="1">

> <hands-on-title> Merge paired reads </hands-on-title>
>
> 1. {% tool [dada2: mergePairs](toolshed.g2.bx.psu.edu/repos/iuc/dada2_mergepairs/dada2_mergePairs/1.34.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Dada results for forward reads"*: output of **dada2: dada** with `#forward` tag
>    - {% icon param-file %} *"Forward reads"*: output of **Unzip collection** with `#forward` tag
>    - {% icon param-file %} *"Dada results for reverse reads"*:  output of **dada2: dada** with `#reverse` tag
>    - {% icon param-file %} *"Reverse reads"*: output of **Unzip collection** with `#reverse` tag
>    - *"Concatenated rather than merge"*: `No`
>
>      > <comment-title>Non-overlapping reads</comment-title>
>      > 
>      > Non-overlapping reads are supported, but not recommended. In this case, *"Concatenated rather than merge"* should be `Yes`.
>      > 
>      {: .comment}
>
>    - *"Output detailed table"*: `Yes`
{: .hands_on}

</div>

Let's inspect the merged data, more specifically the table in `details` collection

> <question-title></question-title>
>
> 1. How many reads have been merged?
> 2. What are the different columns in the detail table for F3D0?
>
> > <solution-title></solution-title>
> >
> > 1. The detail table for F3D0 has 108 lines including a header. So 107 reads (over 128) have been merged
> > 2. The table contains:
> >    1. merged sequence,
> >    2. its abundance,
> >    3. the indices of the forward sequence variants that were merged
> >    4. the indices of the reverse sequence variants that were merged
> >    5. number of match in the alignment between forward and reverse
> >    6. number of mismatch in the alignment between forward and reverse
> >    7. number of indels in the alignment between forward and reverse
> >    8. prefer
> >    9. status to accept
> >   
> >    Paired reads that did not exactly overlap were removed by mergePairs, further reducing spurious output.
> >
> {: .solution}
>
{: .question}

> <comment-title>Not enough reads merged</comment-title>
> 
> Most of the reads should successfully merge. If that is not the case upstream parameters may need to be revisited: Did you trim away the overlap between your reads? 
> 
{: .comment}

# Make sequence table

In this step, we construct an amplicon sequence variant table (ASV) table, a higher-resolution version of the OTU table produced by traditional methods.

<div class="Step-by-step" markdown="1">

> <hands-on-title> Make sequence table </hands-on-title>
>
> 1. {% tool [dada2: makeSequenceTable](toolshed.g2.bx.psu.edu/repos/iuc/dada2_makesequencetable/dada2_makeSequenceTable/1.34.0+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"samples"*: output of **dada2: mergePairs**
>    - *"Length filter method"*: `No filter`
>
{: .hands_on}

</div>

**dada2: makeSequenceTable** generates 2 outputs:
1. A sequence table: a matrix with rows corresponding to (and named by) the samples, and columns corresponding to (and named by) the sequence variants. 

   > <question-title></question-title>
   >
   > How many ASVs have been identified?
   >
   > > <solution-title></solution-title>
   > >
   > > The file contains 294 lines, including an header. So 293 ASVs have been identified.
   > >
   > {: .solution}
   >
   {: .question}

2. An image with sequence length distribution

   ![Histogram representing sequence length distribution](./images/sequence_length_distribution.png)

   > <question-title></question-title>
   >
   > 1. What is the range of merged sequences?
   > 2. Does the range make sense given the type of amplicon?
   >
   > > <solution-title></solution-title>
   > >
   > > 1. The lengths of merged sequences are between 251 and 255bp
   > > 2. The range falls within the expected range for this V4 amplicon.
   > >
   > {: .solution}
   >
   {: .question}

   > <comment-title>Sequences too long or too short</comment-title>
   > 
   > Sequences that are much longer or shorter than expected may be the result of non-specific priming. Non-target-length sequences can be removed directly from the sequence table. This is analogous to "cutting a band" in-silico to get amplicons of the targeted length.
   > 
   {: .comment}

## Remove chimeras

The core DADA method corrects substitution and indel errors, but chimeras remain. Fortunately, the accuracy of sequence variants after denoising makes identifying chimeric ASVs simpler than when dealing with fuzzy OTUs. Chimeric sequences are identified if they can be exactly reconstructed by combining a left-segment and a right-segment from two more abundant "parent" sequences.

<div class="Step-by-step" markdown="1">

> <hands-on-title> Remove chimeras </hands-on-title>
>
> 1. {% tool [dada2: removeBimeraDenovo](toolshed.g2.bx.psu.edu/repos/iuc/dada2_removebimeradenovo/dada2_removeBimeraDenovo/1.34.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"sequence table"*: output of **dada2: makeSequenceTable**
>
{: .hands_on}

</div>

> <question-title></question-title>
>
> How many non chimeric reads have been found?
>
> > <solution-title></solution-title>
> >
> > remaining nonchimeric:  96.40374 % (visible when expanding the dataset in history)
> >
> {: .solution}
>
{: .question}

The frequency of chimeric sequences varies substantially from dataset to dataset, and depends on on factors including experimental procedures and sample complexity. Here chimeras make up about 21% of the merged sequence variants, but when we account for the abundances of those variants we see they account for only about 4% of the merged sequence reads.

> <comment-title>Sequences too long or too short</comment-title>
> 
> Most of the **reads** should remain after chimera removal (it is not uncommon for a majority of **sequence variants** to be removed though). If most of the reads were removed as chimeric, upstream processing may need to be revisited. In almost all cases this is caused by primer sequences with ambiguous nucleotides that were not removed prior to beginning the DADA2 pipeline.
> 
{: .comment}

## Track reads through the pipeline

As a final check of our progress, we’ll look at the number of reads that made it through each step in the pipeline.

<div class="Step-by-step" markdown="1">

> <hands-on-title> Track reads through the pipeline </hands-on-title>
>
> 1. {% tool [dada2: sequence counts](toolshed.g2.bx.psu.edu/repos/iuc/dada2_seqcounts/dada2_seqCounts/1.34.0+galaxy0) %} with the following parameters:
>    - In *"data sets"*:
>        - {% icon param-repeat %} *"Insert data sets"*
>            - {% icon param-collection %} *"Dataset(s)"*: output of **dada2: filterAndTrim**
>            - *"name"*: `FiltTrim`
>        - {% icon param-repeat %} *"Insert data sets"*
>            - {% icon param-collection %} *"Dataset(s)"*: output of **dada2: dada** with `#forward` tag
>            - *"name"*: `Dada Forward`
>        - {% icon param-repeat %} *"Insert data sets"*
>            - {% icon param-collection %} *"Dataset(s)"*: output of **dada2: dada** with `#reverse` tag
>            - *"name"*: `Dada Reverse`
>        - {% icon param-repeat %} *"Insert data sets"*
>            - {% icon param-collection %} *"Dataset(s)"*: output of **dada2: mergePairs**
>            - *"name"*: `MakePairs`
>        - {% icon param-repeat %} *"Insert data sets"*
>            - {% icon param-file %} *"Dataset(s)"*: output of **dada2: makeSequenceTable**
>            - *"name"*: `SeqTab`
>        - {% icon param-repeat %} *"Insert data sets"*
>            - {% icon param-file %} *"Dataset(s)"*: output of **dada2: removeBimeraDenovo**
>            - *"name"*: `Bimera`
>
{: .hands_on}

</div>

> <question-title></question-title>
>
> 1. Have we kept the majority of the raw reads for F3D143?
> 2. Is there any exceptionally large drop associated with any single step for any sample?
>
> > <solution-title></solution-title>
> >
> > 1. Yes
> > 2. No
> >
> {: .solution}
>
{: .question}

> <comment-title>Sequences too long or too short</comment-title>
> 
> This is a great place to do a last sanity check. Outside of filtering, there should no step in which a majority of reads are lost. If a majority of reads failed to merge, you may need to revisit the truncLen parameter used in the filtering step and make sure that the truncated reads span your amplicon. If a majority of reads were removed as chimeric, you may need to revisit the removal of primers, as the ambiguous nucleotides in unremoved primers interfere with chimera identification.
> 
{: .comment}

# Assign taxonomy

It is common at this point, especially in 16S/18S/ITS amplicon sequencing, to assign taxonomy to the sequence variants. The DADA2 package provides a native implementation of the naive Bayesian classifier method ({% cite Wang_2007 %}) for this purpose. The **dada2: assignTaxonomy** function takes as input a set of sequences to be classified and a training set of reference sequences with known taxonomy, and outputs taxonomic assignments with at least `minBoot` bootstrap confidence.

DADA2 maintains [formatted training fastas for the RDP training set, GreenGenes clustered at 97% identity, and the Silva reference database](https://benjjneb.github.io/dada2/training.html), and additional trainings fastas suitable for protists and certain specific environments have been contributed. For fungal taxonomy, the General Fasta release files from the [UNITE ITS database](https://unite.ut.ee/repository.php) can be used as is. 
<div class="Step-by-step" markdown="1">

> <hands-on-title> Assign taxonomy </hands-on-title>
>
> 1. {% tool [dada2: assignTaxonomy and addSpecies](toolshed.g2.bx.psu.edu/repos/iuc/dada2_assigntaxonomyaddspecies/dada2_assignTaxonomyAddspecies/1.34.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"sequences to be assigned"*: output of **dada2: removeBimeraDenovo**
>    - *"Select a reference dataset your history or use a built-in?"*: `Use a built-in reference`
>        - *"Select reference data set"*: `Silva version 132`
>    - *"Add genus-species binomials to the taxonomic table"*: `Yes`
>        - *"Select a reference dataset your history or use a built-in?"*: `Use a built-in reference`
>           - *"Select reference data set"*: `Silva version 132`
>       - *"reporting options"*: `only unambiguous identifications`
>
>       > <comment-title>Species level assignments based on **exact matching**</comment-title>
>       > 
>       > The method of DADA2 make [species level assignments is based on **exact matching**](https://benjjneb.github.io/dada2/assign.html#species-assignment) between ASVs and sequenced reference strains. Recent analysis suggests that exact matching (or 100% identity) is the only appropriate way to assign species to 16S gene fragments ({% cite edgar2018updating %}). Currently, [species-assignment training fastas are available for the Silva and RDP 16S databases](https://benjjneb.github.io/dada2/training.html).
>       > 
>       {: .comment}
>
{: .hands_on}

</div>

**dada2: assignTaxonomy and addSpecies** generates a table with rows being the ASVs and columns the taxonomic levels. Let's inspect the taxonomic assignments.

> <question-title></question-title>
>
> 1. Which kingdom have been identified?
> 2. Which phyla have been identified? And how many ASVs per phylum?
> 3. How many ASVs have species assignments?
>
> > <solution-title></solution-title>
> >
> > 1. Only bacteria
> >    
> >    > <comment-title>Wrong kingdom</comment-title>
> >    > 
> >    > If your reads do not seem to be appropriately assigned, for example lots of your bacterial 16S sequences are being assigned as `Eukaryota`, your reads may be in the opposite orientation as the reference database. Tell DADA2 to try the reverse-complement orientation (*"Try reverse complement"*) with **dada2: assignTaxonomy and addSpecies** and see if this fixes the assignments.
> >    > 
> >    {: .comment}
> > 
> > 2. To get the identified phyla and the number of associated ASVs, we run {% tool [ Group data by a column](Grouping1) %} with the following parameters:
> >    - {% icon param-file %} *"Select data"*: output of **dada2: assignTaxonomy and addSpecies**
> >    - *"Group by column"*:`Column 3`
> >    - {% icon param-repeat %} *Insert Operation*
> >         - *"Type"*: `Count`
> >         - *"On column"*: `Column 3`
> >
> >     Phyla | Number of ASVs
> >     --- | ---
> >     Actinobacteria | 6
> >     Bacteroidetes | 20
> >     Cyanobacteria | 3
> >     Deinococcus-Thermus | 1
> >     Epsilonbacteraeota | 1
> >     Firmicutes | 185
> >     Patescibacteria | 2
> >     Phylum | 1
> >     Proteobacteria | 7
> >     Tenericutes | 6
> >     Verrucomicrobia | 1 
> >
> >     Unsurprisingly, the Bacteroidetes and Firmicutes are well represented among the most abundant taxa in these fecal samples.
> >
> > 3. 16/232 species assignments were made, both because it is often not possible to make unambiguous species assignments from subsegments of the 16S gene, and because there is surprisingly little coverage of the indigenous mouse gut microbiota in reference databases.
> >
> {: .solution}
{: .question}

# Exploration of ASVs with phyloseq

**phyloseq** ({% cite McMurdie_2013 %}) is a powerful framework for further analysis of microbiome data. It imports, stores, analyzes, and graphically displays complex phylogenetic sequencing data that has already been clustered into Operational Taxonomic Units (OTUs) or ASV, especially when there is associated sample data, phylogenetic tree, and/or taxonomic assignment of the OTUs/ASVs. 

## Prepare metadata table and launch phyloseq

To use phyloseq properly, the best is to provide it a metadata table. Here, we will also uses the small amount of metadata we have: the samples are named by the gender (G), mouse subject number (X) and the day post-weaning (Y) it was sampled (eg. GXDY).

We will construct a simple sample table from the information encoded in the filenames. Usually this step would instead involve reading the sample data in from a file. Here we will:

1. Extract element identifiers of a collection
2. Add information about gender, mouse subject number and the day post-weaning using the extracted element identifiers
4. Remove last line
5. Add extra column `When` to indicate if `Early` (days < 100) or `Late` (day > 100)
3. Add a header

> <hands-on-title>Prepare metadata table</hands-on-title>
> 1. {% tool [Extract element identifiers](toolshed.g2.bx.psu.edu/repos/iuc/collection_element_identifiers/collection_element_identifiers/0.0.2) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: output of **dada2: mergePairs**
>
>    We will now extract from the names the factors:
>
> 2. {% tool [Replace Text in entire line](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.5+galaxy2) %}
>      - {% icon param-file %} *"File to process"*: output of **Extract element identifiers** {% icon tool %}
>      - In *"Replacement"*:
>         - In *"1: Replacement"*
>            - *"Find pattern"*: `(.)(.)D(.*)`
>            - *"Replace with"*: `\1\2D\3\t\1\t\2\t\3`
>
>    The given pattern `..D.` will be applied to the identifiers, where `.` will match any single character and `.*` any number of any character. The additional parentheses will record the matched character such that it can be used in the replacement (e.g. `\1` will refer to the content that matched the pattern in the 1st pair of parentheses)
>     This step creates 3 additional columns with the gender, the mouse subject number and the day post-weaning
>
> 3. Change the datatype to `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 5. {% tool [Select first lines from a dataset (head)](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_head_tool/9.5+galaxy2) %} to remove the last line
>    - {% icon param-file %} *"File to select"*: output of **Replace Text**
>    - *"Operation"*: `Remove last lines`
>    - *"Number of lines"*: `1`
>
> 6. {% tool [Compute](toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1) %} on rows with the following parameters:
>    - {% icon param-file %} *"Input file"*: output of **Select first lines**
>    - *"Input has a header line with column names?"*: `No`
>    - In *"Expressions"*:
>      - *"Add expression"*: `bool(c4>100)`
>      - *"Mode of the operation?"*: `Append`
> 
> 7. {% tool [Replace Text in a specific column](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1) %} on rows with the following parameters:
>    - {% icon param-file %} *"File to process"*: output of **Compute**
>    - In *"Replacement"*:
>      - In *"1: Replacement"*
>        - *"in column"*: `Column: 5`
>        - *"Find pattern"*: `True`
>        - *"Replace with"*: `Late`
>      - {% icon param-repeat %} *"Insert Replacement"*
>      - In *"2: Replacement"*
>        - *"in column"*: `Column: 5`
>        - *"Find pattern"*: `False`
>        - *"Replace with"*: `Early`
>      - {% icon param-repeat %} *"Insert Replacement"*
>      - In *"3: Replacement"*
>        - *"in column"*: `Column: 2`
>        - *"Find pattern"*: `F`
>        - *"Replace with"*: `Female`
>      - {% icon param-repeat %} *"Insert Replacement"*
>      - In *"4: Replacement"*
>        - *"in column"*: `Column: 2`
>        - *"Find pattern"*: `M`
>        - *"Replace with"*: `Male`
>
> 4. Create a new file (`header`) from the following (header line of the metadata table)
>
>    ```text
>    Sample	Gender	Mouse	Day	When
>    ```
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md name="header" format="tabular" %}
>
> 5. {% tool [Concatenate multiple datasets or collections](cat1) %} to add this header line to the **Replace Text** output:
>    - {% icon param-file %} *"Concatenate Dataset"*: the `header` dataset
>    - *"Dataset"*
>       - {% icon param-repeat %} *"Insert Dataset"*
>         - {% icon param-file %} *"select"*: output of **Replace Text**
>
> 6. Rename the output to `Metadata table`
{: .hands_on}

We have now a metadata table with five columns:
1. Sample
2. Gender
3. Mouse subject
4. Day post-weaning 
5. When

We now construct a phyloseq object directly with it and the DADA2 outputs and launch Shiny-phyloseq ({% cite mcmurdie2015shiny %}) to explore the phyloseq object.

> <hands-on-title> Create phyloseq object and launch phyloseq exploration </hands-on-title>
>
> 1. {% tool [Create phyloseq object from dada2](toolshed.g2.bx.psu.edu/repos/iuc/phyloseq_from_dada2/phyloseq_from_dada2/1.50.0+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Sequence table"*: output of **dada2: removeBimeraDenovo**
>    - {% icon param-file %} *"Taxonomy table"*: output of **dada2: assignTaxonomy and addSpecies**
>    - {% icon param-file %} *"Sample table"*: `Metadata table`
>
> 2. {% tool [Phyloseq](interactive_tool_phyloseq) %} with the following parameters:
>    - {% icon param-file %} *"Phyloseq R object"*: output of **Create phyloseq object from dada2**
>
>    {% snippet faqs/galaxy/interactive_tools_open.md %}
>
{: .hands_on}

A new page will be launched with Shiny-phyloseq ready to explore the phyloseq object.

## Visualize alpha-diversity

A **diversity index** is a quantitative measure that is used to assess the level of diversity or variety within a particular system, such as a biological community, a population, or a workplace. It provides a way to capture and quantify the distribution of different types or categories within a system.

Related to ecology, the term **diversity** describes the number of different species present in one particular area and their relative abundance. More specifically, several different metrics of diversity can be calculated. The most common ones are α, β and γ diversity.

The **α diversity** describes the diversity within a community. It considers the number of different species in an environment (also referred to as species **richness**). Additionally, it can take the abundance of each species into account to measure how evenly individuals are distributed across the sample (also referred to as species **evenness**).

There are several different indices used to calculate α diversity because different indexes capture different aspects of diversity and have varying sensitivities to different factors. These indexes have been developed to address specific research questions, account for different ecological or population characteristics, or highlight certain aspects of diversity. 

![Explanation of the α diversity](./images/alphadiversity_metrics.png "<a href="https://medium.com/pjtorres-high-gut-alpha-diversity-and-health/high-alpha-diversity-and-health-65e5eca7fa36">Source</a>")

Here we will plot the **α diversity** for the samples given the Days using 2 measures:

- The **Shannons** index, which calculates the uncertainty in predicting the species identity of an individual that is selected from a community ({% cite Shannon.1948 %}).

- The **Simpsons** index, which calculates the probability that two individuals selected from a community will be of the same species. Obtains small values in datasets of high diversity and large values in datasets of low diversity ({% cite SIMPSON.1949 %}).

> <hands-on-title> Plot α diversity </hands-on-title>
>
> 1. Click on **Alpha Diversity** tab
> 2. Select the following parameters:
>    - *"X"*: `Day`
>    - *"Color"*: `When`
>    - *"alpha Measures"*: `Shannon`, `Simpson`
{: .hands_on}

![2 panels for 2 α diversity metrics (Shannon, Simpson) with α diversity value on Y axis and days on X axis. The points are colored given When](./images/alpha_diversity.png)

> <question-title></question-title>
>
> Is there any obvious systematic difference in alpha-diversity between early and late samples?
>
> > <solution-title></solution-title>
> >
> > No
> >
> {: .solution}
>
{: .question}

## Plot Ordination

Ordination refers to a set of multivariate techniques used to analyze complex ecological data. It involves summarizing large datasets, such as species abundance across different samples, by projecting them onto a low-dimensional space. This visualization makes it easier to identify patterns and relationships within the data, such as how species distributions correspond to environmental gradients. Ordination helps understand the underlying factors that structure ecological communities and the relative importance of different environmental variables.

Phyloseq offers several ordination methods:

1. **MDS (Multidimensional Scaling)**: MDS is a technique that visualizes the level of similarity or dissimilarity of individual cases of a dataset by positioning them in a low-dimensional space.

2. **PCoA (Principal Coordinates Analysis)**: PCoA is a distance-based ordination method that transforms a distance matrix into a set of coordinates, allowing the visualization of the relationships between samples in a reduced dimensional space.

3. **DCA (Detrended Correspondence Analysis)**: DCA is an ordination method that removes the arch effect in Correspondence Analysis, providing a more accurate representation of ecological gradients.

2. **CCA (Canonical Correspondence Analysis)**: CCA is a direct gradient analysis technique that relates species composition data to measured environmental variables, highlighting the influence of these variables on community structure.

3. **RDA (Redundancy Analysis)**: RDA is a linear o

tools_json:
["Sort collection", "__SORTLIST__", "dada2: plotQualityProfile", "toolshed.g2.bx.psu.edu/repos/iuc/dada2_plotqualityprofile/dada2_plotQualityProfile/1.34.0+galaxy0", "dada2: filterAndTrim", "toolshed.g2.bx.psu.edu/repos/iuc/dada2_filterandtrim/dada2_filterAndTrim/1.34.0+galaxy0", "Unzip collection", "__UNZIP_COLLECTION__", "dada2: learnErrors", "toolshed.g2.bx.psu.edu/repos/iuc/dada2_learnerrors/dada2_learnErrors/1.34.0+galaxy0", "dada2: dada", "toolshed.g2.bx.psu.edu/repos/iuc/dada2_dada/dada2_dada/1.34.0+galaxy0", "dada2: mergePairs", "toolshed.g2.bx.psu.edu/repos/iuc/dada2_mergepairs/dada2_mergePairs/1.34.0+galaxy0", "dada2: makeSequenceTable", "toolshed.g2.bx.psu.edu/repos/iuc/dada2_makesequencetable/dada2_makeSequenceTable/1.34.0+galaxy0", "dada2: removeBimeraDenovo", "toolshed.g2.bx.psu.edu/repos/iuc/dada2_removebimeradenovo/dada2_removeBimeraDenovo/1.34.0+galaxy0", "dada2: sequence counts", "toolshed.g2.bx.psu.edu/repos/iuc/dada2_seqcounts/dada2_seqCounts/1.34.0+galaxy0", "dada2: assignTaxonomy and addSpecies", "toolshed.g2.bx.psu.edu/repos/iuc/dada2_assigntaxonomyaddspecies/dada2_assignTaxonomyAddspecies/1.34.0+galaxy0", "Group data by a column", "Grouping1", "Extract element identifiers", "toolshed.g2.bx.psu.edu/repos/iuc/collection_element_identifiers/collection_element_identifiers/0.0.2", "Replace Text in entire line", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.5+galaxy2", "Select first lines from a dataset (head)", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_head_tool/9.5+galaxy2", "Compute", "toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1", "Replace Text in a specific column", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1", "Concatenate multiple datasets or collections", "cat1", "Create phyloseq object from dada2", "toolshed.g2.bx.psu.edu/repos/iuc/phyloseq_from_dada2/phyloseq_from_dada2/1.50.0+galaxy3", "Phyloseq", "interactive_tool_phyloseq"]


