# tutorials

topic: genome-annotation
tutorial: braker3
title: Genome annotation with Braker3
difficulty: intermediate
time_estimation: 8h
requirements_json: []
tags_json: ["eukaryota", "Braker3", "jbrowse1", "biodiversity"]
content_hash: 592fa6887f72693b6782db7919205f11
last_modified: 2026-05-27T09:01:48.773894
zenodo_link: https://zenodo.org/records/14770765
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/braker3/tutorial.html

content:
Annotating the eukaryotic genome represents a somewhat more complex challenge than that of prokaryotes, mainly due to the generally larger size of eukaryotic genomes and their greater number of genes. This annotation can be carried out at different levels of precision, ranging from simple identification of coding and non-coding parts to detailed structural labeling, including for example the precise location of exons, introns and other regulatory elements.

In this tutorial we will use a software tool called Braker3 to annotate the genome sequence of a small eukaryote: [*Mucor mucedo*](https://en.wikipedia.org/wiki/Mucor_mucedo) (a fungal plant pathogen).

[Braker3](https://github.com/Gaius-Augustus/BRAKER) is an automated bioinformatics tool that uses RNA-seq and protein data to annotate genomes. It integrates GeneMark-ETP and AUGUSTUS software to predict genes with a high degree of precision. By combining the results of these two tools, Braker3 generates a final file containing gene annotations with strong extrinsic support (i.e. based on external experimental data).

Braker3 facilitates genome annotation by leveraging transcriptomic and protein data to produce more reliable and robust gene predictions.

In this tutorial, you will learn how to perform structural annotation of the genome and assess its quality. To simplify learning and reduce analysis time, we will use a lightweight dataset, which may result in a less complete annotation than in the context of an analysis under real conditions.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data upload

To annotate our genome using Braker3, we will use the following files:

- The **genome sequence** in fasta format. For best results, the sequence should be soft-masked beforehand. You can learn how to do it by following the [RepeatMasker tutorial]({% link topics/genome-annotation/tutorials/repeatmasker/tutorial.md %}). For this tutorial, we will try to annotate the genome assembled in the [Flye assembly tutorial]({% link topics/assembly/tutorials/flye-assembly/tutorial.md %}). The size of the genome has been reduced to reduce Braker3 execution time.
- Some **RNAseq data** in bam format. These reads are already aligned on the genome, and Braker3 will use it as evidences to annotate genes.
- A set of **protein sequences**, like UniProt/SwissProt. It is important to have good quality, curated sequences here, and the UniProt/SwissProt databank fits very well. For this tutorial, we have prepared a subset of this databank to speed up computing, but you should use UniProt/SwissProt for real life analysis.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/14770765/files/genome.fasta
>    https://zenodo.org/records/14770765/files/RNASeq.bam
>    https://zenodo.org/records/14770765/files/protein_sequences.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

# Preparing RNAseq data

In this tutorial, the alignments from RNA-seq are already prepared.
This data was obtained using the RNA STAR tool, as explained in detail in the [Funannotate]({% link topics/genome-annotation/tutorials/funannotate/tutorial.md %}) tutorial.

Note that when using Braker3 with STAR to align RNA-Seq data, the **--outSAMstrandField intronMotif** parameter must be added.
This parameter adds specific intron information to the alignment files (BAM).
This information is necessary for Braker3 to correctly understand and use the alignments to annotate genes.
Without this parameter, Braker3 may not function correctly or may produce incomplete results.

Here's how to run STAR with this specific parameter (**"Read alignement tags to include in the BAM output"** option), before annotating with Braker3:

> <hands-on-title></hands-on-title>
>
> 1. {% tool [RNA STAR](toolshed.g2.bx.psu.edu/repos/iuc/rgrnastar/rna_star/2.7.8a+galaxy0) %} with the following parameters:
>    - *"Single-end or paired-end reads"*: `Paired-end (as individual datasets)`
>        - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file, forward reads"*: `rnaseq_R1.fq.gz` (Input dataset)
>        - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file, reverse reads"*: `rnaseq_R2.fq.gz` (Input dataset)
>    - *"Custom or built-in reference genome"*: `Use reference genome from history and create temporary index`
>        - {% icon param-file %} *"Select a reference genome"*: `genome_masked.fasta` (Input dataset)
>        - *"Length of the SA pre-indexing string"*: `11`
>    - In *"BAM output format specification"*:
>        - In *"Read alignement tags to include in the BAM output"*:
>             - Select `XS (strand flag, see parameter help below)`
>
{: .hands_on}

# Structural annotation

We now have the following required inputs:

- A masked genome in FASTA format
- RNAseq sequence alignments in BAM format
- Protein annotation in FASTA format

With these files, We can run [**Braker3**](https://github.com/Gaius-Augustus/BRAKER) to perform the structural annotation of the genome.


> <hands-on-title>Genome annotation with Braker3</hands-on-title>
>
> 1. {% tool [Braker3](toolshed.g2.bx.psu.edu/repos/iuc/braker3/braker3/3.0.8+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Assembly to annotate"*: `genome_masked.fasta` (Input dataset)
>    - *"Species name"*: `Mucor mucedo`
>    - {% icon param-file %} *"RNA-seq mapped to genome to train Augustus/GeneMark"*: `rnaseq.bam` (Input dataset)
>    - {% icon param-file %} *"Proteins to map to genome"*: `SwissProt_subset.fasta` (Input dataset)
>    - {% icon param-file %} *"Fungal genome"*: select `Yes`
>    - *"Output format"*: `GFF3`
>
{: .hands_on}

> <comment-title>On parameters</comment-title>
>
> - If you are working with fungi, we recommend that you activate the **--fungus** parameter. This enables Braker3 to run the GeneMark-ETP algorithm with a branch point model.
{: .comment}

> <comment-title>Don't wait</comment-title>
>
> - This step will take a bit of time to run: it could take around 2 hours for this tutorial. While it runs, we can already schedule the following functional annotation steps. Galaxy will run them automatically as soon as the structural annotation is ready.
> - There are several reasons for this: the size of the genome, the quality and quantity of the RNA-Seq data, the supervised learning model for prediction, and more.
{: .comment}

Braker3 generates an output file in GTF format by default. But it is possible to generate the output file in GFF3 output. This file contains informations on gene locations, exons, introns, etc.
GFF3 files are generally preferred for complex and varied annotations, while GTF is often used for transcript annotations in RNA-seq studies.

The GFF3 format is a standard bioinformatics format for storing genome annotations. Each row describes a genomic entity, with columns detailing its identifier, location, score and other attributes.

## Evaluation with **Busco**

[BUSCO](http://busco.ezlab.org/) (Benchmarking Universal Single-Copy Orthologs) is a widely used tool to evaluate the quality of a genome assembly and annotation. By comparing genomes from
various related and distantly related species, the authors determined sets of ortholog genes that are present in single copy in (almost) all the species of a clade (Bacteria, Fungi, Plants, Insects, Mammalians, …).
Most of these genes are essential for the organism to live, and are expected to be found in any newly sequenced and annotated genome from the corresponding clade. Using this data, BUSCO evaluates the "completeness" of genome annotation by assessing the proportion of these essential genes (also named BUSCOs) found in a set of (predicted) transcript or protein sequences.

We want to run BUSCO on the protein sequences predicted from gene sequences of the Braker3 annotation.

So first generate these sequences:

> <hands-on-title>Extract protein sequences</hands-on-title>
>
> 1. {% tool [GFFread](toolshed.g2.bx.psu.edu/repos/devteam/gffread/gffread/2.2.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input GFF3 or GTF feature file"*: output of {% tool [Braker](toolshed.g2.bx.psu.edu/repos/iuc/braker3/braker3/3.0.8+galaxy0) %}
>    - In *"Reference Genome"* select: `From your history` (Input dataset)
>    - *"Genome Reference Fasta"*: `masked genome` (Input dataset)
>    - In *"Select fasta outputs"* select: `fasta file with spliced exons for each GFF transcript (-y)`
>    - *"full GFF attribute preservation (all attributes are shown)"*: `Yes`
>    - *"decode url encoded characters within attributes"*: `Yes`
>    - *"warn about duplicate transcript IDs and other potential problems with the given GFF/GTF records"*: `Yes`
>
{: .hands_on}

The parameters for running BUSCO on these protein sequences:

> <hands-on-title>BUSCO in proteome mode</hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Sequences to analyse"*: `gffread: pep.fa`
>    - *"Mode"*: `annotated gene sets (protein)`
>    - *"Auto-detect or select lineage?"*: `Select lineage`
>        - *"Lineage"*: `Mucorales`
>    - In *"Advanced Options"*:
>        - *"Which outputs should be generated"*: select all outputs
{: .hands_on}

Several output files are generated:

- short summary: statistical summary of the quality of genomic assembly or annotation, including total number of genes evaluated, percentage of complete genes, percentage of partial genes, etc.
- full table: list of universal orthologs found in the assembled or annotated genome, with information on their completeness, location in the genome, quality score, etc.
- missing buscos: list of orthologs not found in the genome, which may indicate gaps in assembly or annotation.
- summary image: graphics and visualizations to visually represent the results of the evaluation, such as bar charts showing the proportion of complete, partial and missing genes.
- GFF: contain information on gene locations, exons, introns, etc.

This gives information about the completeness of the Braker3 annotation. A good idea is to compare this first result with the one you get on the initial genome sequence,and see if the annotation tool found all the genes that BUSCO finds in the raw genome sequence.

The parameters for running BUSCO on the masked genome:

> <hands-on-title>BUSCO in genome mode</hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Sequences to analyse"*: `masked genome` (Input dataset)
>    - *"Mode"*: `Genome assemblies (DNA)`
>    - *"Auto-detect or select lineage?"*: `Select lineage`
>        - *"Lineage"*: `Mucorales`
>    - In *"Advanced Options"*:
>        - *"Which outputs should be generated"*: select all outputs
{: .hands_on}


> <comment-title>What can we deduce from these results?</comment-title>
>
> - Completeness: The results of 50.3% complete BUSCOs for the genome and 43.9% for proteins are relatively low. These results can be explained by the reduction in file size in order to speed up analyses with Braker3.
> - Duplication and fragmentation: The duplication rate is low and the proportion of fragmented genes is small, suggesting that the annotation is correct overall, but incomplete.
{: .comment}

## Evaluation with **OMArk**

[OMArk](https://github.com/DessimozLab/OMArk) is proteome quality assessment software.
It provides measures of proteome completeness, characterises the consistency of all
protein-coding genes with their homologues and identifies the presence of contamination
by other species. OMArk is based on the OMA orthology database, from which it exploits
orthology relationships, and on the OMAmer software for rapid placement of all proteins
in gene families.

OMArk's analysis is based on HOGs (Hierarchical Orthologous Groups), which play a central
role in its assessment of the completeness and coherence of gene sets. HOGs make it possible
to compare the genes of a given species with groups of orthologous genes conserved across a
taxonomic clade.

> <hands-on-title>OMArk on extracted protein sequences</hands-on-title>
>
> 1. {% tool [OMArk](toolshed.g2.bx.psu.edu/repos/iuc/omark/omark/0.3.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Protein sequences"*: `gffread: pep.fa`
>    - *"OMAmer database*: select `LUCA-v2.0.0`
>    - In *"Which outputs should be generated"*: select `Detailed summary`
>
{: .hands_on}

The OMArk tool generated an output file in .txt format containing detailed information on
the assessment of the completeness, consistency and species composition of the proteome
analysed. This report includes statistics on conserved genes, the proportion of duplications,
missing genes and the identification of reference lineages.

> <comment-title>What can we deduce from these results?</comment-title>
>
> - Number of conserved HOGs: OMArk has identified a set of 5622 HOGs which are thought to be conserved in the majority of species in the Mucorineae clade.
> - However, 54.43% of these HOGs were absent from the analysed proteome, which may indicate a potential incompleteness of the annotation. This observation could be linked to the reduction in the number of RNA sequences aligned on the genome when Braker was run, a choice made to optimise analysis time.
> - 80.27% of genes are complete, so the annotation is of good quality in terms of genomic completeness.
> - Number of proteins in the whole proteome: 5642. Of which 80.27% are present and 15.37% of the proteome does not share sufficient similarities with known gene families.
> - No contamination detected.
> - The OMArk analysis is based on the Mucorineae lineage, a more recent and specific clade than that used in the BUSCO assessment, which selected the Mucorales as the reference group.
{: .comment}

# Visualisation with a genome browser

You can visualize the annotation generated using a genomic browser like [JBrowse](https://jbrowse.org/jbrowse1.html).
This browser enables you to navigate along the chromosomes of the genome and view the structure of each predicted gene.

> <hands-on-title>JBrowse visualisation</hands-on-title>
>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: `genome_masked.fasta` (Input dataset)
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Annotation`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `gff3` (output of **Braker3** {% icon tool %})
{: .hands_on}

Click on the newly created dataset's eye to display it. You will see a JBrowse genome browser. You can have a look at the [JBrowse tutorial]({% link topics/visualisation/tutorials/jbrowse/tutorial.md %}) for a more in-depth description of JBrowse.

We have embedded a copy of the resulting JBrowse here, if something went wrong during one of the steps you can always just check this output:


{% capture jbtracks %}DNA%2Caa6aca08d6fd82b94c7a86712a632760_0{% endcapture %}
{% snippet topics/visualisation/faqs/visualizations_jbrowse.html datadir="data" tracks=jbtracks %}

Check that **Braker3 annotation** is selected in the top left-hand corner. If not, select it to view the annotation.

# Conclusion

Congratulations on reaching the end of this tutorial! You now know how to perform a structural annotation of a new eukaryotic genome, using Braker3. And you've learned how to evaluate its quality and how to visualize it using JBrowse.

If you'd like to complete this annotation, we recommend you to follow the tutorial on [functional annotation]({% link topics/genome-annotation/tutorials/functional/tutorial.md %}) with EggNOG Mapper and InterProScan. You can follow it with the protein sequences we generated earlier with gffread.

tools_json:
["RNA STAR", "toolshed.g2.bx.psu.edu/repos/iuc/rgrnastar/rna_star/2.7.8a+galaxy0", "Braker3", "toolshed.g2.bx.psu.edu/repos/iuc/braker3/braker3/3.0.8+galaxy0", "GFFread", "toolshed.g2.bx.psu.edu/repos/devteam/gffread/gffread/2.2.1.4+galaxy0", "Braker", "Busco", "toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.7.1+galaxy0", "OMArk", "toolshed.g2.bx.psu.edu/repos/iuc/omark/omark/0.3.0+galaxy2", "JBrowse", "toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1"]


topic: genome-annotation
tutorial: apollo-euk
title: Refining Genome Annotations with Apollo (eukaryotes)
difficulty: intermediate
time_estimation: 3h
requirements_json: ["type: \"internal\"", "type: \"internal\""]
tags_json: ["gmod", "eukaryote", "cyoa", "jbrowse1", "apollo2", "biodiversity"]
content_hash: 4939d62b8c7fe392fc900c269bfbf481
last_modified: 2026-05-27T09:01:48.732238
zenodo_link: https://zenodo.org/record/6920962
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/apollo-euk/tutorial.html

content:
After automatically annotating your genome using [Funannotate](../funannotate/tutorial.html) or [Maker](../annotation-with-maker/tutorial.html) for example, it is important to visualize your results so you can understand what your organism looks like, and then to manually refine these annotations along with any additional data you might have. This process is most often done as part of a group, smaller organisms may be annotated individually though.

> <warning-title>Only works on UseGalaxy.eu</warning-title>
> Currently this tutorial requires an Apollo server to be deployed by the administrator. This will currently only work on UseGalaxy.eu, hopefully this list will expand in the future.
{: .warning}

[Apollo](https://github.com/gmod/apollo) {% cite Dunn2019 %} provides a platform to do this. It is a web-based, collaborative genome annotation editor. Think of it as "Google Docs" for genome annotation, multiple users can work together simultaneously to curate evidences and annotate a genome.

This demo is inspired by the [Apollo User's Guide](https://genomearchitect.readthedocs.io/en/latest/UsersGuide.html), which provides additional guidance.

> <comment-title>Ask your instructor!</comment-title>
> This tutorial shows you how to use Galaxy to set up a genome curation project by loading a new organism in Apollo. If you only want to focus on the usage of Apollo, your instructor might already have created for you an organism in Apollo. In this case select the `Use a pre-created organism` option below (you will skip a few steps in Galaxy), and select the organism prepared for you by your instructor.
>
> If you're doing this tutorial on your own, or if the instructor has not set up a specific Apollo organism for you, select `Create your own organism`.
{: .comment}

{% include _includes/cyoa-choices.html option1="Create your own organism" option2="Use a precreated organism" default="Create your own organism"
       text="You can decide to create an organism or use a pre-created one." %}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Preparing for Apollo

<div class="Create-your-own-organism" markdown="1">

## Data upload


To annotate a genome using Apollo, we need the reference genome sequence in FASTA format, and any evidence tracks we want to refine into our annotations. "Evidence tracks" can be any data like:

- A set of prior gene predictions or other genomic feature predictions
- The output of a bioinformatics analysis like BLAST or InterProScan
- Sequencing reads from RNA-Seq or another HTS analysis
- If you are not doing a *de novo* annotation, then a previous released {OGS}.

In this tutorial we use the same data as in the [Funannotate](../funannotate/tutorial.html) tutorial: the genome sequence, the annotation produced by **Funannotate** {% icon tool %}, and the mapping of RNA-Seq data along the genome using **RNA star** {% icon tool %}. We will visualise this data and use it to make some manual annotations in order to familiarise you with the process.

> <comment-title>Real Data: Unreal Circumstances</comment-title>
> While the data for this tutorial is sourced from publicly available databases, and is all related to different experiments on *Mucor mucedo*, this is not necessarily the data *you* might use to annotate your genomes. You probably know best what data you should be using in your own circumstances, for the specific features on which you are focused.
{: .comment}

### Get data


> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/record/6920962/files/genome.fasta
>    https://zenodo.org/record/6920962/files/annotation.gff3
>    https://zenodo.org/record/6920962/files/rnaseq.bam
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

## Using Apollo for Annotation


Refining genome annotations happens in multiple steps:

- Create a JBrowse instance from the reference genome FASTA file and evidence tracks
- Import this data into Apollo
- Refine the annotations
- Export the refined genome annotations

In this tutorial we will focus more on the practical portions than the theoretical part of genome annotation, that are covered in other tutorials. When you've completed this tutorial you should be comfortable manipulating genomic data in Galaxy and Apollo.

> <details-title>Why bother?</details-title>
>
> Automated annotation programs continue to improve, however a simple score may not provide evidence necessary to confirm an accurate prediction.
> Therefore, it is necessary to both visually inspect the results and manually fix any issues with the predictions.
>
> Additionally, many times assemblies are less than perfect or read depth and quality may be insufficient, leading to imperfect automatic annotation.
{: .details}

### Build the JBrowse Instance

Let's begin by building a JBrowse instance with all the data we have for this genome.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: `genome.fasta` (Input dataset)
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Annotation`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-files %} *"GFF/GFF3/BED Track Data"*: `annotation.gff3` (Input dataset)
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `RNA-Seq`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BAM Pileups`
>                        - {% icon param-files %} *"BAM Track Data"*: `rnaseq.bam` (Input dataset)
>                        - *"Autogenerate SNP Track"*: `yes`
>
>    > <comment-title>JBrowse is highly configurable</comment-title>
>    >
>    > JBrowse is highly configurable, we have set a very basic configuration but there are many more advanced features available to you, if you need them. You can choose precisely how data is displayed, and even what menu options are available when users click on features. If your features have some external identifiers like an NCBI Gene ID, you can even configure JBrowse that when the user clicks on the feature, it should show the gene page for that feature in a new tab. These sort of features are incredibly helpful for building very rich experiences.
>    >
>    > A static genome browser like this (just JBrowse, not in Apollo) is very useful for summarising results of a genomics workflow, where the next step is simply interpretation and not annotation. Have a look at the [JBrowse tutorial]({% link topics/visualisation/tutorials/jbrowse/tutorial.md %}) for more information.
>    >
>    > Currently we have built a standalone genome browser (data + the html page and user interface and javascript), but it's possible to just compile the data directory if you intend to send this data to Apollo, and don't need to view the static data in Galaxy.
>    {: .comment}
{: .hands_on}

This tool will take some time to run dependent on data size. All of the inputs need to be pre-processed by JBrowse into a form that it can render and visualise easily. Once this is complete, you can click on the {% icon galaxy-eye %} eyeball to view the JBrowse instance. This is a static view into the data, JBrowse does not let you make any annotations or save any changes. We will convert it into a dynamic view where we can make persistent annotations and share these with our colleagues.

### Sending data to Apollo

Now that we have a good looking static JBrowse instance, it is time to load it into Apollo to turn it into a dynamic view where you can make modifications to the genes.

> <hands-on-title>Import to Apollo</hands-on-title>
>
> 1. {% tool [Create or Update Organism](toolshed.g2.bx.psu.edu/repos/gga/apollo_create_or_update/create_or_update/4.2.5) %} with the following parameters:
>    - {% icon param-file %} *"JBrowse HTML Output"*: output of **JBrowse** {% icon tool %}
>    - *"Organism Common Name Source"*: `Direct Entry`
>        - *"Organism Common Name"*: `Mucor mucedo`
>    - *"Genus"*: `Mucor`
>    - *"Species"*: `mucedo`
>
> 2. {% tool [Annotate](toolshed.g2.bx.psu.edu/repos/gga/apollo_iframe/iframe/4.2.5) %} with the following parameters:
>    - {% icon param-file %} *"Apollo Organism Listing"*: output of **Create or Update Organism** {% icon tool %}
>
> 3. View {% icon galaxy-eye %} the output of the Annotate tool, when it is ready.
>
{: .hands_on}

Viewing the output will open a view into Apollo in the main panel. Here you can interact with your genome and make annotations. This "Annotate" output is a quick link to that specific genome, and while Apollo allows you to manage and annotate multiple genomes, this dataset will always take you back to that specific genome. You can additionally access the Apollo server outside of Galaxy. While the URL will be different for each Galaxy server that supports Apollo, UseGalaxy.eu's Apollo server is available at [https://usegalaxy.eu/apollo](https://usegalaxy.eu/apollo).

</div>
<div class="Use-a-precreated-organism" markdown="1">

While the URL will be different for each Galaxy server that supports Apollo, to access UseGalaxy.eu's Apollo server:

- First login on [https://usegalaxy.eu](https://usegalaxy.eu)
- Then go to [https://usegalaxy.eu/apollo](https://usegalaxy.eu/apollo).

It is important to first login on [https://usegalaxy.eu](https://usegalaxy.eu) before connecting to Apollo as it will set up everything to make Apollo recognize your user account.

Your instructor should give you the name of the organism you will work on, you need to select it in the drop down box at the top of the right panel of Apollo. If multiple users are using the same organism, you will experiment the true collaborative aspect of Apollo: all the actions performed on the gene models by anyone will be seen by everyone instantly.
</div>
# Apollo
<div class="Create-your-own-organism" markdown="1">

![Screenshot of Apollo in Galaxy, apollo takes up most of the space in the center iframe with the history shown on the right.](../../images/apollo/ui.png "This is the Apollo interface, when viewed from inside Galaxy. The Annotate tool produces a special output which acts like a link to the Apollo server. It is <b>not</b> sufficient to download this file to save a copy of your annotations, unlike other datasets in Galaxy. In the main panel Apollo is displayed, with two panels of its own: the Annotation Window (left), and the Annotator Panel (right). The annotation window is the main view into our genome and here we will see evidence, reconcile it, and make our annotations. The right panel allows us to show or hide individual evidence tracks, switch between organisms, and navigate around the genome.")

</div>
<div class="Use-a-precreated-organism" markdown="1">

![Screenshot of Apollo.](../../images/apollo/ui_apollo.png "This is the Apollo interface, with two panels of its own: the Annotation Window (left), and the Annotator Panel (right). The annotation window is the main view into our genome and here we will see evidence, reconcile it, and make our annotations. The right panel allows us to show or hide individual evidence tracks, switch between organisms, and navigate around the genome.")

> <details-title>Organism naming</details-title>
>
> At the top of the right panel, you can switch from one organism to another. Depending on the server you are using, the organism names can be suffixed with `gx[number]` or an email address: this is to avoid possible confusion between organisms with the same name created by multiple users.
{: .details}

</div>

From the Apollo user manual:

> The major steps of manual annotation using Apollo can be summarized as follows:
>
> 1. Locate a chromosomal region of interest.
> 2. Determine whether a feature in an existing evidence track provides a reasonable gene model to start annotating.
> 3. Drag the selected feature to the ‘User Annotation’ area, creating an initial gene model.
> 4. Use editing functions to edit the gene model if necessary.
> 5. Check your edited gene model for consistency with existing homologs by exporting the FASTA formatted sequence and searching a protein sequence database, such as UniProt or the NCBI Non Redundant (NR) database, and by conducting preliminary functional assignments using the Gene Ontology (GO) database.
>
{: .quote}

The first four steps are generally the process of structural annotation (the process of identifying the correct gene model), and the last includes functional annotation (the process of assigning a putative function to a gene in your annotations).

## Evidence tracks

Let's start by looking at the tracks available to us, and then turning on the annotation track so we can start exploring our data.

> <hands-on-title>Visualize the annotation</hands-on-title>
>
> 1. In the right hand panel at the top click on **Tracks** to open the track listing
>
>    ![Zoomed in screenshot of the track menu with the Annotation set of tracks expanded showing Augustus and NCBI AnnotWriter genes.](../../images/apollo/tracks_eu.png)
>
> 2. In the **Annotation** group, select the `annotation.gff3` track.
>
>    You can either activate tracks in bulk, by clicking on the checkbox to the right of the group name ("Annotation"), or by clicking on the group name to expand the section, and then selecting individual tracks.
>
> 3. Zoom to a region of the genome.
>
>    1. In the left hand Annotation Window, at the top navigation bar you will find a textbox which shows the current location on the genome.
>    2. Edit this and enter `scaffold_1:3000..12000`
>    3. Press *Go* or use <kbd>Enter</kbd> on your keyboard.
>
>    ![Screenshot of Jbrowse header with the control panel with magnifying glasses and zoom location.](../../images/apollo/location_euk.png)
>
{: .hands_on}

> <comment-title>Track names</comment-title>
> Apollo uses the dataset names from your Galaxy history to name the tracks in the track list.
{: .comment}

We can now see an evidence track: `annotation.gff3` is the output of the [Funannotate tutorial](../funannotate/tutorial.html). In a *de novo* annotation project, we probably will only have the outputs of various gene callers, and potentially some expression evidence like RNA-Seq.

For this organism, we only have one track in the **Annotation** group, but an organism can have multiple annotations coming from different sources.

## Adding new genes

With the selected track, if you look along the genome, you will see many genes that were predicted by Funannotate. Each of them as a name assigned by Funannotate based on automatic functional annotation. If you right click on gene, an select `View details`, you can get access to detailed information on the predicted gene, including its unique name (e.g. `FUN_000002`), functional annotation (e.g. `InterPro:IPR006594`, `GO:0005515`) or the coding sequence (CDS).

> <details-title>Gene colors</details-title>
> Each gene color corresponds to an open reading frame on the genome. This allows to quickly see if two genes that are very close are on the same open reading frame, which could mean that they can be merged into a single gene, if other evidences support this.
{: .details}

What we want now is first to check that the structure of the genes predicted by Funannotate are in good shape. To do this, we will display an additional evidence tracks.

> <hands-on-title>Display RNA-Seq track</hands-on-title>
>
> 1. In the right hand panel at the top click on **Tracks** to open the track listing
>
> 2. In the **RNA-Seq** group, select the `rnaseq.bam - SNPs/Coverage` track and then the `rnaseq.bam` track.
>
{: .hands_on}

These new tracks represent the result of aligning RNA-Seq reads along the genome, with two different representation. It should look like that:

![JBrowse with RNA-Seq data shown, genes appear as red green and blue stripes with a pointed end, RNA-Seq appear as a coverage plot along the genome, and as individual reads as red and blue stripes](../../images/apollo/rnaseq_euk.png)

The first track (`rnaseq.bam - SNPs/Coverage`) is a coverage xy-plot, representing the number of reads aligned on the genome at each position. The little coloured teardrops represent positions where mismatches where found between some reads and the reference sequence (SNPs or sequenceing errors). Light grey regions corresponding to portions of the genomes where RNASeq were mapped, and darker grey regions corresponding to introns (= regions were some reads were found to match both the end of an exon, and the start of the next one).

The second track (`rnaseq.bam`) represent the alignment of each individual RNA-Seq read. Red reads were aligned in the forward strand, and blue ones in the reverse strand.

Let's suppose you want to improve the annotation of one of the gene. To do it, you can either click on one of its intron and drag and drop it to the User-created Annotations track (yellow background, at the top), or if you prefer, right click on one of its intron and select **Create new annotation** > **gene**. It should appear shortly in the **User-created Annotation** track.

<div class="Use-a-precreated-organism" markdown="1">
> <warning-title>Working with others</warning-title>
> Maybe you're not doing this tutorial alone, if you see that other people have started working on this region of the genome, it is best to move to a different region to avoid disturbing them by modifying the same genes at the same time.
{: .warning}
</div>

![One gene is added to the User-created Annotation track](../../images/apollo/added_genes_euk.png)

You can also see that it appears now in the list of genes in the right panel, at the gene and the mRNA level (as a gene can have multiple isoforms).

![Gene list showing gene FUN_000005 which is expanded to reveal a child mRNA](../../images/apollo/gene_list_euk.png)

The **User-created Annotation** track is where you can make modifications to genes, like changing their coordinates, or their name and functional annotation. If you right click on a gene in this track, you will see all the possibilities offered by apollo.

![Gene contextual menu showing way too many options, the mouse hovers over Open Annotation (alt-click)](../../images/apollo/gene_menu_euk.png)

Currently, the gene we added has a meaningless name. Let's improve that: right click on the gene, and click on **Open Annotation (alt-click)**.

![Gene details menu in apollo showing type set to gene, the name field has been filled with FUN_000004-T1 and a number of other empty fields like symbol, aliases, description.](../../images/apollo/gene_details_euk.png)

One way to give a better name is to find a similarity with genes of another species: if you right click on the gene, you can get the protein sequence by clicking on **Get Sequence**, and then compare it with a sequence database with Blast for example (using Galaxy, or the [NCBI blast form](https://blast.ncbi.nlm.nih.gov/Blast.cgi) for example).

In this case, the blast result suggests that this name is similar to `alpha/beta-hydrolase` genes in other fungi species. This is not a very strong evidence, but for this exercise, let's use this as a name: write `alpha/beta-hydrolase` in the **Name** field (type it manually and select it from the drop down list).

> <details-title>Naming genes</details-title>
>
> Giving a proper name to a gene is not always easy. Should it include "Putative" or not? What if multiple names can apply? Should it be lowercase or uppercase?
> The important thing is to always use the same naming rules when working on a full annotation, and to agree on these rules with other collaborators.
> Usually, big annotation consortiums have naming guidelines that you are supposed to follow.
{: .details}

We have just edited the gene name, but Apollo allows to edit information at the mRNA level too. Click on the **Sync name with transcript** button to copy the gene name to the mRNA name. It should now display in the **User-created Annotation** track. To check what you can edit at the mRNA level, just click on the corresponding mRNA in the list above:

![very zoomed screenshot showing clicking on the transcript below the gene to open the mRNA info.](../../images/apollo/mrna_level_euk.png)

You should see `alpha/beta-hydrolase` in the **Name** field.

> <comment-title>Saving your work</comment-title>
>
> You do not need to do anything specific to *Save* your work in Apollo. Just like Google Docs, each modification is immediately saved, and any other user working on the same genome will instantly see the changes you make.
{: .comment}

## Editing a gene structure

Apollo allows to edit the whole structure of a gene. If you zoom to the 5' end of the gene, you will notice two things:

- a few RNA-Seq reads were mapped a little before the 5' limit of our gene
- there is another potential start codon a few bases after the one that was predicted by Funannotate

Let's modify the 5' of the gene to add a 5' untranslated region. Although we don't have enough evidences to be sure that the start codon should be changed, for this exercise will we do it too. To do it, all you need to do is click on the 5' limit of the gene and drag it to the desired position. You will notice that the structure of the gene will be shortly changed. Changing the start codon can be done by right clicking on the new position and clicking on "Set Translation Start"

![Gif showing modifying the 5' region of a gene by dragging it.](../../images/apollo/change_start_euk.gif)

The new white region corresponds to the newly create 5' untranslated region.

This kind of modifications is very common when using Apollo, and you can perform it at the gene level, and at the exon/intron level (an exclaim mark appears when Apollo detects that an exon/intron junction doesn't match with the cannonical acceptor/donor sequences).

![Gif showing modifying a exon/intron junction of a gene by dragging it.](../../images/apollo/exon_edit.gif)

To guide you doing these changes, you should look at all the tracks available for the genome you study. RNA-Seq track are very helpful to determine the limits of coding sequences on the genome. Other tracks can be helpful, like alignements of transcripts or proteins from closely related species (or even big databanks like Swissprot or NR).

## Working with isoforms

In the eukaryote world, with the process of alternative splicing, multiple mRNA sequences (isoforms) can be generated for a single gene, based on the use of various splice site combinations. This process has a complex regulation mechanism allowing to express or not each isoform depending on multiple conditions (tissue, developmental stage, environmental conditions, ...).

Often automatic annotation softwares produce gene models composed of many exons, but does not give the detail of the sequence of each potential isoforms. However, in some cases, the RNA-Seq data and the scientific knowledge on specific gene families can be used to annotate each isoform of a gene.

Apollo allows to do that: just duplicate an existing gene model at the same position. Each copy will be considered as an isoform of the same gene. You can then make all the modifications you want to each one independently.

![Gif showing creating two isoforms and removing exons from them.](../../images/apollo/isoforms.gif)

The isoforms will appear in the gene list, under their common gene.

![Gene list showing the three isoforms of the same gene.](../../images/apollo/isoform_list.png)

As isoforms regulation depends heavily on multiple conditions (tissue, developmental stage, environmental conditions, ...), it is important when annotating a new genome to have access to a lot of RNA-Seq data sequenced in multiple conditions and from different tissues.

## Viewing and reverting changes

Everything you do in Apollo is tracked in a database. If you right click on the gene, and select **Show History**, you have access to the full list of all the actions that were performed on it.

![History of changes on a gene showing many changes to Set Name and a final Set Exon Boundaries change](../../images/apollo/history.png)

When you click on one of the steps, you can see below the list a preview of how the gene looked at the time. And you can revert to a specific version of the gene by clicking on the arrow button on the right.

## Adding more functional annotation

Sometimes you'll want to modify a gene that was predicted by FunAnnotate, just to add functional annotation to it, like Gene Ontology terms, of references to external databases (UniProt, InterPro, ...). This can be done using Apollo, have a look at the [Apollo tutorial for prokaryotes]({% link topics/genome-annotation/tutorials/apollo/tutorial.md %}) for more details.

# Exporting and collaborating

<div class="Create-your-own-organism" markdown="1">
{% include topics/genome-annotation/tutorials/apollo/collab.md eukaryote=true %}
</div>
<div class="Use-a-precreated-organism" markdown="1">
It is of course possible to export all your annotation from Apollo in standard formats, directly from its interface (`Ref Sequence` tab), or from Galaxy. If you create your own organism in Apollo, you can choose to make it open to the public, or to restrict it to only your collaborators.

Click on **Create your own organism** at the beginning to learn more.

</div>

# Conclusion


Congratulations, you finished this tutorial! By using Apollo and JBrowse, you learned how to manually refine predicted annotations<span class="Create-your-own-organism"> and export them to Galaxy for future analyses. You also learn how to give access to your project at any other researcher, making it a real collaborative solution</span><span class="Use-a-pre-created-organism">. Although you use a pre-created organism, remember that you can use Galaxy to add your own organism to Apollo, and give access to your project at any other researcher, making it a real collaborative solution</span>.

A [similar tutorial for prokaryote genomes]({% link topics/genome-annotation/tutorials/apollo/tutorial.md %}) exists, using different types of evidence tracks, feel free to have a look at it to learn more.

When refinement is sufficient an updated or new version of the annotation may be exported as GFF3 as well as published as a new JBrowse directory for inspection.

# What's next?

After generating your refined annotation, you'll want to merge it back into the official gene sets. A future tutorial will show you how to do it within Galaxy.

If a *de novo* set, you can export it as GFF3 and load it into a tool like [Tripal](http://tripal.info) to provide visualization.

tools_json:
["JBrowse", "toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1", "Create or Update Organism", "toolshed.g2.bx.psu.edu/repos/gga/apollo_create_or_update/create_or_update/4.2.5", "Annotate", "toolshed.g2.bx.psu.edu/repos/gga/apollo_iframe/iframe/4.2.5"]


topic: genome-annotation
tutorial: annotation-with-maker-short
title: Genome annotation with Maker (short)
difficulty: intermediate
time_estimation: 2h
content: {% include topics/genome-annotation/tutorials/annotation-with-maker/content.md short=true %}
tools_json: []
requirements_json: []
tags_json: ["gmod", "eukaryote", "maker", "jbrowse1"]
content_hash: f1a705520163589b3c417cb47f67fbad
last_modified: 2026-05-15T22:28:13.461658
zenodo_link: https://doi.org/10.5281/zenodo.4406623
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/annotation-with-maker-short/tutorial.html


topic: genome-annotation
tutorial: helixer
title: Genome annotation with Helixer
difficulty: intermediate
time_estimation: 4h
requirements_json: []
tags_json: ["eukaryota", "helixer", "jbrowse1", "biodiversity"]
content_hash: 99a04e80a479d24ba096af6a79dfe38a
last_modified: 2026-05-27T09:01:48.986042
zenodo_link: https://zenodo.org/record/7867921
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/helixer/tutorial.html

content:
Annotating the eukaryotic genome represents a somewhat more complex challenge than that of prokaryotes, mainly due to the generally larger size of eukaryotic genomes and their greater number of genes, but also to the complexity of eukaryotic genes structure (e.g. exons and {UTR}). This annotation can be carried out at different levels of precision, ranging from simple identification of coding and non-coding parts to detailed structural labeling, including for example the precise location of exons, introns and other regulatory elements.


In this tutorial we will use a software tool called Helixer to annotate the genome sequence of a small eukaryote: [*Mucor mucedo*](https://en.wikipedia.org/wiki/Mucor_mucedo) (a fungal plant pathogen).

[Helixer](https://github.com/weberlab-hhu/Helixer) is an annotation software with a new and different approach: it performs evidence-free predictions (no need for RNASeq data or sequence aligments), using {GPU}, with a much faster execution time. The annotation is based on the development and use of a cross-species deep learning model. The software is used to configure and train models for *ab initio* prediction of gene structure. In other words, it identifies the base pairs in a genome that belong to the UTR/CDS/Intron genes.

In this tutorial, you'll learn how to perform a structural annotation of the genome and how to assess its quality.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data upload

To annotate our genome using Helixer, we will use the following files:

- The **genome sequence** in fasta format. For this tutorial, we will try to annotate the genome assembled in the [Flye assembly tutorial]({% link topics/assembly/tutorials/flye-assembly/tutorial.md %}). (Note: Helixer will ignore soft-masking. Hard-masking is not recommnded for Helixer either, as it does not ignore the hard-masked regions, but will get less information from them, which could influence your predictions in a negative way.)

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/record/7867921/files/genome_masked.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

# Structural annotation

We can run [**Helixer**](https://github.com/weberlab-hhu/Helixer) to perform the structural annotation of the genome.

We need to input the genome sequence we want to annotate.

We also need to choose between 4 different lineages: *invertebrate*, *vertebrate*, *land plant* or *fungi*. Select the one that fits the best to the species you're studying: *fungi* in our case. Helixer is shipped with these 4 models that were trained specifically to annotate genes from each of these lineages. Advanced users can upload their own lineage model in .h5 format with the *"Lineage model"* option.

As an option, we can also enter a species name.


> <hands-on-title></hands-on-title>
>
> 1. {% tool [Helixer](toolshed.g2.bx.psu.edu/repos/genouest/helixer/helixer/0.3.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Genomic sequence"*: `genome_masked.fasta` (Input dataset)
>    - In *"Available lineages"*: "*select*" `fungi`
>    - In *"Species name"*: `Mucor mucedo`
>
{: .hands_on}

> <comment-title>Advanced parameters</comment-title>
>
> Depending on the lineage, the parameters *"Subsequence length"*, *"Overlap offset"* and *"Overlap corelength"* are adjusted to corresponding default values (listed in the help of each option).
>
> This is due in particular to the size of the genomes. Indeed, it is recommended to increase the value of *"Subsequence length"* for genomes containing large genes. This is particularly important for vertebrates and invertebrates.
>
> The default values used by Galaxy are the ones recommended by Helixer authors. If you wish to modify these default values, you can do so by entering your values in the *"Subsequence length"*, *"Overlap offset"* and *"Overlap corelength"* parameters.
>
{: .comment}

> <comment-title>Don't wait</comment-title>
>
> This step can take a bit of time to run: although Helixer runs much faster than many other annotation tools (typically <20min for this tutorial), it requires specific hardware (GPU) that is often available in limited quantity on computing systems. It means your job can be placed in queue for a longer time than a more standard Galaxy job.
>
> While it runs, we can already schedule the following steps. Galaxy will run them automatically as soon as the Helixer annotation is ready.
{: .comment}

Helixer produces a single output dataset: a GFF3 file. The GFF3 format is a standard bioinformatics format for storing genome annotations. Each row describes a genomic entity, with columns detailing its identifier, location, score and other attributes.

# Quality evaluation

## General statistics

Genome Annotation Statistics is a program designed to analyze and provide statistics on genomic annotations. This software performs its analyses from a GFF3 file.

> <hands-on-title>Genome Annotation Statistics</hands-on-title>
>
> 1. {% tool [Genome Annotation Statistics](toolshed.g2.bx.psu.edu/view/iuc/jcvi_gff_stats/jcvi_gff_stats/0.8.4) %} with the following parameters:
>    - {% icon param-file %} *"Annotation to analyse "*: `GFF3 file` (Output of **Helixer**)
>    - In *"Reference genome"*: select `Use a genome from history`
>    - In *"Corresponding genome sequence"*: `genome_masked.fasta` (Input dataset)
{: .hands_on}

Two output files are generated:
- a file containing graphs in pdf format
- a summary in txt format

> <comment-title>What can we deduce from these results?</comment-title>
>
> - The summary file provides statistics on the genome annotation and gives a complete overview of the genomic structure and characteristics of the genes, exons and introns in the analysed genome.
> - We can see that there are 19,299 genes, 77% of which are multi-exons (i.e. 14,860) and 23% single-exons (i.e. 4,439).
> - We can obtain other information such as the average size of exons, the percentage in GC or the average size of transcripts.
>
{: .comment}

These statistics are interesting on their own: you often have a rough idea of the expected number of genes or mean length when annotating a new genome, by comparing with published similar species. You can also use them to compare the quality of annotations produced by different tools.

## Evaluation with **Busco**

[BUSCO](http://busco.ezlab.org/) (Benchmarking Universal Single-Copy Orthologs) is a tool allowing to evaluate the quality of a genome assembly or of a genome annotation. By comparing genomes from various more or less related species, the authors determined sets of ortholog genes that are present in single copy in (almost) all the species of a clade (Bacteria, Fungi, Plants, Insects, Mammalians, ...). Most of these genes are essential for the organism to live, and are expected to be found in any newly sequenced and annotated genome from the corresponding clade. Using this data, BUSCO is able to evaluate the proportion of these essential genes (also named BUSCOs) found in a set of (predicted) transcript or protein sequences. This is a good evaluation of the "completeness" of the annotation.

As an alternative for genomes only one can use [**compleasm**](https://github.com/huangnengCSU/compleasm) with the same BUSCO gene sets, as compleasm is a bit more sensitive and thus allows finding slightly more conserved genes. 

We want to run BUSCO on the protein sequences predicted from gene sequences of the Helixer annotation. So first generate these sequences:

> <hands-on-title>Extract protein sequences</hands-on-title>
>
> 1. {% tool [GFFread](toolshed.g2.bx.psu.edu/repos/devteam/gffread/gffread/2.2.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input GFF3 or GTF feature file"*: output of {% tool [Helixer](toolshed.g2.bx.psu.edu/repos/genouest/helixer/helixer/0.3.3+galaxy1)) %}
>    - In *"Reference Genome"* select: `From your history` (Input dataset)
>    - *"Genome Reference Fasta"*: `masked genome` (Input dataset)
>    - In *"Select fasta outputs"* select: `protein fasta file with the translation of CDS for each record (-y)`
>    - *"full GFF attribute preservation (all attributes are shown)"*: `Yes`
>    - *"decode url encoded characters within attributes"*: `Yes`
>    - *"warn about duplicate transcript IDs and other potential problems with the given GFF/GTF records"*: `Yes`
>
{: .hands_on}

To run BUSCO on these protein sequences:

> <hands-on-title>BUSCO in proteome mode</hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Sequences to analyse"*: `gffread: pep.fa`
>    - *"Mode"*: `annotated gene sets (protein)`
>    - *"Auto-detect or select lineage?"*: `Select lineage`
>        - *"Lineage"*: `Mucorales`
>    - In *"Advanced Options"*:
>        - *"Which outputs should be generated"*: select all outputs
>
{: .hands_on}

Several output files are generated:

- short summary : statistical summary of the quality of genomic assembly or annotation, including total number of genes evaluated, percentage of complete genes, percentage of partial genes, etc.
- full table : list of universal orthologs found in the assembled or annotated genome, with information on their completeness, location in the genome, quality score, etc.
- missing buscos : list of orthologs not found in the genome, which may indicate gaps in assembly or annotation.
- summary image : graphics and visualizations to visually represent the results of the evaluation, such as bar charts showing the proportion of complete, partial and missing genes.
- GFF : contain information on gene locations, exons, introns, etc.

This gives information about the completeness of the Helixer annotation. A good idea is to compare this first result with the one you get on the initial genome sequence, and see if the annotation tool found all the genes that BUSCO finds in the raw genome sequence. So run BUSCO in genome mode:

> <hands-on-title>BUSCO in genome mode</hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Sequences to analyse"*: `masked genome` (Input dataset)
>    - *"Mode"*: `Genome assemblies (DNA)`
>    - *"Auto-detect or select lineage?"*: `Select lineage`
>        - *"Lineage"*: `Mucorales`
>    - In *"Advanced Options"*:
>        - *"Which outputs should be generated"*: select all outputs
>
{: .hands_on}

> <comment-title>What can we deduce from these results?</comment-title>
>
> - 94.6% of genes are complete, so the annotation is of high quality in terms of genomic completeness.
> - It is a little bit lower than what BUSCO is able to find in genome mode (95.7%), but the difference is quite small so Helixer seems to have generated a quite good result.
> - The duplication rate is low, with 1.3% and 1.5% of genes duplicated.
> - So the Helixer annotation looks like a good one, with high completeness and low duplication.
>
{: .comment}

## Evaluation with **OMArk**

[OMArk](https://github.com/DessimozLab/OMArk) is proteome quality assessment software. It provides measures of proteome completeness, characterises the consistency of all protein-coding genes with their homologues and identifies the presence of contamination by other species. OMArk is based on the OMA orthology database, from which it exploits orthology relationships, and on the OMAmer software for rapid placement of all proteins in gene families.

OMArk's analysis is based on HOGs (Hierarchical Orthologous Groups), which play a central role in its assessment of the completeness and coherence of gene sets. HOGs make it possible to compare the genes of a given species with groups of orthologous genes conserved across a taxonomic clade. 

> <hands-on-title>OMArk on extracted protein sequences</hands-on-title>
>
> 1. {% tool [OMArk](toolshed.g2.bx.psu.edu/repos/iuc/omark/omark/0.3.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Protein sequences"*: `gffread: pep.fa`
>    - *"OMAmer database*: select `LUCA-v2.0.0`
>    - In *"Which outputs should be generated"*: select `Detailed summary`
>
{: .hands_on}

The OMArk tool generated an output file in .txt format containing detailed information on the assessment of the completeness, consistency and species composition of the proteome analysed. This report includes statistics on conserved genes, the proportion of duplications, missing genes and the identification of reference lineages.

> <comment-title>What can we deduce from these results?</comment-title>
>
> - Number of conserved HOGs: OMArk has identified a set of 5622 HOGs which are thought to be conserved in the majority of species in the Mucorineae clade.
> - 85.52% of genes are complete, so the annotation is of good quality in terms of genomic completeness.
> - Number of proteins in the whole proteome: 19 299. Of which 62.83% are present and 30.94% of the proteome does not share sufficient similarities with known gene families.
> - No contamination detected.
> - The OMArk analysis is based on the Mucorineae lineage, a more recent and specific clade than that used in the BUSCO assessment, which selected the Mucorales as the reference group.
{: .comment}




# Visualisation with a genome browser

You can visualize the annotation generated using a genomic browser like [JBrowse](https://jbrowse.org/jbrowse1.html). This browser enables you to navigate along the chromosomes of the genome and view the structure of each predicted gene.

> <hands-on-title>JBrowse visualisation</hands-on-title>
>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: `genome_masked.fasta` (Input dataset)
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Annotation`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `gff3` (output of **Helixer** {% icon tool %})
{: .hands_on}

Click on the newly created dataset's eye to display it. You will see a JBrowse genome browser. You can have a look at the [JBrowse tutorial]({% link topics/visualisation/tutorials/jbrowse/tutorial.md %}) for a more in-depth description of JBrowse.

# Conclusion

Congratulations on reaching the end of this tutorial! You now know how to perform a structural annotation of a new eukaryotic genome, using Helixer. And you've learned how to evaluate its quality and how to visualize it using JBrowse.

If you'd like to complete this annotation, we recommend you to follow the tutorial on [functional annotation]({% link topics/genome-annotation/tutorials/functional/tutorial.md %}) with EggNOG Mapper and InterProScan. You can follow it with the protein sequences we generated earlier with gffread.

tools_json:
["Helixer", "toolshed.g2.bx.psu.edu/repos/genouest/helixer/helixer/0.3.3+galaxy1", "Genome Annotation Statistics", "toolshed.g2.bx.psu.edu/view/iuc/jcvi_gff_stats/jcvi_gff_stats/0.8.4", "GFFread", "toolshed.g2.bx.psu.edu/repos/devteam/gffread/gffread/2.2.1.4+galaxy0", "Busco", "toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.5.0+galaxy0", "OMArk", "toolshed.g2.bx.psu.edu/repos/iuc/omark/omark/0.3.0+galaxy2", "JBrowse", "toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1"]


topic: genome-annotation
tutorial: apollo
title: Refining Genome Annotations with Apollo (prokaryotes)
difficulty: intermediate
time_estimation: 3h
requirements_json: ["type: \"internal\"", "type: \"internal\""]
tags_json: ["gmod", "prokaryote", "microgalaxy", "jbrowse1", "biodiversity", "apollo2"]
content_hash: 7ccc4f8efe118d1e3e255bcea72f3dca
last_modified: 2026-05-27T09:01:48.732502
zenodo_link: https://zenodo.org/record/4889110
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/apollo/tutorial.html

content:
After automatically annotating your genome using [Prokka](../annotation-with-prokka/tutorial.html) for example, it is important to visualize your results so you can understand what your organism looks like, and then to manually refine these annotations along with any additional data you might have. This process is most often done as part of a group, smaller organisms may be annotated individually though.

> <warning-title>Only works on UseGalaxy.eu</warning-title>
> Currently this tutorial requires an Apollo server to be deployed by the administrator. This will currently only work on UseGalaxy.eu, hopefully this list will expand in the future.
{: .warning}

[Apollo](https://github.com/gmod/apollo) {% cite Dunn2019 %} provides a platform to do this. It is a web-based, collaborative genome annotation editor. Think of it as "Google Docs" for genome annotation, multiple users can work together simultaneously to curate evidences and annotate a genome.

This demo is inspired by the [Apollo User's Guide](https://genomearchitect.readthedocs.io/en/latest/UsersGuide.html), which provides additional guidance.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data upload

To annotate a genome using Apollo, we need the reference genome sequence in FASTA format, and any evidence tracks we want to refine into our annotations. "Evidence tracks" can be any data like:

- A set of prior gene predictions or other genomic feature predictions
- The output of a bioinformatics analysis like BLAST or InterProScan
- Sequencing reads from RNA-Seq or another HTS analysis
- If you are not doing a *de novo* annotation, then a previous released {OGS}

In this tutorial we have obtained some data from NCBI related to [*Escherichia coli K12 str. MG1655*](https://ecoliwiki.org/colipedia/index.php/Category:Strain:MG1655), and we will visualise this data and use it to make some annotations in order to familiarise you with the process.

> <comment-title>Real Data: Unreal Circumstances</comment-title>
> While the data for this tutorial is sourced from publicly available databases, and is all related to different experiments on *E. coli K12*, this is not necessarily the data *you* might use to annotate your genomes. You probably know best what data you should be using in your own circumstances, for the specific features on which you are focused.
{: .comment}

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 0. Create a new history and give it a good name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 1. Click the upload icon {% icon galaxy-upload %}
>
> 2. Switch to the "Rule-based" tab
>
> 3. Copy & Paste the following table into the Rule-based uploader textbox:
>
>    ```
>    https://zenodo.org/records/4889110/files/augustus.gff3	Augustus	gff3
>    https://zenodo.org/records/4889110/files/blastp_vs_swissprot_2018-01-22.blastxml	Blastp vs swissprot	blastxml
>    https://zenodo.org/records/4889110/files/BWA-MEM_K12_Coverage.bigwig	BWA-MEM K12 Coverage	bigwig
>    https://zenodo.org/records/4889110/files/BWA-MEM_K12_Mapping.bam	BWA-MEM K12 Mapping	bam
>    https://zenodo.org/records/4889110/files/BWA-MEM_O104_Coverage.bigwig	BWA-MEM O104 Coverage	bigwig
>    https://zenodo.org/records/4889110/files/BWA-MEM_O104_Mapping.bam	BWA-MEM O104 Mapping	bam
>    https://zenodo.org/records/4889110/files/E._coli_str_K-12_substr_MG1655_100kb_subset.fasta	Genome	fasta
>    https://zenodo.org/records/4889110/files/K12_Variants.vcf	K12 Variants	vcf
>    https://zenodo.org/records/4889110/files/NCBI_AnnotWriter_Genes.gff3	NCBI AnnotWriter Genes	gff3
>    https://zenodo.org/records/4889110/files/O104_H4_LASTZ_Alignment.bed	O104 H4 LASTZ Alignment	bed
>    https://zenodo.org/records/4889110/files/O104_Variants.vcf	O104 Variants	vcf
>    https://zenodo.org/records/4889110/files/TopHat_SRR1927169_rep1.bam	TopHat SRR1927169 rep1	bam
>    https://zenodo.org/records/4889110/files/TopHat_SRR1927169_rep1_Coverage.bigwig	TopHat SRR1927169 rep1 Coverage	bigwig
>    https://zenodo.org/records/4889110/files/TopHat_SRR1927170_rep2.bam	TopHat SRR1927170 rep2	bam
>    https://zenodo.org/records/4889110/files/TopHat_SRR1927170_rep2_Coverage.bigwig	TopHat SRR1927170 rep2 Coverage	bigwig
>    ```
> 4. Click **Build**
>
> 5. From **Rules** menu select `Add / Modify Column Definitions`
>
>    - Click `Add Definition` button and select `URL`
>      - *"URL"*: `A`
>    - Repeat this again and select `Name` instead.
>      - *"Name"*: `B`
>    - Repeat this again and select `Type` instead.
>      - *"Type"*: `C`
>    - Click `Apply`
>
> 7. Click **Upload**
>
{: .hands_on}

# Using Apollo for Annotation

Refining genome annotations happens in multiple steps:

- Create a JBrowse instance from the reference genome FASTA file and evidence tracks
- Import this data into Apollo
- Refine the annotations
- Export the refined genome annotations

In this tutorial we will focus more on the practical portions than the theoretical part of genome annotation, that are covered in other tutorials. When you've completed this tutorial you should be comfortable manipulating genomic data in Galaxy and Apollo.

> <details-title>Why bother?</details-title>
>
> Automated annotation programs continue to improve, however a simple score may not provide evidence necessary to confirm an accurate prediction.
> Therefore, it is necessary to both visually inspect the results and manually fix any issues with the predictions.
>
> Additionally, many times assemblies are less than perfect or read depth and quality may be insufficient, leading to imperfect automatic annotation.
{: .details}

## Build the JBrowse Instance

Let's begin by building a JBrowse instance with all the data we have for this genome.

> <comment-title>Reduced data</comment-title>
>
> To reduce the size of the data, in this tutorial we will only work on a portion of the *Escherichia coli K12 str. MG1655* genome.
{: .comment}

> <hands-on-title></hands-on-title>
>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: Select the `Genome` fasta file
>    - *"Genetic Code"*: `11. The Bacterial, Archael and Plant Plastid Code`
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Gene Calls`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-files %} *"GFF/GFF3/BED Track Data"*: `Augustus` and `NCBI AnnotWriter Genes`
>                        - In *"JBrowse Styling Options [Advanced]"*
>                            - *"JBrowse style.className"*: `transcript-CDS`
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Sequencing`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BAM Pileups`
>                        - {% icon param-files %} *"BAM Track Data"*: Both BWA-MEM Mappings
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BigWig XY`
>                        - {% icon param-files %} *"BAM Track Data"*: Both of the BWA-MEM Coverage files (**not** the `(as bigwig)` files)
>                        - *"Use XYPlot"*: `Yes`
>                        - *"Show Variance Band"*: `Yes`
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `RNA-Seq`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BAM Pileups`
>                        - {% icon param-files %} *"BAM Track Data"*: Both TopHat Mappings
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BigWig XY`
>                        - {% icon param-files %} *"BAM Track Data"*: Both of the `TopHat ... Coverage` files (**not** the `(as bigwig)` files)
>                        - *"Use XYPlot"*: `Yes`
>                        - *"Show Variance Band"*: `Yes`
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Variation`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `VCF SNPs`
>                        - {% icon param-files %} *"SNP Track Data"*: Both Variants files
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Similarity`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `O104:H4 LASTZ Alignments`
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `Blast XML`
>                        - {% icon param-file %} *"Blast XML Track Data"*: The `blastp` results from swissprot (`Blastp vs swissprot`)
>                        - {% icon param-file %} *"Features used in Blast Search"*: The `NCBI AnnotWriter Genes` file
>                        - *"Minimum Gap Size"*: `3`
>                        - *"Is this a protein blast search?"*: `Yes`
>
>    > <comment-title>JBrowse is highly configurable</comment-title>
>    >
>    > JBrowse is highly configurable, we have set a very basic configuration but there are many more advanced features available to you, if you need them. You can choose precisely how data is displayed, and even what menu options are available when users click on features. If your features have some external identifiers like an NCBI Gene ID, you can even configure JBrowse that when the user clicks on the feature, it should show the gene page for that feature in a new tab. These sort of features are incredibly helpful for building very rich experiences.
>    >
>    > A static genome browser like this (just JBrowse, not in Apollo) is very useful for summarising results of a genomics workflow, where the next step is simply interpretation and not annotation. Have a look at the [JBrowse tutorial]({% link topics/visualisation/tutorials/jbrowse/tutorial.md %}) for more information.
>    >
>    > Currently we have built a standalone genome browser (data + the html page and user interface and javascript), but it's possible to just compile the data directory if you intend to send this data to Apollo, and don't need to view the static data in Galaxy.
>    {: .comment}
{: .hands_on}

This tool will take some time to run dependent on data size. All of the inputs need to be pre-processed by JBrowse into a form that it can render and visualise easily. Once this is complete, you can click on the {% icon galaxy-eye %} eyeball to view the JBrowse instance. This is a static view into the data, JBrowse does not let you make any annotations or save any changes. We will convert it into a dynamic view where we can make persistent annotations and share these with our colleagues.

## Sending data to Apollo

Now that we have a good looking static JBrowse instance, it is time to load it into Apollo to turn it into a dynamic view where you can make modifications to the genes.

> <hands-on-title>Import to Apollo</hands-on-title>
>
> 1. {% tool [Create or Update Organism](toolshed.g2.bx.psu.edu/repos/gga/apollo_create_or_update/create_or_update/4.2.5) %} with the following parameters:
>    - {% icon param-file %} *"JBrowse HTML Output"*: output of **JBrowse** {% icon tool %}
>    - *"Organism Common Name Source"*: `Direct Entry`
>        - *"Organism Common Name"*: `E. coli K12`
>    - *"Genus"*: `Escherichia`
>    - *"Species"*: `coli`
>
> 2. {% tool [Annotate](toolshed.g2.bx.psu.edu/repos/gga/apollo_iframe/iframe/4.2.5) %} with the following parameters:
>    - {% icon param-file %} *"Apollo Organism Listing"*: output of **Create or Update Organism** {% icon tool %}
>
> 3. View {% icon galaxy-eye %} the output of the Annotate tool, when it is ready.
>
{: .hands_on}

Viewing the output will open a view into Apollo in the main panel. Here you can interact with your genome and make annotations. This "Annotate" output is a quick link to that specific genome, and while Apollo allows you to manage and annotate multiple genomes, this dataset will always take you back to that specific genome. You can additionally access the Apollo server outside of Galaxy. While the URL will be different for each Galaxy server that supports Apollo, UseGalaxy.eu's Apollo server is available at [https://usegalaxy.eu/apollo](https://usegalaxy.eu/apollo).


# Apollo


![Screenshot of Apollo in Galaxy, apollo takes up most of the space in the center iframe with the history shown on the right.](../../images/apollo/ui.png "This is the Apollo interface, when viewed from inside Galaxy. The Annotate tool produces a special output which acts like a link to the Apollo server. It is <b>not</b> sufficient to download this file to save a copy of your annotations, unlike other datasets in Galaxy. In the main panel Apollo is displayed, with two panels of its own: the Annotation Window (left), and the Annotator Panel (right). The annotation window is the main view into our genome and here we will see evidence, reconcile it, and make our annotations. The right panel allows us to show or hide individual evidence tracks, switch between organisms, and navigate around the genome.")

From the Apollo user manual:

> The major steps of manual annotation using Apollo can be summarized as follows:
>
> 1. Locate a chromosomal region of interest.
> 2. Determine whether a feature in an existing evidence track provides a reasonable gene model to start annotating.
> 3. Drag the selected feature to the ‘User Annotation’ area, creating an initial gene model.
> 4. Use editing functions to edit the gene model if necessary.
> 5. Check your edited gene model for consistency with existing homologs by exporting the FASTA formatted sequence and searching a protein sequence database, such as UniProt or the NCBI Non Redundant (NR) database, and by conducting preliminary functional assignments using the Gene Ontology (GO) database.
>
{: .quote}

The first four steps are generally the process of structural annotation (the process of identifying the correct gene model), and the last includes functional annotation (the process of assigning a putative function to a gene in your annotations).

## Evidence tracks

Let's start by looking at the tracks available to us, and then turning on the gene call tracks so we can start exploring our data.

> <hands-on-title>Visualize the Gene Calls</hands-on-title>
>
> 1. In the right hand panel at the top click on **Tracks** to open the track listing
>
>    ![Zoomed in screenshot of the track menu with the Gene Calls set of tracks expanded showing Augustus and NCBI AnnotWriter genes.](../../images/apollo/tracks.png)
>
> 2. In the **Gene Calls** group, select the `Augustus` track.
>
>    You can either activate tracks in bulk, by clicking on the checkbox to the right of the group name ("Gene Calls"), or by clicking on the group name to expand the section, and then selecting individual tracks.
>
> 3. Zoom to the first 10kb of the genome.
>
>    1. In the left hand Annotation Window, at the top navigation bar you will find a textbox which shows the current location on the genome.
>    2. Edit this and enter `1..10000`
>    3. Press *Go* or use <kbd>Enter</kbd> on your keyboard.
>
>    ![Screenshot of Jbrowse header with the control panel with magnifying glasses and zoom location.](../../images/apollo/location.png)
>
{: .hands_on}

We can now see an evidence track: `Augustus` is the output of [AUGUSTUS](https://github.com/Gaius-Augustus/Augustus) {% cite Stanke_2008 %}. In a *de novo* annotation project, we probably will only have the outputs of various gene callers, and potentially some expression evidence like RNA-Seq.

We will use the other track in the **Gene Calls** group later in this tutorial, leave it unchecked for now.

## Adding new genes

With the selected track, if you look along the genome, you will see many genes that were predicted by Augustus. Each of them as an unique name assigned by Augustus (e.g. `NC_000913.3.g7.t1`, which means `the 7th gene on the NC_000913.3 chromosome`). If you right click on gene, an select `View details`, you can get access to the coding sequence (CDS).

> <details-title>Gene colors</details-title>
> Each gene color corresponds to an open reading frame on the genome. This allows to quickly see if two genes that are very close are on the same open reading frame, which could mean that they can be merged into a single gene, if other evidences support this.
{: .details}

What we want now is first to check that the structure of the genes predicted by Augustus are in good shape. To do this, we will display an additional evidence tracks.

> <hands-on-title>Display blastp vs swissport track</hands-on-title>
>
> 1. In the right hand panel at the top click on **Tracks** to open the track listing
>
> 2. In the **Similarity** group, select the `Blastp vs swissprot` track.
>
{: .hands_on}

This new track represents the result of aligning sequences from the Swissprot databank along the genome, using Blastp. It should look like that:

![JBrowse with blastp results shown, genes appear as red green and blue stripes with a pointed end, blast results appear as stripes with portions missing where they don't align](../../images/apollo/blastp.png)

Each red box is the alignment of a protein from Swissprot on the genome. You can get more information on the aligned sequence by passing your mouse over the alignement, or clicking on it.

Most of the genes predicted by Augustus look very similar to aligned Swissprot sequences, with the coordinates. Note that our example is an ideal situation that you will probably not see on other genomes: as E. coli is a very studied and well annotated genome, which means Swissprot contains many high quality sequences that can be aligned perfectly by Blastp.

If you look at the region `55000..63000` you will notice however that 2 sequences were aligned on the genome, but Augustus didn't predict any corresponding genes.

![JBrowse showing no genes on the Augustus track, and one blast hit.](../../images/apollo/missing_genes.png)

The blastp alignment looks solid, and the sequences seem to correspond to real proteins references in the litterature. We want to add them to our final annotation, to do it, right click on each gene, and select **Create new annotation** > **gene**. They should appear shortly in the **User-created Annotation** track (yellow background, at the top).

![The two genes are added to the User-created Annotation track](../../images/apollo/added_genes.png)

You can also see that they appear now in the list of genes in the right panel, at the gene and the mRNA level (as a gene can have multiple isoforms, in particular for eukaryotes).

![Gene list showing genes b2g.56.0.0 and b2g.55.0.0 which is expanded to reveal a child mRNA](../../images/apollo/gene_list.png)

The **User-created Annotation** track is where you can make modifications to genes, like changing their coordinates, or their name and functional annotation. If you right click on a gene in this track, you will see all the possibilities offered by apollo.

![Gene contextual menu showing way too many options, the mouse hovers over Open Annotation (alt-click)](../../images/apollo/gene_menu.png)

Currently, the two genes we added have meaningless names. Let's improve that: right click on the leftmost gene, and click on **Open Annotation (alt-click)**.

![Gene details menu in apollo showing type set to gene, the name field has been filled with b2g.55.0.0 and a number of other empty fields like symbol, aliases, description.](../../images/apollo/gene_details.png)

From the blast results, we know that this gene is similar to a Swissprot protein, named `Putative uncharacterized protein YabP`. Let's write this in the **Name** field (type it manually and select it from the drop down list), and `YabP` in the **Symbol** field.

> <details-title>Naming genes</details-title>
>
> Giving a proper name to a gene is not always easy. Should it include "Putative" or not? What if multiple names can apply? Should it be lowercase or uppercase?
> The important thing is to always use the same naming rules when working on a full annotation, and to agree on these rules with other collaborators.
> Usually, big annotation consortiums have naming guidelines that you are supposed to follow.
{: .details}

We have just edited the gene name, but Apollo allows to edit information at the mRNA level. Click on the **Sync name with transcript** button to copy the gene name to the mRNA name. It should now display in the **User-created Annotation** track. To check what you can edit at the mRNA level, just click on the corresponding mRNA in the list above:

![very zoomed screenshot showing clicking on the transcript below the gene to open the mRNA info.](../../images/apollo/mrna_level.png)

You should see `Putative uncharacterized protein YabP` in the **Name** field.

You can do the same for the other gene you created, which is similar to `Uncharacterized protein YabQ` according to Blastp vs Swissprot.

> <comment-title>Saving your work</comment-title>
>
> You do not need to do anything specific to *Save* your work in Apollo. Just like Google Docs, each modification is immediately saved, and any other user working on the same genome will instantly see the changes you make.
{: .comment}

## Editing a gene structure

Apollo allows to edit the whole structure of a gene. If you zoom to the 5' end of `YabP`, you will notice, that a few nucleotides after the start codon, there is another one. Let's change the start of `YabP` to this alternate start codon (even if the blast result suggests that we shouldn't do it, it's an exercise!). To do it, all you need to do is click on the 5' limit of the gene and drag it to the desired position. You will notice that the structure of the gene will be shortly changed.

![Gif showing modifying the 5' limit of a gene by dragging it downstream.](../../images/apollo/change_start.gif)

This kind of modifications is very common when using Apollo, and you can perform it at the gene level, or (for eukaryotes) at the exon/intron level. To guide you doing these changes, you should look at all the tracks available for the genome you study. RNA-Seq track are very helpful to determine the limits of coding sequences on the genome, you can find 2 RNA-Seq libraries in the track list, they were aligned on the genome using TopHat. Other tracks can be helpful, like alignements of transcripts or proteins from closely related species (or even big databanks like Swissprot or NR).

## Viewing and reverting changes

Everything you do in Apollo is tracked in a database. If you right click on the `YabP` gene, and select **Show History**, you have access to the full list of all the actions that were performed on it.

![History of changes on a gene showing many changes to Set Name and a final Set Exon Boundaries change](../../images/apollo/history.png)

When you click on one of the steps, you can see below the list a preview of how the gene looked at the time. And you can revert to a specific version of the gene by clicking on the arrow button on the right.

## Adding more functional annotation

Sometimes you'll want to modify a gene that was predicted by Augustus, just to add functional annotation to it. Navigate to position `3000..5700`, you will see an Augustus gene named **NC_000913.3.g3.t1**. At the same position, there are a few Blastp hits with high scores that correspond to `Threonine synthase`, meaning that the gene found by Augustus is probably an homolog. Let's add this Augustus gene to the **User-created Annotation** track: just drag and drop it there. Now, modify the Name (`Threonine synthase`) and Symbol (`TS`), just as we did earlier, for the gene and mRNA.

If you look at the details of the blast hits, you will notice an identifier looking like that: `gi|11387170|sp|P57289.1|`. In this identifier, `P57289` is the id of an [UniProt record](https://www.uniprot.org/uniprot/P57289). Navigate to this UniProt page and you'll find a lot more details about the protein which was found to be very similar to the gene we are currently annotating. In particular, you can see that 3 [Gene Ontology (GO)](http://www.geneontology.org/) terms are associated with it, in the **GO - Molecular function** and **GO - Biological process** sections. Let's add these terms to our gene with Apollo.


> <comment-title>Gene Ontology (GO) Consortium</comment-title>
>
> The [Gene Ontology Consortium](http://www.geneontology.org/) provides with its Ontology a framework for the model of biology.
> The GO defines concepts/classes used to describe gene function, and relationships between these concepts. It classifies functions along three aspects:
>
>
> molecular function
> :  molecular activities of gene products
>
> cellular component
> :  where gene products are active
>
> biological process
> :  pathways and larger processes made up of the activities of multiple gene products.
>
> [more information](http://geneontology.org/page/ontology-documentation) can be found in the GO website.
>
{: .comment}

With the gene selected, click on the **GO** panel, then click on the **New** button. We will then follow the [GO annotation guidelines](http://geneontology.org/docs/go-annotations/), filling the form like this.

![Adding a GO term with a complex and confusing interface. Aspect is set to MF, all of the parameters are described in the following paragraph.](../../images/apollo/go_form.png)

This form means that our gene will be tagged with the **Go Term** `GO:0030170` (select it in the drop down list while typing `pyridoxal phosphate binding`) from the Molecular Function GO branch (`MF` in **Aspect**). This gene `enables` this `pyridoxal phosphate binding` activity, and we declare it based on the evidence code `ECO:0000250` which means `sequence similarity evidence used in manual assertion` (exactly what we are doing). This similarity is **With** the `UniProt:P57289` record, and we add a **Reference** to this record (we could add a pubmed id to reference a published result for example). Finally we add a **Note** saying that there's a strong similarity.

Before saving, don't forget to click on the two **Add** buttons to save the **With** and **Note** fields.

You can now do the same for the two other GO terms found on the UniProt page: `threonine synthase activity` and `threonine biosynthetic process`. Adding these terms this way allows to save the information in a computing-friendly way, which means other bioinformatics tools will be able to use make this information automatically.

Other tabs are available in the annotation panel, allowing to add **Comments**, external database references (**DbXref**) or **Attributes* to genes, to record more infortmation about them.

## Comparing with the official annotation

In this tutorial we are lucky as we are studying a very well known reference organism. It means we have a [very good quality reference annotation](https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi?mode=Info&id=511145&lvl=3&lin=f&keep=1&srchmode=1&unlock) provided by NCBI. You can display it in Apollo by enabling the corresponding track:

> <hands-on-title>Display NCBI annotation track</hands-on-title>
>
> 1. In the right hand panel at the top click on **Tracks** to open the track listing
>
> 2. In the **Gene Calls** group, select the `NCBI AnnotWriter Genes` track.
>
{: .hands_on}

Now navigate along the genome, and notice the differences between the Augustus annotation, the changes you made in the **User-created Annotation** track, and the reference annotation. This illustrates what happens in real life: you start with an automatic annotation which is not perfect, and using Apollo you improve it and add functional annotation, which allows in the end to release a better annotation to the community.

## Sequence alterations

Until now we have supposed that the quality of the genome sequence is perfect. But you might work on othr genomes where the sequence contains errors (substitutions, insertions, deletions) due to assembly problems for example. In this case, automatic annotation programs will have difficulties producing good looking gene models: there can be frame shifts within genes, or broken start/stop codons which are not detected. In this case, Apollo allows to edit the genome sequence itself.

Navigate to position `42500..44500`, and show the `K12 Variants` track from the `Variation` track group. This track shows variants that were detected in this genome sequence after resequencing it. Drag the `NC_000913.3.g36.t1` gene to the **User-created Annotation** track. Now we will register in Apollo the SNP that was detected on position `43988` (C->T): zoom to this position until you see the 6 open reading frame and the sequence. Then right click on the `C` nucleotide and select **Create Genomic Substitution**.

![Right click menu showing options to toggle strands and create genome insertions, deletions, and substitutions. The mouse hovers over substitution.](../../images/apollo/alteration_menu.png)

Fill the form like this, and the SNP will be saved:

![SNP menu with +strand and -strand (seems to be auto-filled out), and a mandatory comment field set to SNP in K12](../../images/apollo/substitution.png)

You can add other types of alterations like insertions or deletions.

Once you have added some alterations, Apollo will automatically display the effect it has on the overlapping genes.

# Exporting and collaborating

{% include topics/genome-annotation/tutorials/apollo/collab.md eukaryote=false %}

# Conclusion


Congratulations, you finished this tutorial! By using Apollo and JBrowse, you learned how to manually refine predicted annotations and export them to Galaxy for future analyses. You also learn how to give access to your project at any other researcher, making it a real collaborative solution.

A [similar tutorial for eukaryote genomes]({% link topics/genome-annotation/tutorials/apollo-euk/tutorial.md %}) exists, using different types of evidence tracks, feel free to have a look at it to learn more.

When refinement is sufficient an updated or new version of the annotation may be exported as GFF3 as well as published as a new JBrowse directory for inspection.

# What's next?

After generating your refined annotation, you'll want to merge it back into the official gene sets. A future tutorial will show you how to do it within Galaxy.

If a *de novo* set, you can export it as GFF3 and load it into a tool like [Tripal](http://tripal.info) to provide visualization.

tools_json:
["JBrowse", "toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1", "Create or Update Organism", "toolshed.g2.bx.psu.edu/repos/gga/apollo_create_or_update/create_or_update/4.2.5", "Annotate", "toolshed.g2.bx.psu.edu/repos/gga/apollo_iframe/iframe/4.2.5"]


topic: genome-annotation
tutorial: tnseq
title: Essential genes detection with Transposon insertion sequencing
difficulty: intermediate
time_estimation: 7H
tools_json: []
requirements_json: []
tags_json: ["bacteria", "tnseq", "essential genes", "microgalaxy"]
content_hash: 846b9717f469e14799f92a8e535a73ca
last_modified: 2026-05-15T22:28:13.517009
zenodo_link: https://doi.org/10.5281/zenodo.2579335
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/tnseq/tutorial.html

content:
In microbiology, identifying links between genotype and phenotype is key to understand bacteria growth and virulence mechanisms, and to identify targets for drugs and vaccines. These analysis are limited by the lack of bacterial genome annotations (*e.g.* 30% of genes for *S. pneumoniae* are of unknown function) and by the fact that genotypes often arose from complex composant interactions.

## Transposon insertion Sequencing

Transposon insertion sequencing is a technique used to functionally annotate bacterial genomes. In this technique, the genome is saturated by insertions of transposons. Transposons are highly regulated, discrete DNA segments that can relocate within the genome. They have a large influence on gene expression and can be used to determine the function of genes.

When a transposon inserts itself in a gene, the gene's function will be disrupted, affecting the fitness (growth) of the bacteria. We can then manipulate transposons for use in insertional mutagenesis, *i.e.* creation of mutations of DNA by the addition of transposons. The genomes can be then sequenced to locate the transposon insertion site and the function affected by a transposon insertion can be linked to the disrupted gene.

![Illustration of tnseq Method](../../images/tnseq/principle_tnseq.png "Transposon insertion sequencing method (from <a href='#Chao2016'> Chao <i>et al.</i> 2016</a>)")

1. **Data production** (**a** in the previous image)

    An initial population of genomes is mutated so that the genome of the bacteria is saturated with transposon insertions. A library is called *saturated* if in the genomes across the whole population of bacteria, each potential insertion site has at least one insertion. The population is then divided into several media containing different growth conditions, to identify the impact of the insertions on the bacteria growth. After growth, the regions flanking the insertion are amplified and sequenced, allowing to determine the location of the insertion.

2. **Analysis** (**b** in the previous image)

    The sequences are aligned to the reference genome to identify the location of the regions flanking the insertions. The resulting data will show a discrete repartition of reads on each site. If a gene present several insertions, like the two leftmost genes in *Condition A*, it means that its disruption has little or no impact to the bacterial growth. On the other hand, when a gene shows no insertions at all, like the rightmost gene in *Condition A*, is means that any disruption in this gene killed the bacteria, meaning it is a gene essential to bacteria survival. If the library is sufficiently saturated, there is a clear threshold between essential and non-essential genes when you analyze the insertion rate per gene.

Two type of transposon insertion methods exist:
- Gene disruption, where we analyze only the disruptions. This will covered by of this tutorial
- Regulatory element insertion, where different promoters are inserted by the transposon, and we analyze the change in gene expression in addition to the disruption. This method will be the subject of another tutorial.

## Building a TnSeq library


Different types of transposons can be used depending of the goal of your analysis.

- Randomly pooled transposon
    - Mariner-based transposons, common and stable transposons which target the "TA" dinucleotides

        The TA are distributed relatively evenly along genome. The Mariner-based transposons can be inserted to impact statistically every gene, with in average more than 30 insertions site per kb. With the low insertion bias, it is easy to build saturated libraries. But local variations means less loci and less statistical power.

    - Tn5-based vectors, which insert at random sites

        This transposon do not require any target sequences. It is useful for specie where it is difficult to build mariner based transposons. But it has a preference for high GC content, causing insertion bias

- Defined sequence transposon

    It can be used to study interactions in pathways of interest, but also more precise targeting (small genes, pathways) for specific analyses

Independently of the transposon choice we need to be careful about the library complexity. With a large complex library, multiple insertion can be found in every potential locus. The higher density of insertion, the greater precision in identifying limits of regions of interest. If the density of the library is too low, some genes might by chance not be disrupted and mistaken for essential. The advantage of a target specific transposon, like the mariner, in opposition of a Tn5-based transposon inserting randomly, it that the limited number of insertion sites makes it easier to build high complexity libraries.

After selection of the type of transposon, we need to modify it to allow insertion site amplification and sequencing to get a library fitting the transposon insertion. Biases could be introduced during the process due to uneven fragment sizes. To avoid that, we can introduce a Type I restriction site to cleave DNA downstream of transposon, and get uniform fragment sizes and therefore avoid a bias in the representation of the insertions.

As we just want to identify the TA site affected by an insertion, we only need the location of the start of the reads and not a good coverage of the entire genome. Long reads are then not so important. On the other hand, a minimum transposon length of 16 bp is necessary for precise mapping on the genome {% cite Kwon2015 %}. We can therefore not use the BsmFI restriction site (11 to 12 bp) but MmeI.

**In this tutorial, we are using mariner transposon targeting TA sequences, in ordered to target the whole genome uniformly,** with two specific regions used to specifically sequence the region upstream of the insertion {% cite Santiago2015 %}

![Structure of the transposon containing several parcodes and adapters](../../images/tnseq/tranposon_structure.png "Structure of the transposon containing several parcodes and adapters (from <a href='#Santiago2015'> Santiago <i>et al.</i> 2015</a>)")

The transposon inserts itself at TA site at the ITR junctions. These ITR junctions have been modified to include a Mme1 restriction site and a NotI restriction site which cut 21 bp upstream the restriction site. These two site are the 5' and 3' limits to the genomic DNA we want to sequence.

1. After digestion by NotI restriction enzyme, the fragments are attached to biotinylated adaptors that link to NotI restriction site. The attached fragment are then digested by MMeI at a site upstream, where an Illumina primer is then linked. The sequencing is then done, adding Illumina adaptors and an additional barcode to the read for multiplexed sequencing.

2. An insertion can sometimes be composed of one or more copies of the transposon (multimer). There is therefore a risk to select plasmid backbone sequence. To solve this problem, an additional NotI has been added in the backbone to create different length construct, that can later be filtrated. Different promoters are added with an additional 3 bp barcode to analyze differential expression impact. This type of complex analysis will be covered in a follow-up tutorial.

Because of this complex transposon structure, the reads obtained after sequencing contain a lot of adapters and foreign sequences used to insert and target the transposon. Several steps of preprocessing are then need to extract only the transposon sequence before finding its location on the genome.

## Tnseq analysis


Once the genomic sequences are extracted from the initial reads (*i.e.* remove non genomic sequences from the reads), they need to be located each on the genome to link them to a TA site and genes. To do that we map them to a reference genome, link them to a specific insertion site, and then count the number of insertion for each TA site and identify essential genes of regions.

We will apply this approach in this tutorial using a subset of TnSeq reads from {% cite Santiago2015 %}.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data upload

Let's start with uploading the data.

> <hands-on-title>Import the data</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import from [Zenodo](https://zenodo.org/record/2579335) or a data library (ask your instructor):
>   - FASTQ file with the Tnseq reads: `Tnseq-Tutorial-reads.fastqsanger.gz`
>   - Set of barcodes to separate reads from different experimental conditions: `condition_barcodes.fasta`
>   - Set of barcodes to separate reads from different transposon constructs: `construct_barcodes.fasta`
>   - Genome file for *Staphylococcus aureus*: `staph_aur.fasta`
>   - Annotation file for *Staphylococcus aureus*: `staph_aur.gff3`
>
>    ```
>    https://zenodo.org/record/2579335/files/Tnseq-Tutorial-reads.fastqsanger.gz
>    https://zenodo.org/record/2579335/files/condition_barcodes.fasta
>    https://zenodo.org/record/2579335/files/construct_barcodes.fasta
>    https://zenodo.org/record/2579335/files/staph_aur.fasta
>    https://zenodo.org/record/2579335/files/staph_aur.gff3
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
> 3. Rename the files
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}


# Remove all non genomic sequences from the sequenced reads

Because of the experimental design for transposon insertion sequencing, the raw reads contain a lot of adapters and foreign sequences used to insert and target the transposon. To obtain the core reads that contain only genomic sequence, the reads have to go through several steps (using the **Cutadapt** tool) to remove them and divide the reads per experimental condition and type of transposon:

![Workflow of data pre-processing](../../images/tnseq/preprocessing.png "Data pre-processing")

1. We separate the reads of each experimental condition based on a 8 bp barcode at the beginning of each read. These barcodes were added to be able to pool different conditions together before the transposon insertion and sequencing.
2. The tail of each set of read is then removed. It immediately follows the 3 bp barcode specific to transposon constructs, and contains illumina adapter sequence and downstream
3. To be sure all our reads have been trimmed correctly we filter out the reads too large.
4. We then separate the reads per transposon construct
5. We remove the remaining transposon sequence containing MmeI.

## Separate reads by experimental conditions

First we divide the initial data set by experimental conditions using the 8 bp barcode added during the Illumina multiplexing protocol.

> <hands-on-title>Inspect condition barcodes</hands-on-title>
>
> 1. Inspect the `condition_barcodes` file
>
{: .hands_on}

This fasta file contains the barcodes for each condition:

```
>control
^CTCAGAAG
>condition
^GACGTCAT
```

We can see 2 barcodes there: one for control and one for condition.

> <comment-title>Barcode symbols used by <b>Cutadapt</b></comment-title>
>
> The `^` at the beginning of the sequence means we want to anchor the barcode at the beginning of the read. To know more about the symbols used by **Cutadapt**, you check the [**Cutadapt** manual](https://cutadapt.readthedocs.io/en/stable/guide.html#adapter-types).
{: .comment}

We would like now to split our Tnseq reads in `Tnseq-Tutorial-reads` given the barcodes.

> <hands-on-title>Split reads by condition barcodes using Cutadapt</hands-on-title>
>
> 1. **Cutadapt** {% icon tool %} with:
>     - *"Single-end or Paired-end reads?"*: `Single-end`
>       - {% icon param-file %} *"FASTQ/A file"*: `Tnseq-Tutorial-reads`
>       - In *"Read 1 Options"*
>         - In *"5' (Front) Adapters"*
>           - Click on *"Insert 5' (Front) Adapters"*
>             - *"Source"*: `File From History`
>               - *"Choose file containing 5' adapters"*: `condition_barcodes` file
>     - In *"Adapter Options"*
>       - *"Maximum error rate"*: `0.15` (to allow 1 mismatch)
>       - *"Match times"*: `3` (to cover cases where barcodes are attached several times)
>     - In *"Output Selector"*, select
>       -  *"Report"*
>       -  *"Multiple output"* (to separate the reads into one file per condition)
>       -  *"Untrimmed reads"* (to write reads that do not contain the adapter to a separate file)
>
>     The output is a collection of the different conditions datasets, here control and condition, and a report text file.
>
> 2. Inspect the report text generated by **Cutadapt** {% icon tool %}
{: .hands_on}

> <question-title></question-title>
>
> 1. What percentage of reads has been trimmed for the adapter?
> 2. How many reads have been trimmed for each condition?
> 3. What should we change if our barcodes were at the end of the reads ?
>
> > <solution-title></solution-title>
> >
> > 1. 100% of the reads were trimmed (line `Reads with adapters`)
> > 2. 2,847,018 for control (`CTCAGAAG` barcode) and 2,862,108 for condition (`GACGTCAT` barcode)
> > 3. We should use the option *"Insert 3' (End) Adapters"* and anchored them at the end of the read with the symbol `$` in our fasta file containing the barcodes.
> {: .solution }
>
{: .question}


## Remove Adapter sequence

Our reads are now divided by condition. We need to trim their tail containing the Illumina adapters, used for initiating the sequencing (`CGTTATGGCACGC`, here). To do so, we remove the adapter and everything downstream, using the end adapter option of **Cutadapt** and not anchor the sequence anywhere. To eliminate reads that might not have been trimmed because of too many mismatches or other reasons, we filter the reads by size, given the known approximate size of the remaining sequences (**how is it computed??**).

> <hands-on-title>Remove Adapter with Cutadapt</hands-on-title>
>
> 1. **Cutadapt** {% icon tool %} to remove adapters with:
>    - *"Single-end or Paired-end reads?"*: `Single-end`
>      - {% icon param-collection %} *"FASTQ/A file"*: collection output of the previous **Cutadapt**
>      - In *"Read 1 Options"*
>        - In *"3' (End) Adapters"*
>          - Click on *"Insert 3' (End) Adapters"*
>            - *"Source"*: `Enter custom Sequence`
>              - *"Enter custom 3' adapter sequence"*: `CGTTATGGCACGC`
>    - In *"Adapter Options"*
>      - *"Match times"*: `3` (to cover cases where barcodes are attached several times)
>    - In *"Output Selector"*, select
>      -  *"Report"*
>
>    > <question-title></question-title>
>    > What are the outputs at this step?
>    > > <solution-title></solution-title>
>    > > The outputs are two collections: one containing the reads in both conditions, and one containing the **Cutadapt** reports for each condition.
>    > {: .solution }
>    {: .question}
>
> 2. Inspect the generated report files
>
>    > <question-title></question-title>
>    > What percentage of the reads contained the adapter?
>    > > <solution-title></solution-title>
>    > > More than 99% of the reads contained the adapter in both conditions (line `Reads with adapters` in the reports)
>    > {: .solution }
>    {: .question}
>
> 3. **Cutadapt** {% icon tool %} to filter reads based on length with:
>    - *"Single-end or Paired-end reads?"*: `Single-end`
>      - {% icon param-collection %} *"FASTQ/A file"*: collection output of the previous **Cutadapt**
>    - In *"Filter Options"*
>      - *"Minimum length"* : `64`
>      - *"Maximum length"* : `70`
>    - In *"Output Selector"*, select
>      -  *"Report"*
>
> 4. Inspect the generated report files
>
>    > <question-title></question-title>
>    > How many reads where discarded after filtering?
>    > > <solution-title></solution-title>
>    > > Less than 2% of the reads were discarded in both conditions (lines `Reads that were too short` and `Reads that were too long` in the reports)
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

We can see that is both samples the reads have pass the filtering at more than 98%. If this percentage is very low, it means the previous trimming steps is incomplete or faulty.

## Separate reads from different transposon constructs

The constructs used in this experiment contain different strengths and directions of promoters. We use the different constructs as replicates, so we need now to separate the reads based on the construct specific 3 bp barcodes.

> <comment-title></comment-title>
> In addition of disrupting a gene at the location of the insertion, such constructs can modify the expression of either upstream or downstream regions. The analysis of such modification will be studied in another training material, but for now we consider that the construct does not impact the essentiality analysis.
{: .comment}

> <hands-on-title>Inspect construct barcodes</hands-on-title>
>
> 1. Inspect the `construct_barcodes` file
>
>    > <question-title></question-title>
>    >
>    > What does the `$` mean in the barcode sequence file?
>    >
>    > > <solution-title></solution-title>
>    > > It means the barcode is anchored at the end of the reads.
>    > {: .solution}
>    {: .question}
{: .hands_on}

> <hands-on-title>Barcode split with Cutadapt</hands-on-title>
>
> 1. **Cutadapt** {% icon tool %} with:
>    - *"Single-end or Paired-end reads?"*: `Single-end`
>      - {% icon param-collection %} *"FASTQ/A file"*: collection output of the previous **Cutadapt**
>      - In *"Read 1 Options"*
>        - In *"3' (End) Adapters"*
>          - Click on *"Insert 3' (End) Adapters"*
>            - *"Source"* : `File From History`
>               - {% icon param-file %} *"Choose file containing 5' adapters"*: `construct_barcodes` file
>    - In *"Adapter Options"*
>       - *"Match times"*: `3` (to cover cases where barcodes are attached several times)
>    - In *"Output Selector"*, select
>       -  *"Report"*
>       - *"Multiple output"* (to separate the reads into one file per condition)
>
> 2. Inspect the report files
>
>    > <question-title></question-title>
>    >
>    > Are the reads equally divided between constructs?
>    >
>    > > <solution-title></solution-title>
>    > > When you look at the reports, you can see that most of the reads have been assigned to the *blunt* construct (`TAC` barcode): the blunt construct is the control and does not contain any promoters. This means that there is less negative selective pressure on blunt than the other ones, that have affected flanking region in addition to the disrupted gene at the insertion site. This won't be a problem here as the tool we use for the essentiality prediction consider the sum of reads in the replicates.
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

You can notice that the output of this split is a *nested collection*, a collection of collection.

## Remove the remaining transposon sequence

The last remaining transposon sequence in the reads is the linker with the MmeI restriction site (`ACAGGTTGGATGATAAGTCCCCGGTCTATATTGAGAGTAACTACATTT`).

> <hands-on-title> Remove Linker with Cutadapt</hands-on-title>
>
> 1. **Cutadapt** {% icon tool %} with:
>    - *"Single-end or Paired-end reads?"*: `Single-end`
>      - {% icon param-collection %} *"FASTQ/A file"*: collection output of the previous **Cutadapt**
>      - In *"Read 1 Options"*
>        - In *"3' (End) Adapters"*
>          - Click on *"Insert 3' (End) Adapters"*
>            - *"Source"*: `Enter custom Sequence`
>              - *"Enter custom 3' adapter sequence"*: `ACAGGTTGGATGATAAGTCCCCGGTCTATATTGAGAGTAACTACATTT`
>    - In *"Adapter Options"*
>      - *"Maximum error rate"*: `0.15`
>    - In *"Output Selector"*, select
>      -  *"Report"*
>
> 2. Inspect the report files and check that the majority of the read have been trimmed
{: .hands_on}

Now that we isolated the genomic sequences from the initial reads, we want to align them to count how many insertion have been retained at each TA site.

# Count the number of insertion per TA sites

## Align the reads to a reference genome

To identify the location of each TA site to the count them, the first step is to map the reads on the reference genome. We use the **Bowtie**.

> <comment-title></comment-title>
> We could also use **Bowtie2** with an end-to-end option, but **Bowtie** is more suitable for very short reads like ours (16-17 bp).
{: .comment}


> <hands-on-title>Map reads with Bowtie</hands-on-title>
>
> 1. **Map with Bowtie for Illumina** {% icon tool %} with:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use one from the history`
>      - {% icon param-file %} *"Select the reference genome"*: `staph_aur.fasta` file.
>    - *"Is this library mate-paired?"*: `Single-end`
>      - {% icon param-collection %} *"FASTQ file"*: collection output of the last **Cutadapt**
>      - *"Bowtie settings to use"*: `Full parameters list`
>      - *"Skip the first n reads (-s)"*: `0`
>      - *"Maximum number of mismatches permitted in the seed (-n)"*: `0`
>      - *"Seed length (-l)"*: `17`
>      - *"Whether or not to make Bowtie guarantee that reported singleton alignments are 'best' in terms of stratum and in terms of the quality values at the mismatched positions (--best)"*: `Use best`
>      - *"Whether or not to try as hard as possible to find valid alignments when they exist (-y)"* : `Try Hard`
>
>    > <question-title></question-title>
>    > Why are we strictly enforcing no mismatch mapping?
>    > > <solution-title></solution-title>
>    > > Our reads being very short, the smallest size needs precise mapping, allowing even one mismatch would risk having reads mapping in wrong positions.
>    > {: .solution }
>    {: .question}
>
> 2. Rename your collection for better clarity
>
>    {% snippet faqs/galaxy/collections_rename.md %}
>
{: .hands_on}

## Compute coverage of the genome

We have now our reads mapped on the reference genome. We can compute the genome coverage, *i.e.* how much the nucleotides of the genome are covered by reads. This information will be later crossed with our TA sites position.

In our case, the sequenced reads cover the  5' region flanking region of the TA site where the transposon inserted. We should not then extract the coverage across the whole reads, as it could cover several TA sites, but only the coverage at the 3' end of the read:

![A read align to the genome with its 3' end covering half of the TA site](../../images/tnseq/Map_cov.png "Mapping read and TA site coverage")

> <hands-on-title>Compute genome coverage</hands-on-title>
>
> 1. **bamCoverage** {% icon tool %} with:
>    - {% icon param-collection %} *"BAM/CRAM file"*: collection output of the last **Map with Bowtie for Illumina**
>    - *"Bin size in bases"*: `1`
>    - *"Scaling/Normalization method"*: `Do not normalize or scale`
>    - *"Coverage file format"*: `bedgraph`
>    - *"Show advanced options"*: `yes`
>      - *"Ignore missing data?"*: `Yes` (to get only region with read counts)
>      - *"Offset inside each alignment to use for the signal location"*: `-1` (to read the signal of the coverage only at the 3' end of the read)
>
{: .hands_on}

## Identify TA sites positions

In order to get the coverage on each TA site we need to prepare a file containing the position of each TA site.

As you can see on the [previous figure](#figure-4), the read can cover one side or the other of the TA depending on the direction of insertion of the transposon. Depending on the direction of insertion the coverage will be counted on the leftmost position of the TA site or on the rightmost. As we are not considering strand separately in this analyses, we will consider both count as attached to the leftmost base of the TA site. To do that we will create two list of TA site positions, listing the 5' end of each TA site for forward and reverse strand, and then merge them to get a global count per TA site.

> <hands-on-title>Get TA sites coordinates</hands-on-title>
>
> 1. **Nucleotide subsequence search** {% icon tool %} with:
>    - {% icon param-file %} *Nucleotide sequence to be searched"*: `staph_aur.fasta`
>    - *"Search with a"*: `user defined pattern`
>    - *"Search pattern"*: `TA`
>    - *"Search pattern on"*: `forward strand`
> 4. Inspect the generated file
{: .hands_on}

**Nucleotide subsequence search** generate one BED file with the coordinates of each TA on the genome:
1. Chrom
2. Start
3. End
4. Name
5. Strand

The only information we need here are the positions of the 5' end of each TA site for each strand.

> <hands-on-title>Get forward and reverse strand positions</hands-on-title>
>
>
> 1. **Cut columns from a table** {% icon tool %} with:
>    - Cut columns : `c2`
>    - *"Delimited by"*: `tab`
>    - {% icon param-file %} *"From"*: output of **Nucleotide subsequence search**
>
> 4. Add the `#forward` tag to the output
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
> 2. **Cut columns from a table** {% icon tool %} with:
>    - Cut columns : `c1,c3`
>    - *"Delimited by"*: `tab`
>    - {% icon param-file %} *"From"*: output of **Nucleotide subsequence search**
>
> 3. **Compute an expression on every row** {% icon tool %} to shift the end position of TA sites  by 1 and get the start of the TA site on the reverse strand with:
>    - *Add expression"*: `c2-1`
>    - {% icon param-file %} *"as a new column to"*: output of **cut**
>    - *"Round result?"*: `yes`
>
>
> 4. **Cut columns from a table** {% icon tool %} with:
>    - Cut columns : `c3`
>    - *"Delimited by"*: `tab`
>    - {% icon param-file %} *"From"*: output of **Compute**
>
> 8. Add the `#reverse` tag to the output
>
{: .hands_on}


## Merge overall coverage and positions of TA sites to get the coverage of each TA sites

We have now 2 files containing the coordinates of our TA sites for both strands. We should cross them with the coverage files to get the coverage on both sides of each TA site.

> <hands-on-title> Get coverage of TA sites</hands-on-title>
>
> 1. **Join two files** {% icon tool %} with:
>    - {% icon param-collection %} *1st file"*: output of **bamCoverage**
>    - *"Column to use from 1st file"*: `Column 2` (to select the position of the end of the read)
>    - {% icon param-file %} *"2nd File"*: output of last **cut** with the `forward` tags
>    - *"Column to use from 2nd file"*: `Column 1`
>    - *"Output lines appearing in"*: `Both 1st & 2nd file, plus unpairable lines from 2st file. (-a 2)` (to get all TA sites)
>    - *"First line is a header line"*: `No`
>    - *"Value to put in unpaired (empty) fields"*: `0` (to assign a count of 0 to TA sites not covered in the bamCoverage file)
>
> 2. Add `#forward` tag to the output collections
>
>    {% snippet faqs/galaxy/collections_add_tag.md %}
>
> 3. **Cut columns from a table** {% icon tool %} with:
>    - Cut columns : `c1,c4`
>    - *"Delimited by"*: `tab`
>    - {% icon param-file %} *"From"*: output of**Join two files** with the `forward` tag
>
>
> 5. Repeat the same steps for `reverse`
>
{: .hands_on}

We now have a read count for each nucleotide at the different TA sites. The insertions on the forward strand files are in a different direction than those on the reverse strand file. In our case, we are only interested in studying the gene disruption, and therefore we do not want to consider the directions separately. We can then combine the forward and reverse counts together to get a total count per TA site. In order to do that we need to assign the count of both strand to the same nucleotide, by shifting by one position the count on the reverse strand.

> <hands-on-title>Get total count per TA site</hands-on-title>
>
> 1. **Compute an expression on every row** {% icon tool %} to shift the positions of reverse strand counts with:
>    - *Add expression"*: `c1-1` (to shift the position by 1)
>    - {% icon param-collection %} *"as a new column to"*: **Cut** output collection with the tags `reverse`
>    - *"Round result?"* : `yes`
>
> 2. **Cut columns from a table** {% icon tool %} with:
>    - *"Cut columns"*: `c3,c2`
>    - *"Delimited by"*: `tab`
>    - {% icon param-collection %} *File to cut"*: output of the previous **Compute**
>
> 3. **Sort** {% icon tool %} with:
>    - {% icon param-collection %} *"Sort Query"*: the newest `reverse` collection
>    - *"Number of header lines"*: `0`
>    - In *"Column selections"*
>      - *"on column"*: `Column: 1` (to sort on positions)
>      - *"in"*: `Ascending Order`
>      - *"Flavor"*: `Fast numeric sort (-n)`
>
> 4. Repeat the **Sort** with the newest `forward` collection
>
> 6. **Join two files** {% icon tool %} to merge the two collections and get the forward and reverse count in two different columns with:
>    - {% icon param-collection %} *1st file"* : the `forward` collection from **Sort**
>    - *"Column to use from 1st file"*: `Column 1` to join on positions
>    - {% icon param-collection %} *"2nd File"* : the `reverse` collection from **Sort**
>    - *"Column to use from 2nd file"*: `Column 1`
>    - *"Output lines appearing in"*: `Both 1st & 2nd file` (to get all TA sites)
>    - *"First line is a header line"*: `No`
>    - *"Value to put in unpaired (empty) fields"*: `.`
>
> 7. **Compute an expression on every row** {% icon tool %} to get the total read count with:
>    - *Add expression"*: `c2+c3` to get the total count
>    - {% icon param-collection %} *"as a new column to"*: output of **Join two files** with the tags `reverse` and `forward`
>    - *"Round result?"*: `yes`
>
>
> 2. **Cut columns from a table** {% icon tool %} with:
>    - *"Cut columns"*: `c1,c4`
>    - *"Delimited by"*: `tab`
>    - {% icon param-collection %} *File to cut"*:output of the last **Compute**
>
> 9. **Sort data in ascending or descending order** {% icon tool %} with:
>    - {% icon param-collection %} *Sort Query"*: output of the last **Cut**
>    - *"Number of header lines"*: `0`
>    - In *"Column selections"*
>      - *"on column"*: `Column: 1` (to sort on positions)
>      - *"in"*: `Ascending Order`
>      - *"Flavor"*: `Fast numeric sort (-n)`
>
{: .hands_on}

> <question-title></question-title>
>
> Take a look at the output files:
>
> 1. What are the data contained in the two columns?
> 2. What do the 5 first lines of the blunt sample in the treatment with the condition look like?
>
> > <solution-title></solution-title>
> > 1. The first column contain the position of the TA site, and the second column the number of insertions observed.
> > 2. Your five first lines should look like this :
> >     - One column with the positions : `4,10,16,42,79`
> >     - One column with the number of insertions : `0,3,20,0,11`
> >
> > If your counts are only zeros in one of the blunt sample, you might have made a mistake in merging the counts to the TA sites.
> >
> {: .solution }
{: .question}


# Predicting Essential Genes with Transit

Now that we have the counts of insertions per TA site, we can use them to predict gene essentiality. Several methods exist to identify essential genes of regions. They can be divided in two major categories:

![Different types of TnSeq Analyses](../../images/tnseq/type_of_analyses.png "Methods of TnSeq Analyses (from <a href='#Chao2016'> Chao <i>et al.</i> 2016</a>)")

1. **Annotation dependent method**

    The total read count an/or percentage of disrupted site are computed per annotated regions. The values are then compared to the rest of the genome to classify the genes into the categories *essential* or *non-essential*.

2. **Annotation independent method**

    The total read count and/or disrupted sites are computed independently of annotated regions. One of these methods is using a sliding window. Each window is then classified into the categories *essential* or *non-essential*. After the windows have been classified, they are linked annotations, and the genes/regions can be classified as *essential*, *non-essential*, or *domain essential* according to the classification of the windows they cover. The same classification can be done using HMM based methods instead of sliding windows. In that case, each insertion site will be predicted as *essential* or *non essential* {% cite Chao2016 %}).

To predict the essential genes in our datasets, we will use the Transit tool {% cite dejesus2015transit %}.

## Transit

Transit is a software that can be used to analyse TnSeq Data. It is compatible with Mariner and Tn5 transposon. In total, 3 methods are available to assess gene essentiality in one sample.

### Gumbel method

The Gumbel method performs a gene by gene analysis of essentiality for Mariner data based on the longest consecutive sequence of TA site without insertions in a gene. This allows to identify essential domains regardless of insertion at other location of the gene.

The distribution governing the longest sequence of consecutive non insertion is  characterized by a Gumbel distribution. This distribution is often used to model the distribution of maximums (or minimums) of a variable (See the [Wikipedia article on Gumbel distribution ](https://en.wikipedia.org/wiki/Gumbel_distribution)). The Gumbel distribution is used as a likelihood of non-essentiality, as the longest run of non-insertions should follow what is expected given the global distribution of non-insertion runs. The essential genes, whose runs of empty sites are longer than expected, are modeled through a normalized sigmoid function. This function reflect that any gene can be essential, except the gene were the run of non insertions are too smell to represent a domain.

The total distribution of the maximum run of non-insertion per gene is therefore represented y a bimodal distribution composed of a Gumbel distribution of non-essential genes and a normalized sigmoid function of essential ones. The parameters of these distributions are estimated through a Metropolis–Hastings (MH) sampling procedure {% cite dejesus2013bayesian %}).

Using these two distribution, the posterior probability of each distribution is calculated for each gene. Some genes can be classified as "Unclear" if one probability is not winning over the other. Some other can be classified as "Small" if the space of TA sites covered by the gene is insufficient to categorize it (See the [Transit Manual for the Gumbel method](https://transit.readthedocs.io/en/latest/transit_methods.html#gumbel)).

### HMM method

The HMM method performs a whole genome essentiality analysis for Mariner data. This approach uses the clustering of TA sites along the genome to identify essential regions, and then apply results to the annotation to identify genes containing essential regions. The HMM method provides a classification for each TA site into 4 states : Essential, Non-Essential, Growth Advantage, and Growth Defect.

This method require a well-saturated library and is sensitive to sparse datasets {% cite dejesus2013hidden %}.

### Tn5Gaps method

The Tn5Gaps method is a method dedicated to the identification of essential genes in studies using Tn5 transposons. The analysis is performed on the whole genome to identify regions of non insertion overlapping with genes. It is based on a Gumbel analysis method {% cite griffin2011high %} and adapted to Tn5 transposon specificity. The main difference comes from the fact that Tn5 transposon can insert everywhere, thus
creating libraries with lower insertion rates. The difference from the Gumbel method described above is that the run of non insertion are computer on the whole genome instead of individual genes. The longest run of non insertion considered is not the longest within the gene, but the longest one overlapping the gene.


## Predict the essentiality of genes

In our case, we will use the Gumbel method, because we are using Mariner data and our data look pretty sparse. Gumbel is therefore the safe choice compared to HMM which is more sensitive to sparse data.


In order to use transit, we need to create a an annotation file in the `prot_table` format, specific to the Transit tool. It can be created this file from a GFF3 from GenBank like the one we uploaded earlier.

> <hands-on-title>Hands-on : Create annotation file in prot_table format</hands-on-title>
> 1. Check that the format of `staph_aur.gff3` file is `gff3` and not `gff`
> 2. Change the datatype if needed
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="gff3" %}
>
> 3. **Convert GFF3 to prot_table for TRANSIT** {% icon tool %} with:
>    - {% icon param-file %} *GenBank GFF file"*: the `staph_aur.gff3` file
>
{: .hands_on}


Now that we have prepared the annotation file, we can use the count per TA site to predict essential genes using Transit tool, with some customized parameters:

- The number of sites with a single read could be artefactual. We ignore then counts lower than 2
- A disrupted site that would be very close to the border of the gene may not actually disturb the gene function, and therefore not be an actual signal of disruption. We ignore then insertion near the extremities of the genes.

> <hands-on-title>Predict gene essentiality with Transit</hands-on-title>
>
> 1. **Gumbel** {% icon tool %} with:
>    - *"Operation mode"*: `Replicates`
>      - {% icon param-collection %} *"Input .wig files"*: output of the latest **Sort**
>    - {% icon param-file %} *"Input annotation"*: output of the latest **Convert**
>    - *"Smallest read-count to consider"*: `2` (to ignore single count insertions)
>    - *"Ignore TAs occurring at given fraction of the N terminus."*: `0.1` (to ignore 10% of the insertion at the N-terminus extremity of the gene)
>    - *"Ignore TAs occurring at given fraction of the C terminus."*: `0.1` (to ignore 10% of the insertion at the C-terminus extremity of the gene)
>
{: .hands_on}

The output of Transit is a tabulated file containing the following columns:
- Gene ID
- Name of the gene
- Gene description
- Number of Transposon Insertions Observed within the Gene
- Total Number of TA sites within the Gene
- Length of the Maximum Run of Non-Insertions observed (in number of TA sites)
- Span of nucleotides for the Maximum Run of Non-Insertions
- Posterior Probability of Essentiality
- Essentiality call for the gene
  - E=Essential
  - U=Uncertain
  - NE=Non-Essential
  - S=too short

> <comment-title>More details about Transit</comment-title>
> You can find more information on [Transit manual page](https://transit.readthedocs.io/en/latest/transit_methods.html)
{: .comment}

We can obtain the list of genes predicted as essential by filtering on the essentiality call.

> <hands-on-title>Hands-on : Get table of essential genes</hands-on-title>
>
> 1. **Filter data on any column using simple expressions** {% icon tool %} with:
>    - {% icon param-collection %} *"Filter"*: output of the last **Gumbel**
>    - *With following condition"*: `c9=='E'` (to select essential genes)
>
{: .hands_on}


> <question-title></question-title>
>
> How many genes are essential in each conditions?
>
> > <solution-title></solution-title>
> >
> > 359 genes are essential in control, and 381 in condition.
> {: .solution }
>
{: .question}


## Compare the essential genes between two conditions

Now let's compare the results between out two conditions.

> <hands-on-title>Separate Files from the collection</hands-on-title>
>
> 1. **Extract Dataset from a list** {% icon tool %} to extract 1st file in the collection with:
>    - {% icon param-collection %} *Input List"*: output of the last **Filter**
>    - *How should a dataset be selected?"*: `Select by index`
>      - *Element index:"*: `0` to select the first dataset
>
> 2. Add the `#condition` tag
>
> 3. **Extract Dataset from a list** {% icon tool %} to extract 2nd file in the collection with:
>    - {% icon param-collection %} *Input List"*: output of the last **Filter**
>    - *How should a dataset be selected?"*: `Select by index`
>      - *Element index:"*: `1` to select the second dataset
>
> 4. Add the `#control` tag
{: .hands_on}

We would like now to get the list of essential gene in both conditions

> <hands-on-title>Get gene essential in both conditions</hands-on-title>
>
> 1. **Join two files** {% icon tool %} with:
>    - {% icon param-file %} *1st file"*: output of **Extract Dataset** with `control` tag
>    - *Column to use from 1st file"*: `Column : 1` (to compare on gene ID)
>    - {% icon param-file %} *2nd file"*: output of **Extract Dataset** with `condition` tag
>    - *Column to use from 2nd file"*: `Column : 1` (to compare on gene ID)
>    - *Output lines appearing in"*: `Both 1st and 2nd files` (to get common essential genes)
>
{: .hands_on}


Let's now get the list of essential gene that are different between the two conditions.

> <hands-on-title>Get gene essential only in control</hands-on-title>
>
> 1. **Join two files** {% icon tool %} with:
>    - {% icon param-file %} *1st file"*: output of **Extract Dataset** with `control` tag
>    - *Column to use from 1st file"*: `Column : 1` (to compare on gene ID)
>    - {% icon param-file %} *2nd file"*: output of **Extract Dataset** with `condition` tag
>    - *Column to use from 2nd file"*: `Column : 1` (to compare on gene ID)
>    - *Output lines appearing in"*: `1st but not 2nd`
{: .hands_on}



> <hands-on-title>Hands-on : Get gene essential only in condition</hands-on-title>
>
>  1. **Join two files** {% icon tool %} with:
>   - *1st file"*: output of **Extract Dataset** with `control` tag
>   - *Column to use from 1st file"*: `Column : 1` (to compare on gene ID)
>   - *2nd file"*: output of **Extract Dataset** with `condition` tag
>   - *Column to use from 2nd file"*: `Column : 1` (to compare on gene ID)
>   - *Output lines appearing in"*: `2nd but not 1st`
{: .hands_on}


> <question-title></question-title>
>
> 1. How many genes are essential in both conditions?
> 2. How many genes are essential only in control?
> 3. How many genes are essential only in condition?
>
>
> > <solution-title></solution-title>
> >
> > 1. 320 genes are essential in both conditions.
> > 2. 39 genes are essential only in control.
> > 3. 61 genes are essential only in condition.
> {: .solution }
>
{: .question}

# Conclusion

In this tutorial, we learned how to predict essential genes in different conditions. Although it seems to be a long process, keep in mind that the steps allowing to get the list of TA site does not need to be repeated for each analysis as long as you study the same genome.
For repetitive analyses, the steps are fewer : mapping, coverage, attribution to TA sites, merging of strands and Transit analysis.

This tutorial focused on simple gene essentiality prediction in individual samples. You could be interested in intergenic regions as well, or comparing many samples together. Other methods exist in transit for these goals, that follow the same workflow. It is important to look at your data and to perform quality control to evaluate the coverage and sparsity of your data. This will influence not only the choice of method (HMM does not manage sparse data well), but the choice of parameters for your analysis.

Once you get this list of gene, it is interesting to verify manually the genes classified as essentials. In order to do that, you can visualize the read mapped on your genome by displaying the bam files in IGV.

{% snippet topics/visualisation/faqs/visualizations_igv_add_genome.md %}

{% snippet topics/visualisation/faqs/visualizations_igv_view_mapping.md %}

When looking at the read mapping at the location of predicted essential genes, you may encounter several situation: Some gene will have no read at all mapping to them, some will have an empty region, while other will have no clearly defined empty region but a very low count of reads, possibly indicating a growth defect induced by the gene disruption.

![Example of Essential gene with no reads mapped](../../images/tnseq/empty_gene.png "Example of Essential gene with no reads mapped")
![Example of Essential gene with depleted coverage](../../images/tnseq/low_coverage_gene.png "Example of Essential gene with depleted coverage")

Once you have curated and confirm the genes that appear essential, comparing the annotation of essential genes with the condition of growth will allow you a better understanding of certain mechanism of interest, like the gene involved in resistance to stress for example.

A comparison with published results of essentiality in your organism will be useful but don't expect to find exactly the same results. The conditions of growth differ from one experiment to another, and can highly influence the detection of essential genes.


topic: genome-annotation
tutorial: official-gene-set
title: Creating an Official Gene Set
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: []
tags_json: ["gmod", "apollo2"]
content_hash: c83648c0489b2f2be8413e940aa43066
last_modified: 2026-05-15T22:28:13.513342
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/official-gene-set/tutorial.html

content:
Here's how we manage annotations at BIPAA:

- Automatic annotation with Maker
- Apollo server for manual curation, following our guidelines
- Each night, our Apollo report application checks that each annotated gene follows our guidelines. A report is available for each user with a list of errors and warnings that needs to be fixed, and a list of valid gene.
- Regularly we generate a new OGS by merging the automatic annotation with Apollo annotated genes (the valid ones only). We use the ogs_merge script to do that.
- To submit the OGS to ENA @ EBI in EMBL format, we (would like to) use gff2embl



> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}


topic: genome-annotation
tutorial: lncrna
title: Long non-coding RNAs (lncRNAs) annotation with FEELnc
difficulty: intermediate
time_estimation: 2h
requirements_json: ["type: internal", "funannotate"]
tags_json: ["eukaryote"]
content_hash: d779b8732ac43d60c50b80d18de18476
last_modified: 2026-05-15T22:28:13.512683
zenodo_link: https://zenodo.org/records/11367439
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/lncrna/tutorial.html

content:
{mRNAs} are not the only type of RNAs present in organisms (like mammals, insects or plants) and represent only a small fraction of the transcripts. A vast repertoire of small (miRNAs, snRNAs) and {lncRNAs} are also present. {LncRNAs} are generally defined as transcripts longer than 200 nucleotides that are not translated into functional proteins. They are important because of their major roles in cellular machinery and their presence in large number. Indeed, they are notably involved in gene expression regulation, control of translation or imprinting. Statistics from the [GENCODE project](https://www.gencodegenes.org/human/stats_41.html) reveals that the human genome contains more than 19,095 lncRNA genes, almost as much as the 19,370 protein-coding genes.

Using RNASeq data, we can reconstruct assembled transcripts (with ou without any reference genome) which can then be annotated and identified individually as {mRNAs} or {lncRNAs}.

In this tutorial, we will use a software tool called *StringTie* ({% cite Pertea %}) to assemble the transcripts and then *FEELnc* ({% cite Wucher %}) to annotate the assembled transcripts of a small eukaryote: [*Mucor mucedo*](https://en.wikipedia.org/wiki/Mucor_mucedo) (a fungal plant pathogen).

[*StringTie*](https://github.com/gpertea/stringtie) is a fast and highly efficient assembler of RNA-Seq alignments into potential transcripts.


[*FEELnc (FlExible Extraction of Long non-coding RNA)*](https://github.com/tderrien/FEELnc) is a pipeline to annotate {lncRNAs} from RNASeq assembled transcripts. It is composed of 3 modules:
* FEELnc_filter: Extract, filter candidate transcripts.
* FEELnc_codpot: Compute the coding potential of candidate transcripts.
* FEELnc_classifier: Classify {lncRNAs} based on their genomic localization wrt others transcripts.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data upload

To assemble transcriptome with StringTie and annotate {lncRNAs} with FEELnc, we will use the following files :

- The **genome sequence** in fasta format. For this tutorial, we will use the genome assembled in the [Flye assembly tutorial]({% link topics/assembly/tutorials/flye-assembly/tutorial.md %}).
- The **genome annotation** in GFF3 format. We will use the genome annotation obtained in the [Funannotate tutorial]({% link topics/genome-annotation/tutorials/funannotate/tutorial.md %}).
- Some aligned **RNASeq data** in bam format. Here, we will use some mapped RNASeq data where mapping was done using STAR.


> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/11367439/files/genome_assembly.fasta
>    https://zenodo.org/records/11367439/files/genome_annotation.gff3
>    https://zenodo.org/records/11367439/files/SRR8534859_RNASeq_mapped.bam
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

# Transcripts assembly with StringTie

StringTie is a fast and highly efficient assembler of RNA-Seq alignments into potential transcripts.
StringTie takes as input a SAM, BAM or CRAM file sorted by coordinate (genomic location). This file should contain spliced RNA-seq read alignments such as the ones produced by TopHat, HISAT2 or STAR. The TopHat output is already sorted, but the SAM ouput from other aligners should be sorted using the samtools program.

A reference annotation file in GTF or GFF3 format can be provided to StringTie which can be used as 'guides' for the assembly process and help improve the transcript structure recovery for those transcripts.

> <hands-on-title>Transcripts assembly</hands-on-title>
>
> {% tool [StringTie](toolshed.g2.bx.psu.edu/repos/iuc/stringtie/stringtie/2.2.3+galaxy1) %} with the following parameters:
>    - *"Input options"*: `Short reads`
>    - {% icon param-file %} *"Input short mapped reads"*: `SRR8534859_RNASeq_mapped.bam`
>    - *"Specify strand information"*: Unstranded
>    - *"Use a reference file to guide assembly?"*: Use reference GTF/GFF3
>    - *"Reference file"*: Use a file from history
>        - {% icon param-file %} *"GTF/GFF3 dataset to guide assembly"*: `genome_annotation.gff3`
>    - *"Use Reference transcripts only?"*: `No`
>    - *"Output files for differential expression?"*: `No additional output`
>    - *"Output coverage file"*: `No`
>
{: .hands_on}

We obtain an annotation file (GTF format) which contained all assembled transcripts present in the RNASeq data.

 After this step, the transcriptome is assembled and ready for {lncRNAs} annotation.

> <question-title></question-title>
>
> How many transcripts are assembled  ?
>
> > <solution-title></solution-title>
> >
> > Specific features can be extracted from the GTF file using for example {% tool [Extract features from GFF data](Extract_features1) %}. By selecting `transcript` From `column 3 / Feature`, we can select only the transcript elements present in this annotation file. Assembly contains 17,653 transcripts (corresponding to the number of lines in the filtered GTF file).
> >
> {: .solution}
>
{: .question}

# lncRNAs annotation with FEELnc

FEELnc is a pipeline which is composed of 3 steps. These 3 steps are run automatically when running FEELnc within Galaxy. The first step (FEELnc_filter) consists in filtering out unwanted/spurious transcripts and/or transcripts overlapping (in sense) exons of the reference annotation, and especially protein coding exons as they more probably correspond to new mRNA isoforms.

To use FEELnc, we need to have a reference annotation file in GTF format, which contains protein-coding genes annotation. Presently, we downloaded only the reference annotation file in GFF3 format (`annotation.gff3`). To convert from GFF3 to GTF format, we will use [*gffread*](https://github.com/gpertea/gffread).

> <hands-on-title>FEELnc</hands-on-title>
>
> 1. {% tool [gffread](toolshed.g2.bx.psu.edu/repos/devteam/gffread/gffread/2.2.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input BED, GFF3 or GTF feature file"*: `genome_annotation.gff3`
>    - *"Feature File Output"*: `GTF`
>
> 2. {% tool [FEELnc](toolshed.g2.bx.psu.edu/repos/iuc/feelnc/feelnc/0.2.1) %} with the following parameters:
>    - {% icon param-file %} *"Transcripts assembly"*: `Assembled transcript` (output of **StringTie** {% icon tool %})
>    - {% icon param-file %} *"Reference annotation"*: `genome_annotation.gtf` (Output of **gffread** {% icon tool %})
>    - {% icon param-file %} *"Genome sequence"*: `genome_assembly.fasta`
>
{: .hands_on}

FEELnc provides 3 output files
* lncRNA annotation file: annotation file in GTF format which contains the final set of {lncRNAs}
* mRNA annotation file: annotation file in GTF format which contains the final set of {mRNAs}
* Classifier output file: table containing classification of {lncRNAs} based on their genomic localisation w.r.t other transcripts (direction: `sense` or `antisense`, type: `genic`, if the lncRNA gene overlaps an RNA gene from the reference annotation file or `intergenic` (lincRNA) if not).

FEELnc provides also summary file in stdout.

> <question-title></question-title>
>
> How many RNAs does this annotation contain ?
> How many interactions between {lncRNAs} and {mRNAs} have been identified ?
> Can you describe the different types of {lncRNAs} ?
>
> > <solution-title></solution-title>
> >
> > The summary file indicates 268 {lncRNAs} and 0 new {mRNAs} were annotated by FEELnc. The initial annotation contains 13,795 {mRNAs} annotated. Therefore, a total of 14,063 RNAs are currently annotated.
> >
> > The summary file indicates 772 interactions between {lncRNAs} and {mRNAs}. These interactions are described in the Classifier output file.
> >
> > The different types of {lncRNAs} (intergenic (sense and antisense), intragenic (sense)) are described in the Classifier output file. We observe that the majority of the {lncRNAs} are intergenic. These {lncRNAs} can each have interactions with several {mRNAs}. Only 5 {lncRNAs} are genic. These {lncRNAs} have only one interaction with the mRNA that contains it.
> >
> {: .solution}
>
{: .question}

For future analyses, it would be interesting to use an updated annotation containing {mRNAs} and {lncRNAs} annotations. Thus, we will merge the reference annotation with those obtained with FEELnc.

> <hands-on-title>Merge the annotations</hands-on-title>
>
> {% tool [concatenate](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/1.0.0) %} with the following parameters:
>    - {% icon param-file %} *"Datasets to concatenate"*: `genome_annotation.gtf` (Output of **gffread** {% icon tool %})
>    - Insert Dataset
>    - {% icon param-file %} *"Dataset"*: `lncRNA annotation with FEELnc`
>
{: .hands_on}


# Conclusion


Congratulations for reaching the end of this tutorial! Now you know how to perform an annotation of {lncRNAs} by using RNASeq data.

tools_json:
["StringTie", "toolshed.g2.bx.psu.edu/repos/iuc/stringtie/stringtie/2.2.3+galaxy1", "Extract features from GFF data", "Extract_features1", "gffread", "toolshed.g2.bx.psu.edu/repos/devteam/gffread/gffread/2.2.1.4+galaxy0", "FEELnc", "toolshed.g2.bx.psu.edu/repos/iuc/feelnc/feelnc/0.2.1", "concatenate", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/1.0.0"]


topic: genome-annotation
tutorial: bacterial-genome-annotation
title: Bacterial Genome Annotation
difficulty: introductory
time_estimation: 3H
requirements_json: []
tags_json: ["gmod", "illumina", "bacteria", "microgalaxy", "jbrowse1"]
content_hash: e5e038846ce3800c0cdba74562fc35a8
last_modified: 2026-05-15T22:28:13.497140
zenodo_link: https://zenodo.org/record/10572227
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/bacterial-genome-annotation/tutorial.html

content:
After sequencing and assembly, a genome can be annotated. It is an essential step to describe the genome.

Genome annotation consists in describing the structure and function of the components of the genome, by predicting, analyzing, and interpreting them in order to extract their biological significance and understand the biological processes in which they participate. Among other things, it identifies the locations of genes and all the coding regions in a genome (*structural annotation*) and determines what those genes do (*functional annotation*).

To illustrate the process to annotate a bacterial genome, we take an assembly of a bacterial genome (KUN1163 sample) generated by following a [bacterial genome assembly tutorial]({% link topics/assembly/tutorials/mrsa-illumina/tutorial.md %}) from data produced in "Complete Genome Sequences of Eight Methicillin-Resistant *Staphylococcus aureus* Strains Isolated from Patients in Japan" ({% cite Hikichi_2019 %}).

> Methicillin-resistant *Staphylococcus aureus* (MRSA) is a major pathogen
> causing nosocomial infections, and the clinical manifestations of MRSA
> range from asymptomatic colonization of the nasal mucosa to soft tissue
> infection to fulminant invasive disease. Here, we report the complete
> genome sequences of eight MRSA strains isolated from patients in Japan.
{: .quote cite="{% cite_url Hikichi_2019 %}"}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Galaxy and data preparation

Any analysis should get its own Galaxy history. So let's start by creating a new one and get the data (contig file from the assembly) into it.

> <hands-on-title>Prepare Galaxy and data</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 3. {% tool [Import](upload1) %} the contig file from [Zenodo]({{ page.zenodo_link }}) or from Galaxy shared data libraries:
>
>    ```
>    {{ page.zenodo_link }}/files/DRR187559_contigs.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
{: .hands_on}

# Contig annotation

For annotating the contigs, several tools exists to do that: Prokka ({% cite seemann2014prokka %}), Bakta ({% cite schwengers2021bakta %}), etc. Here, we use **Bakta** as recommended by {% include _includes/contributor-badge.html id="tseemann" %} as the successor of **Prokka**.

**Bakta** is a tool for the rapid & standardized annotation of bacterial genomes and plasmids from both isolates and metagenome-assembled genomes (MAGs). It implements a comprehensive annotation workflow for coding and non-coding genes (*i.e.* tRNA, rRNA).

![Flow diagram of Bakta. It depicts the sequential steps and connections involved in its functioning.](./images/bakta_wf.png "Overview of the Bakta annotation workflow ({% cite schwengers2021bakta %})")

It is also able to detect and annotate small proteins (sORF). Predicted CDS are annotated using an alignment-free protein sequence identification approach with cross-references to public databases via stable identifiers.

> <hands-on-title>Contig annotation</hands-on-title>
>
> 1. {% tool [Bakta](toolshed.g2.bx.psu.edu/repos/iuc/bakta/bakta/1.9.3+galaxy0) %} with the following parameters:
>    - In *"Input/Output options"*:
>        - {% icon param-file %} *"Select genome in fasta format"*: Contig file
>        - *"Bakta database"*: latest one
>        - *"AMRFinderPlus database"*: latest one
>    - In *"Optional annotation"*:
>        - *"Keep original contig header"*: `Yes`
>    - In *"Selection of the output files"*:
>        - *"Output files selection"*:
>          - `Annotation file in TSV`
>          - `Annotation and sequence in GFF3`
>          - `Feature nucleotide sequences as FASTA`
>          - `Summary as TXT`
>          - `Plot of the annotation result as SVG`
{: .hands_on}

**Bakta** can generate many outputs. Here we selected:

- `Analysis_summary`: a summary of the analysis as text file

   > <question-title></question-title>
   >
   > 1. How many contigs have been provided as input?
   > 2. How long is the draft genome?
   > 3. How many CDSs have been found?
   > 4. How many small proteins?
   > 5. Which other components have been found?
   >
   > How does it compare to results for KUN1163 in [Table 1](https://journals.asm.org/doi/10.1128/mra.01212-19#tab1) in {% cite Hikichi_2019 %}?
   >
   > > <solution-title></solution-title>
   > >
   > > 1. 44 (the sequence count)
   > > 2. 2,911,349 bp, with is a bit shorter than the expected 2,914,567 bp in [Table 1](https://journals.asm.org/doi/10.1128/mra.01212-19#tab1) in {% cite Hikichi_2019 %}
   > > 3. 2,717 CDSs, a bit more than the expected 2,704 CDSs in [Table 1](https://journals.asm.org/doi/10.1128/mra.01212-19#tab1) in {% cite Hikichi_2019 %}
   > > 4. 5 sORFs. There is no information about sORFs in {% cite Hikichi_2019 %}
   > > 5. Other components
   > >
   > >    Components | Bakta | {% cite Hikichi_2019 %}
   > >    --- | --- | ---
   > >    tRNAs | 57 | 61
   > >    Transfer-messenger RNA (tmRNAs) | 1 | 1
   > >    rRNAs | 9 | 5
   > >    ncRNAs | 95 | No information
   > >
   > {: .solution}
   {: .question}

- `Nucleotide_sequences` with feature nucleotide sequences as FASTA file

   > <question-title></question-title>
   >
   > 1. How many sequences are in this file?
   > 2. What are the sequences stored there?
   >
   > > <solution-title></solution-title>
   > >
   > > 1. 2,884 sequences
   > > 2. Here are stored:
   > >    - tRNAs
   > >    - tmRNAs
   > >    - rRNAs
   > >    - ncRNAs
   > >    - CDSs
   > >    - sORFs
   > >
   > {: .solution}
   {: .question}

- `annotation_summary` with annotations as simple human readble TSV

   > <question-title></question-title>
   >
   > What is stored there?
   >
   > > <solution-title></solution-title>
   > >
   > > This a table with 9 columns (`Sequence Id`, `Type`, `Start`, `Stop`, `Strand`, `Locus Tag`, `Gene`, `Product`, `DbXrefs`). It contains here 2,916 lines, so the information and location for:
   > >    - tRNAs
   > >    - tmRNAs
   > >    - rRNAs
   > >    - ncRNAs
   > >    - CDSs
   > >    - sORFs
   > >    - gaps
   > >    - oriCs
   > >    - oriVs
   > >    - oriTs
   > >
   > {: .solution}
   {: .question}


- `Annotation_and_sequences` in GFF3

   GFF is a file format used for describing genes and other features of DNA, RNA and protein sequences. It is a tab delimited file with 9 fields per line:

   1. **seqid**: The name of the sequence where the feature is located.
   2. **source**: The algorithm or procedure that generated the feature. This is typically the name of a software or database.
   3. **type**: The feature type name, like "gene" or "exon". In a well structured GFF file, all the children features always follow their parents in a single block (so all exons of a transcript are put after their parent "transcript" feature line and before any other parent transcript line). In GFF3, all features and their relationships should be compatible with the standards released by the Sequence Ontology Project.
   4. **start**: Genomic start of the feature, with a 1-base offset. This is in contrast with other 0-offset half-open sequence formats, like BED.
   5. **end**: Genomic end of the feature, with a 1-base offset. This is the same end coordinate as it is in 0-offset half-open sequence formats, like BED.
   6. **score**: Numeric value that generally indicates the confidence of the source in the annotated feature. A value of "." (a dot) is used to define a null value.
   7. **strand**: Single character that indicates the strand of the feature. This can be "+" (positive, or 5'->3'), "-", (negative, or 3'->5'), "." (undetermined), or "?" for features with relevant but unknown strands.
   8. **phase**: phase of CDS features; it can be either one of 0, 1, 2 (for CDS features) or "." (for everything else). See the section below for a detailed explanation.
   9. **attributes**: A list of tag-value pairs separated by a semicolon with additional information about the feature.

   > <question-title></question-title>
   >
   > How many features are annotated?
   >
   > > <solution-title></solution-title>
   > >
   > > 51k+ (number of lines in the GFF)
   > >
   > {: .solution}
   {: .question}

- Plot of the annotation as circular genome annotation

   ![A circular plot showcasing the draft genome, providing a visual representation of its genetic information.](./images/bakta_plot.png)

   > <question-title></question-title>
   >
   > 1. What the 2 rings in the center?
   > 2. How are plotted the features?
   >
   > > <solution-title></solution-title>
   > >
   > > 1. The first ring represents the GC content per sliding window over the entire sequence(s) with in green representing GC above  and red GC below average. The 2nd ring represents the GC skew in orange and blue.
   > > 2. All features are plotted on two rings representing the forward and reverse strand from outer to inner with CDS in grey (the other colors are hard to distinguish)
   > >
   > {: .solution}
   {: .question}

# Further structural annotation

**Bakta** gives a lot of information already, especially regarding CDSs or RNAs, but some structural annotation might be missing, e.g. plasmids, or interesting to identify independently.

## Plasmids

To identify plasmids in our contigs, we use **PlasmidFinder** ({% cite carattoli2020plasmidfinder %}), a tool for the identification and typing of plasmid sequences in Whole-Genome Sequencing. It uses the [plasmidfinder database](https://bitbucket.org/genomicepidemiology/plasmidfinder_db/) with hundreds of sequences to predict the plasmid in the data.

> <hands-on-title> Plasmid identification</hands-on-title>
>
> 1. {% tool [PlasmidFinder](toolshed.g2.bx.psu.edu/repos/iuc/plasmidfinder/plasmidfinder/2.1.6+galaxy1) %} with the following parameters:
>    - In *"Input parameters"*:
>        - {% icon param-file %} *"Choose a fasta or fastq file"*: Contig file
>        - *"PlasmidFinder database"*: most recent one
{: .hands_on}

**PlasmidFinder** generates several outputs:

- `raw_results.txt`: A text file containing the result table and alignments
- `results.tsv`: A tabular file with the following columns:

   - **Database**
   - **Plasmid**: Plasmid against which the input genome has been aligned.
   - **Identity**: Percent identity in the alignment between the best matching plasmid in the database and the corresponding sequence in the inputgenome (also called the high-scoring segment pair (HSP)). A perfect alignment is 100%, but must also cover the entire length of the plasmid in the database (compare example 1 and 3).
   - **Query/Template Length**: Query length is the length of the best matching plasmid in the database, while HSP length is the length of the alignment between the best matching plasmid and the corresponding sequence in the genome (also called the high-scoring segment pair (HSP)).
   - **Contig**: Name of contig the plasmid is found in.
   - **Position in contig**: Starting position of the found gene in the contig.
   - **Note**: Notes about the plasmid
   - **Accession number**: Reference Genbank accession number accoding to NCBI for the plasmid in the database.

   > <question-title></question-title>
   >
   > 1. How many plasmid sequences have been found?
   > 2. Where are they located?
   > 3. Are these sequences all associated with *Staphylococcus aureus*?
   > 4. What can we conclude about contig00019?
   >
   > > <solution-title></solution-title>
   > >
   > > 1. 5 plasmid sequences
   > > 2. 3 are on the contig00019, 1 on contig00002, and 1 on contig00024
   > > 3. Looking at the accession number on the NCBI, we find that:
   > >    - CP000737, AP003139 (2 times) correspond to *Staphylococcus aureus* plasmids
   > >    - AF503772 corresponds to a *Enterococcus faecalis* plasmid
   > >    - CP003584 corresponds to a *Enterococcus faecium* plasmid
   > >
   > > 4. All plasmid sequences corresponding to *Staphylococcus aureus* plasmids are all on contig00019, making this contig likely a plasmid. In addition, this contig has a length of 30,347 bp, which is similar to the expected length of the plasmid for KUN1163 in [Table 1](https://journals.asm.org/doi/10.1128/mra.01212-19#tab1) in {% cite Hikichi_2019 %}
   > {: .solution}
   {: .question}

- `plasmid.fasta`: A fasta file containing the best matching sequences from the query genome
- `hit_in_genome.fasta`: A fasta file containing the best matching plasmid genes from the database

## Integrons

Integrons are genetic mechanisms that allow bacteria to adapt and evolve rapidly through the stockpiling and expression of new genes. An integron is minimally composed of:

- a gene encoding for a site-specific recombinase (intI)
- a proximal recombination site (attI), which is recognized by the integrase and at which gene cassettes may be inserted
- a promoter (Pc) which directs transcription of cassette-encoded genes

To detect integrons, we will use **IntegronFinder** ({% cite neron2022integronfinder %}). This tool:

1. Annotates the CDS with **Prodigal**
2. Detects independently:

   1. integron integrase using the intersection of two HMM profiles: one specific of tyrosine-recombinase (PF00589) and one specific of the integron integrase, near the patch III domain of tyrosine recombinases
   2. attC recombination sites with a covariance model (CM), which models the secondary structure in addition to the few conserved sequence positions.

3. Integrates the results to distinguish 3 types of elements

   - Complete integron: Integron with integron integrase nearby attC site(s)
   - In0 element: Integron integrase only, without any attC site nearby
   - CALIN element: Cluster of attC sites Lacking INtegrase nearby

> <hands-on-title> Integron identification</hands-on-title>
>
> 1. {% tool [IntegronFinder](toolshed.g2.bx.psu.edu/repos/iuc/integron_finder/integron_finder/2.0.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Replicon file"*: Contig file
>    - *"Thorough local detection"*: `Yes`
>    - *"Search also for promoter and attI sites?"*: `Yes`
>    - *"Remove log file"*: `Yes`
>
{: .hands_on}

**IntegronFinder** generates 2 outputs:

- A summary with for each sequence in the input the number of identified CALIN elements, In0 elements, and complete integrons.

   > <question-title></question-title>
   >
   > How many integron elements have been found?
   >
   > > <solution-title></solution-title>
   > >
   > > No integron elements have been found on any contig. It could be because the genome is too stable, or because the assembly quality is not good enough and some parts useful for the integron detection were removed.
   > {: .solution}
   {: .question}

- An integron annotation file as a tabular

## IS (Insertion Sequence) elements

Insertion sequence (IS) element is a short DNA sequence that acts as a simple transposable element. IS are the smallest but most abundant autonomous transposable elements in bacterial genomes.  They only code for proteins implicated in the transposition activity. They play then a key role in bacterial genome organization and evolution.

To detect IS elements, we will use **ISEScan** ({% cite xie2017isescan %}). **ISEScan** is a highly sensitive software pipeline based on profile hidden Markov models constructed from manually curated IS elements

> <hands-on-title> IS identification </hands-on-title>
>
> 1. {% tool [ISEScan](toolshed.g2.bx.psu.edu/repos/iuc/isescan/isescan/1.7.2.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Genome fasta input"*: Contig file
>
{: .hands_on}

**ISEScan** generates several files:

- A summary as a table

- The results as a table

   > <question-title></question-title>
   >
   > 1. How many IS elements have been detected?
   > 2. Where are they located?
   > 3. What the different IS families?
   >
   > > <solution-title></solution-title>
   > >
   > > 1. 20
   > > 2. Using {% tool [Group data by a column](Grouping1) %} to group and count on 1st column, we find:
   > >
   > >    Contig | IS element number
   > >    --- | ---
   > >    contig00001 | 2
   > >    contig00002 | 1
   > >    contig00003 | 2
   > >    contig00004 | 1
   > >    contig00005 | 1
   > >    contig00006 | 1
   > >    contig00009 | 3
   > >    contig00010 | 1
   > >    contig00011 | 1
   > >    contig00012 | 1
   > >    contig00019 | 3
   > >    contig00027 | 1
   > >    contig00032 | 1
   > >    contig00037 | 1
   > >
   > > 3. As for previous question, when grouping and counting on 2nd column, we find 5 IS families:
   > >
   > >    IS families | Identified IS elements
   > >    --- | ---
   > >    IS1182 | 4
   > >    IS21 | 7
   > >    IS3 | 3
   > >    IS6 | 5
   > >    ISL3 | 1
   > >
   > {: .solution}
   {: .question}

- The results as a GFF file
- Several FASTA files:
   - IS nucleotide sequences
   - ORF nucleotide sequences
   - ORF amino acide sequences

# Visualisation of the annotation

We would like to look at the annotation using [JBrowse](https://jbrowse.org/jb2/) ({% cite diesh2023jbrowse %}) with several information:

1. Annotations identified by **Bakta**
2. Plasmid sequences identified by **PlasmidFinder**
3. Integrons identified by **IntegronFinder**
4. IS elements identified by **ISEscan**

**JBrowse** needs the annotations to be in GFF format. **Bakta** and **ISEscan** generated both GFF files. For **PlasmidFinder** and **IntegronFinder**, we need to format the outputs.

**PlasmidFinder** generated the `results.tsv` with all needed information. To transform it to a GFF, we need to:

1. Split the 6th column on `..` to have start and end into 2 separated columns
2. Remove in the content of column 5 what is after the contig name
2. Remove the 1st line
3. Transform to GFF3

> <hands-on-title> Transform PlasmidFinder to GFF </hands-on-title>
>
> 1. {% tool [Replace Text in a specific column](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `results.tsv` output of **PlasmidFinder**
>    - In *"Replacement"*:
>      - In *"1: Replacement"*
>        - *in column*: `Column: 6`
>        - *"Find pattern"*: `(.*)\.\.(.*)`
>        - *"Replace with"*: `\\1\t\\2`
>
>        This will split the content of the 6th column on `..` and put it into column 6 and column 7. Column 7 will be then replaced.
>
>      - {% icon param-repeat %} *"Insert Factor"*
>      - In *"2: Replacement"*
>        - *in column*: `Column: 5`
>        - *"Find pattern"*: `(.*)( len.*)`
>        - *"Replace with"*: `\\1`
>
>        This will remove in the content of column 5 what is after the contig name
>
> 2. {% tool [Select last lines from a dataset](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_tail_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Text file"*: output of **Replace Text** above
>    - *"Operation"*: `Keep everything from this line on`
>    - *"Number of lines"*: `2`
>
> 3. {% tool [Table to GFF3](toolshed.g2.bx.psu.edu/repos/iuc/tbl2gff3/tbl2gff3/1.2) %} with the following parameters:
>    - {% icon param-file %} *"Table"*: output of the above **Select last** {% icon tool %} step
>    - *"Record ID column or value"*: `5`
>    - *"Start column or value"*: `6`
>    - *"End column or value"*: `7`
>    - *"Type column or value"*: `2`
>    - *"Score column or value"*: `3`
>    - *"Source column or value"*: `1`
>    - {% icon param-repeat %} *"Insert Qualifiers"*
>        - *"Name"*: `name`
>        - *"Qualifier value column or raw text"*: `8`
>    - {% icon param-repeat %} *"Insert Qualifiers"*
>        - *"Name"*: `accession`
>        - *"Qualifier value column or raw text"*: `9`
>
> 4. Rename to `PlasmidFinder GFF`
{: .hands_on}

> <details-title>Transform IntegronFinder output to GFF if integrons found</details-title>
> **IntegronFinder** tabular output can be transformed to GFF by:
>
> 1. Replace `NA` values on column 7 by `0`
> 2. Remove the first two lines
> 3. Transform to GFF3
>
> > <hands-on-title> Transform IntegronFinder to GFF </hands-on-title>
> > 1. {% tool [Replace Text in a specific column](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1) %} with the following parameters:
> >    - {% icon param-file %} *"File to process"*: tabular output of **IntegronFinder**
> >    - In *"Replacement"*:
> >      - In *"1: Replacement"*
> >        - *in column*: `Column: 7`
> >        - *"Find pattern"*: `NA`
> >        - *"Replace with"*: `0`
> >
> > 2. {% tool [Select last lines from a dataset](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_tail_tool/9.3+galaxy1) %} with the following parameters:
> >    - {% icon param-file %} *"Text file"*: output of **Replace Text** above
> >    - *"Operation"*: `Keep everything from this line on`
> >    - *"Number of lines"*: `3`
> >
> > 3. {% tool [Table to GFF3](toolshed.g2.bx.psu.edu/repos/iuc/tbl2gff3/tbl2gff3/1.2) %} with the following parameters:
> >    - {% icon param-file %} *"Table"*: tabular output of **IntegronFinder**
> >    - *"Record ID column or value"*: `2`
> >    - *"Start column or value"*: `4`
> >    - *"End column or value"*: `5`
> >    - *"Type column or value"*: `11`
> >    - *"Score column or value"*: `7`
> >    - *"Source column or value"*: `IntegronFinder`
> >    - {% icon param-repeat %} *"Insert Qualifiers"*
> >        - *"Name"*: `name`
> >        - *"Qualifier value column or raw text"*: `3`
> >    - {% icon param-repeat %} *"Insert Qualifiers"*
> >        - *"Name"*: `annotation`
> >        - *"Qualifier value column or raw text"*: `9`
> >
> > 4. Rename to `IntegronFinder GFF`
> {: .hands_on}
{: .details}

We can now launch **JBrowse** with different information track.

> <hands-on-title>Visualize the Genome</hands-on-title>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: Contig file
>    - *"Genetic Code"*: `11. The Bacterial, Archaeal and Plant Plastid Code`
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Bakta`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `Annotation_and_sequences` (output of **Bakta** {% icon tool %})
>                        - *"JBrowse Track Type [Advanced]"*: `Neat Canvas Features`
>                        - *"Track Visibility"*: `On for new users`
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Plasmid sequences`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `PlasmidFinder GFF`
>                        - *"JBrowse Track Type [Advanced]"*: `Neat Canvas Features`
>                        - *"Track Visibility"*: `On for new users`
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `IS elements`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: GFF output of **ISEScan**
>                        - *"JBrowse Track Type [Advanced]"*: `Neat Canvas Features`
>                        - *"Track Visibility"*: `On for new users`
>
>        *If integrons are found as **IntegronFinder***
>
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Integrons`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `IntegronFinder GFF`
>                        - *"JBrowse Track Type [Advanced]"*: `Neat Canvas Features`
>                        - *"Track Visibility"*: `On for new users`
>
> 3. View the output of JBrowse
>
{: .hands_on}

In the output of the JBrowse you can view the genes, IS, plasmid, etc on the contigs. With the search tools you can easily find genes of interest. JBrowse can handle many inputs and can be very useful.

If it takes too long to build the JBrowse instance, you can view an embedded one here:

{% snippet topics/visualisation/faqs/visualizations_jbrowse.html datadir="data" loc="contig00019:1..30347" tracks="DNA,7e37ab7a5f0a1610de9d213b6a714666_0,dd555b269a8894688c3c7551d0d6a868_0,e1c36f2c4382e60836c6c4b4d8026fa5_0" %}

> <comment-title></comment-title>
> 1. It is ok to have the message stating `Error reading from name store.`.
> 2. The feature name search will not work.
{: .comment}

> <question-title></question-title>
>
> 1. Have all sequences identified by **PlasmidFinder** on contig19 been identified by **Bakta**?
> 2. Have all sequences identified by **ISEScan** on contig19 been identified by **Bakta**?
>
> > <solution-title></solution-title>
> >
> > 1. Yes all sequences in the **PlasmidFinder** track are also in the **Bakta** track. For
> > 2. All Insertion Sequences in the **ISEScan** track are also in the **Bakta** track, but the Terminanl Inverted repeats are not in the **Bakta** track
> {: .solution}
{: .question}

To learn more options for **JBrowse**, check out the [dedicated tutorial]({% link topics/visualisation/tutorials/jbrowse/tutorial.md %})

# Conclusion

In this tutorial, contigs were annotated with different tools and then visualized.

Other visualizations, specially for publications, can be done using **Circos**. To learn how to do it, you can follow the [dedicated tutorial]({% link topics/visualisation/tutorials/circos-microbial/tutorial.md %}).

To refine the genome annotation, you can also use **Apollo** and our [tutorial]({% link topics/genome-annotation/tutorials/apollo/tutorial.md %}).

tools_json:
["Import", "upload1", "Bakta", "toolshed.g2.bx.psu.edu/repos/iuc/bakta/bakta/1.9.3+galaxy0", "PlasmidFinder", "toolshed.g2.bx.psu.edu/repos/iuc/plasmidfinder/plasmidfinder/2.1.6+galaxy1", "IntegronFinder", "toolshed.g2.bx.psu.edu/repos/iuc/integron_finder/integron_finder/2.0.5+galaxy0", "ISEScan", "toolshed.g2.bx.psu.edu/repos/iuc/isescan/isescan/1.7.2.3+galaxy1", "Group data by a column", "Grouping1", "Replace Text in a specific column", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1", "Select last lines from a dataset", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_tail_tool/9.3+galaxy1", "Table to GFF3", "toolshed.g2.bx.psu.edu/repos/iuc/tbl2gff3/tbl2gff3/1.2", "JBrowse", "toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1"]


topic: genome-annotation
tutorial: comparison-braker-helixer-annotation
title: Comparison of two annotation tools - Helixer and Braker3
difficulty: intermediate
time_estimation: 8h
requirements_json: []
tags_json: ["eukaryota", "Braker3", "Helixer", "jbrowse1", "biodiversity"]
content_hash: fbc0f558a4a6f8644d041c308dc22178
last_modified: 2026-05-27T09:01:48.985066
zenodo_link: https://zenodo.org/records/14770765
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/comparison-braker-helixer-annotation/tutorial.html

content:
Annotating the eukaryotic genome represents a somewhat more complex challenge than that of prokaryotes, mainly due to the generally larger size of eukaryotic genomes and their greater number of genes. This annotation can be carried out at different levels of precision, ranging from simple identification of coding and non-coding parts to detailed structural labeling, including for example the precise location of exons, introns and other regulatory elements.

In this tutorial, we will use two software tools, Helixer and Braker3, to annotate the genome sequence of a small eukaryote: [*Mucor mucedo*](https://en.wikipedia.org/wiki/Mucor_mucedo) (a plant pathogenic fungus).
The idea of this tutorial is to compare annotation with two annotation tools that differ in their approach and operation.

[Helixer](https://github.com/weberlab-hhu/Helixer) is an annotation software with a new and different approach: it performs evidence-free predictions (no need for RNA-seq data or sequence alignments), using {GPU}, with a much faster execution time. The annotation is based on the development and use of a cross-species deep learning model. The software is used to configure and train models for *ab initio* prediction of gene structure. In other words, it identifies the base pairs in a genome that belong to the UTR/CDS/Intron genes.
There's a tutorial describing the steps involved in [annotating a genome with Helixer]({% link topics/genome-annotation/tutorials/helixer/tutorial.md %}).

[Braker3](https://github.com/Gaius-Augustus/BRAKER) is an automated bioinformatics tool that uses RNA-seq and protein sequences to annotate genome. It integrates GeneMark-ETP and AUGUSTUS software to predict genes with a high degree of precision. By combining the results of these two tools, Braker3 generates a final file containing gene annotations with strong extrinsic support (i.e. based on external experimental data). Braker3 facilitates genome annotation by leveraging transcriptomic and protein sequences to produce more reliable and robust gene predictions.
There's a tutorial describing the steps involved in [annotating a genome with Braker3]({% link topics/genome-annotation/tutorials/braker3/tutorial.md %}).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data upload

To annotate our genome both using **Helixer** and **Braker3**, we will use the same genome sequence, in fasta format.

Unlike Helixer, **Braker3** needs two other files:
- Some **RNAseq data** in bam format. These reads are already aligned on the genome, and Braker3 will use it as evidences to annotate genes.
- A set of **protein sequences**, like UniProt/SwissProt. It is important to have good quality, curated sequences here, and the UniProt/SwissProt databank fits very well.

Galaxy uses Job Cache to avoid recalculating identical analyses. If a tool has already been run with the same parameters and inputs, previous results are reused, reducing waiting time and resource consumption.

> <hands-on-title>Using Job cache</hands-on-title>
>
> {% snippet faqs/galaxy/re_use_equivalent_jobs.md %}
>
{: .hands_on}

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/15051665/files/genome_masked.fasta
>    https://zenodo.org/records/14770765/files/RNAseq_braker.bam
>    https://zenodo.org/records/14770765/files/swissprot_subset.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

# Preparing RNAseq data for Braker3 annotation

In this tutorial, the alignments from RNA-seq are already prepared.
This data was obtained using the **RNA STAR** tool, as explained in detail in the [Braker3]({% link topics/genome-annotation/tutorials/braker3/tutorial.md %}) tutorial.

Note that when using Braker3 with RNA STAR to align RNA-seq data, the **--outSAMstrandField intronMotif** parameter must be added.
This parameter adds specific intron information to the alignment file (BAM).
This information is necessary for Braker3 to correctly understand and use the alignments to annotate genes.
Without this parameter, Braker3 may not function correctly or may produce incomplete results.

Here's how to run RNA STAR with this specific parameter (**"Read alignement tags to include in the BAM output"** option), before annotating with Braker3:

> <hands-on-title></hands-on-title>
>
> 1. {% tool [RNA STAR](toolshed.g2.bx.psu.edu/repos/iuc/rgrnastar/rna_star/2.7.11a+galaxy1) %} with the following parameters:
>    - *"Single-end or paired-end reads"*: `Paired-end (as individual datasets)`
>        - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file, forward reads"*: `rnaseq_R1.fq.gz` (Input dataset)
>        - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file, reverse reads"*: `rnaseq_R2.fq.gz` (Input dataset)
>    - *"Custom or built-in reference genome"*: `Use reference genome from history and create temporary index`
>        - {% icon param-file %} *"Select a reference genome"*: `genome_masked.fasta` (Input dataset)
>        - *"Length of the SA pre-indexing string"*: `11`
>    - In *"BAM output format specification"*:
>        - In *"Read alignement tags to include in the BAM output"*:
>             - Select `XS (strand flag, see parameter help below)`
>
{: .hands_on}

# Structural annotation
## By Helixer

We can run [**Helixer**](https://github.com/weberlab-hhu/Helixer) to perform the structural annotation of the genome.

Helixer requires only the genome sequence.
We also need to choose between 4 different lineages: *invertebrate*, *vertebrate*, *land plant* or *fungi*. Select the one that fits the best to the species you are studying: *fungi* in our case. Helixer is shipped with these 4 models that were trained specifically to annotate genes from each of these lineages. Advanced users can upload their own lineage model in .h5 format with the *"Lineage model"* option.

As an option, we can also enter a species name.


> <hands-on-title>Genome annotation with Helixer</hands-on-title>
>
> 1. {% tool [Helixer](toolshed.g2.bx.psu.edu/repos/genouest/helixer/helixer/0.3.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Genomic sequence"*: `genome_masked.fasta` (Input dataset)
>    - In *"Available lineages"*: "*select*" `fungi`
>    - In *"Species name"*: `Mucor mucedo`
>
{: .hands_on}

> <comment-title>Advanced parameters</comment-title>
>
> Depending on the lineage, the parameters *"Subsequence length"*, *"Overlap offset"* and *"Overlap corelength"* are adjusted to corresponding default values (listed in the help of each option).
>
> This is due in particular to the size of the genomes. Indeed, it is recommended to increase the value of *"Subsequence length"* for genomes containing large genes. This is particularly important for vertebrates and invertebrates.
>
> The default values used by Galaxy are the ones recommended by Helixer authors. If you wish to modify these default values, you can do so by entering your values in the *"Subsequence length"*, *"Overlap offset"* and *"Overlap corelength"* parameters.
>
{: .comment}

> <comment-title>Don't wait</comment-title>
>
> This step can take a bit of time to run: although Helixer runs much faster than many other annotation tools (typically <20min for this tutorial), it requires specific hardware (GPU) that is often available in limited quantity on computing systems. It means your job can be placed in queue for a longer time than a more standard Galaxy job.
>
> While it runs, we can already schedule the following steps. Galaxy will run them automatically as soon as the Helixer annotation is ready.
{: .comment}

Helixer produces a single output dataset: a GFF3 file. The GFF3 format is a standard bioinformatics format for storing genome annotations. Each row describes a genomic entity, with columns detailing its identifier, location, score and other attributes.

## By Braker3

We have the following required inputs:

- A masked genome in FASTA format
- RNAseq sequence alignments in BAM format
- Protein annotation in FASTA format

With these files, we can run [**Braker3**](https://github.com/Gaius-Augustus/BRAKER) to perform the structural annotation of the genome.

> <hands-on-title>Genome annotation with Braker3</hands-on-title>
>
> 1. {% tool [Braker3](toolshed.g2.bx.psu.edu/repos/iuc/braker3/braker3/3.0.8+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Assembly to annotate"*: `genome_masked.fasta` (Input dataset)
>    - *"Species name"*: `Mucor mucedo`
>    - {% icon param-file %} *"RNA-seq mapped to genome to train Augustus/GeneMark"*: `RNASeq_braker.bam` (Input dataset)
>    - {% icon param-file %} *"Proteins to map to genome"*: `SwissProt_subset.fasta` (Input dataset)
>    - {% icon param-file %} *"Fungal genome"*: select `Yes`
>    - *"Output format"*: `GFF3`
>
{: .hands_on}

> <comment-title>On parameters</comment-title>
>
> - If you are working with fungi, we recommend that you activate the **--fungus** parameter. This enables Braker3 to run the GeneMark-ETP algorithm with a branch point model.
{: .comment}

> <comment-title>Don't wait</comment-title>
>
> - This step will take a long time to complete: it could take several hours for this tutorial. While it's running, we can already schedule the following steps. Galaxy will automatically execute them as soon as the structural annotation is ready.
> - There are several reasons for this: the size of the genome, the quality and quantity of the RNA-Seq data, the supervised learning model for prediction, and more.
{: .comment}

Braker3 generates an output file in GTF format by default. But it is possible to generate the output file in GFF3 output. This file contains informations on gene locations, exons, introns, etc.
GFF3 files are generally preferred for complex and varied annotations, while GTF is often used for transcript annotations in RNA-seq studies.

If you can't wait to have the Braker3 annotation and move on to the comparison step between our two annotation tools, we invite you to import the Braker3 annotation:

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/14770765/files/braker3.gff3
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

# Quality evaluation

## Evaluation with **Busco**

[BUSCO](http://busco.ezlab.org/) (Benchmarking Universal Single-Copy Orthologs) is a widely used tool to evaluate the quality of a genome assembly and annotation. By comparing genomes from various related and distantly related species, the authors determined sets of ortholog genes that are present in single copy in (almost) all the species of a clade (Bacteria, Fungi, Plants, Insects, Mammalians, …).
Most of these genes are essential for the organism to live, and are expected to be found in any newly sequenced and annotated genome from the corresponding clade. Using this data, BUSCO evaluates the "completeness" of genome annotation by assessing the proportion of these essential genes (also named BUSCOs) found in a set of (predicted) transcript or protein sequences.

### Braker3 annotation

We want to run BUSCO on the protein sequences predicted from gene sequences of the Braker3 annotation.

So first generate these sequences:

> <hands-on-title>Extract protein sequences</hands-on-title>
>
> 1. {% tool [GFFread](toolshed.g2.bx.psu.edu/repos/devteam/gffread/gffread/2.2.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input GFF3 or GTF feature file"*: output of {% tool [Braker3](toolshed.g2.bx.psu.edu/repos/iuc/braker3/braker3/3.0.8+galaxy0) %}
>    - In *"Reference Genome"* select: `From your history` (Input dataset)
>    - *"Genome Reference Fasta"*: `genome_masked.fasta` (Input dataset)
>    - In *"Select fasta outputs"* select: `fasta file with spliced exons for each GFF transcript (-y)`
>    - *"full GFF attribute preservation (all attributes are shown)"*: `Yes`
>    - *"decode url encoded characters within attributes"*: `Yes`
>    - *"warn about duplicate transcript IDs and other potential problems with the given GFF/GTF records"*: `Yes`
>
{: .hands_on}

The parameters for running BUSCO on these protein sequences:

> <hands-on-title>BUSCO in proteome mode</hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Sequences to analyse"*: `gffread: pep.fa`
>    - *"Mode"*: `annotated gene sets (protein)`
>    - *"Auto-detect or select lineage?"*: `Select lineage`
>        - *"Lineage"*: `Mucorales`
>    - In *"Advanced Options"*:
>        - *"Which outputs should be generated"*: select all outputs
{: .hands_on}

Several output files are generated:

- short summary: statistical summary of the quality of genomic assembly or annotation, including total number of genes evaluated, percentage of complete genes, percentage of partial genes, etc.
- full table: list of universal orthologs found in the assembled or annotated genome, with information on their completeness, location in the genome, quality score, etc.
- missing buscos: list of orthologs not found in the genome, which may indicate gaps in assembly or annotation.
- summary image: graphics and visualizations to visually represent the results of the evaluation, such as bar charts showing the proportion of complete, partial and missing genes.
- GFF: contain information on gene locations, exons, introns, etc.

This gives information about the completeness of the Braker3 annotation. A good idea is to compare this first result with the one you get on the initial genome sequence, and see if the annotation tool found all the genes that BUSCO finds in the raw genome sequence. To do this, you can restart BUSCO on the masked genome with the same parameters, but selecting the **Genome assemblies (DNA)** mode.

> <comment-title>What can we deduce from these results?</comment-title>
>
> - 94.4% of genes are complete, so the annotation is of high quality in terms of genomic completeness.
> - The duplication rate is low, with 7.6% of genes duplicated which may indicate either real biological duplications or annotation errors.
> - So the Braker3 annotation looks like a good one, with high completeness and low duplication.
{: .comment}

### Helixer annotation

We are going to do the same thing again, but with the Helixer annotation. The first step is to generate the gene sequences predicted by the annotation:

> <hands-on-title>Extract protein sequences</hands-on-title>
>
> 1. {% tool [GFFread](toolshed.g2.bx.psu.edu/repos/devteam/gffread/gffread/2.2.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input GFF3 or GTF feature file"*: output of {% tool [Helixer](toolshed.g2.bx.psu.edu/repos/genouest/helixer/helixer/0.3.3+galaxy1) %}
>    - In *"Reference Genome"* select: `From your history` (Input dataset)
>    - *"Genome Reference Fasta"*: `genome_masked.fasta` (Input dataset)
>    - In *"Select fasta outputs"* select: `protein fasta file with the translation of CDS for each record (-y)`
>    - *"full GFF attribute preservation (all attributes are shown)"*: `Yes`
>    - *"decode url encoded characters within attributes"*: `Yes`
>    - *"warn about duplicate transcript IDs and other potential problems with the given GFF/GTF records"*: `Yes`
>
{: .hands_on}

To run BUSCO on these protein sequences predicted by Helixer annotation:

> <hands-on-title>BUSCO in proteome mode</hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Sequences to analyse"*: `gffread: pep.fa`
>    - *"Mode"*: `annotated gene sets (protein)`
>    - *"Auto-detect or select lineage?"*: `Select lineage`
>        - *"Lineage"*: `Mucorales`
>    - In *"Advanced Options"*:
>        - *"Which outputs should be generated"*: select all outputs
>
{: .hands_on}

> <comment-title>What can we deduce from these results?</comment-title>
>
> - 94.6% of genes are complete, so the annotation is of high quality in terms of genomic completeness.
> - The duplication rate is low, with 1.3% of genes duplicated.
> - So the Helixer annotation looks like a good one, with high completeness and low duplication.
>
{: .comment}

### Compare the two annotations with BUSCO

Braker3 identifies 94.4% complete genes, while Helixer detects 94.6%. Helixer therefore finds slightly more complete genes.

As for duplications, Braker3 detects 7.6%, compared with only 1.3% for Helixer. Helixer's annotation is therefore more precise, with fewer redundant copies.

These differences can be explained by the methods used by each tool to annotate genes.

- BRAKER3 uses an approach based on biological evidence, combining RNA-Seq data and protein alignments. This enables it to identify genes with good accuracy, but can also lead to duplications if several isoforms are poorly distinguished.
- Helixer, on the other hand, relies on a deep learning model that directly analyzes the genomic sequence. This approach delivers a more consistent prediction and reduces duplications, but may be limited if the model has not been well trained on similar genomes.

## Evaluation with **OMArk**

[OMArk](https://github.com/DessimozLab/OMArk) is proteome quality assessment software. It provides measures of proteome completeness, characterises the consistency of all protein-coding genes with their homologues and identifies the presence of contamination by other species. OMArk is based on the OMA orthology database, from which it exploits orthology relationships, and on the OMAmer software for rapid placement of all proteins in gene families.

OMArk's analysis is based on HOGs (Hierarchical Orthologous Groups), which play a central role in its assessment of the completeness and coherence of gene sets. HOGs make it possible to compare the genes of a given species with groups of orthologous genes conserved across a taxonomic clade.

### Braker3 annotation

> <hands-on-title>OMArk on extracted protein sequences</hands-on-title>
>
> 1. {% tool [OMArk](toolshed.g2.bx.psu.edu/repos/iuc/omark/omark/0.3.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Protein sequences"*: `gffread: pep.fa` (Braker3 annotation)
>    - *"OMAmer database*: select `LUCA-v2.0.0`
>    - In *"Which outputs should be generated"*: select `Detailed summary`
>
{: .hands_on}

> <comment-title>What can we deduce from these results?</comment-title>
>
> - Number of conserved HOGs: OMArk has identified a set of 5622 HOGs which are thought to be conserved in the majority of species in the Mucorineae clade.
> - 78.66% of genes are complete, so the annotation is of good quality in terms of genomic completeness.
> - Number of proteins in the whole proteome: 14 209. Of which 77.59% are present and 15.69% of the proteome does not share sufficient similarities with known gene families.
> - No contamination detected.
> - The OMArk analysis is based on the Mucorineae lineage, a more recent and specific clade than that used in the BUSCO assessment, which selected the Mucorales as the reference group.
{: .comment}

### Helixer annotation

> <hands-on-title>OMArk on extracted protein sequences</hands-on-title>
>
> 1. {% tool [OMArk](toolshed.g2.bx.psu.edu/repos/iuc/omark/omark/0.3.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Protein sequences"*: `gffread: pep.fa` (Helixer annotation)
>    - *"OMAmer database*: select `LUCA-v2.0.0`
>    - In *"Which outputs should be generated"*: select `Detailed summary`
>
{: .hands_on}

> <comment-title>What can we deduce from these results?</comment-title>
>
> - Number of conserved HOGs: OMArk has identified a set of 5622 HOGs which are thought to be conserved in the majority of species in the Mucorineae clade.
> - 85.52% of genes are complete, so the annotation is of good quality in terms of genomic completeness.
> - Number of proteins in the whole proteome: 19 299. Of which 62.83% are present and 30.94% of the proteome does not share sufficient similarities with known gene families.
> - No contamination detected.
> - The OMArk analysis is based on the Mucorineae lineage, a more recent and specific clade than that used in the BUSCO assessment, which selected the Mucorales as the reference group.
{: .comment}

### Compare the two annotations with OMArk

Both annotations identify the same set of 5622 HOGs conserved in the majority of species in the Mucorineae clade. Furthermore, no contamination is detected in either case, meaning that no protein has been incorrectly assigned to another lineage.

The annotation obtained with Braker3 displays a percentage of 78.66% complete genes, indicating good annotation quality in terms of genomic completeness. Helixer, on the other hand, shows higher completeness, with 85.52% complete genes, suggesting better genomic coverage.

With regard to the total proteome, Braker3 identified 14,209 proteins, 77.59% of which are consistent with known gene families, while 17.35% of proteins do not share sufficient similarities with these families. Helixer, on the other hand, generates a higher number of proteins (19,299), but only 62.83% of proteins are consistent with known gene families, and 30.94% do not share sufficient similarities. This suggests that Helixer generates a greater number of proteins, but a higher proportion have no clear correspondence with existing gene families.

With our example dataset, Braker3 seems to be a more suitable tool if you're looking for accurate and consistent annotation, with more reliable proteins. Helixer, on the other hand, offers a more complete coverage of the genome, but with a higher number of inconsistent proteins, which may require refinement of the results to improve annotation quality.

# Visualization with a genome browser

You can visualize the annotations generated using a genomic browser like [JBrowse](https://jbrowse.org/jbrowse1.html). This browser enables you to navigate along the chromosomes of the genome, to view the structure of each predicted gene, as well as to compare the genes predicted by each method.

> <hands-on-title>JBrowse visualisation</hands-on-title>
>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: `genome_masked.fasta` (Input dataset)
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Braker3 nnotation`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `gff3` (output of **Braker3** {% icon tool %})
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Helixer nnotation`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `gff3` (output of **Helixer** {% icon tool %})
{: .hands_on}

Click on the newly created dataset's eye to display it. You will see a JBrowse genome browser. You can have a look at the [JBrowse tutorial]({% link topics/visualisation/tutorials/jbrowse/tutorial.md %}) for a more in-depth description of JBrowse.

We have embedded a copy of the resulting JBrowse here, if something went wrong during one of the steps you can always just check this output:

{% capture jbtracks %}DNA%2Caa6aca08d6fd82b94c7a86712a632760_0{% endcapture %}
{% snippet topics/visualisation/faqs/visualizations_jbrowse.html datadir="data" tracks=jbtracks %}

Check that **Braker3 annotation** and **Helixer annotation** are selected in the top left-hand corner. If not, select it to view the annotation.

As confirmed by OMArk, the Braker3 annotation shows a higher duplication rate (7.8%). On JBrowse, these duplications appear as repeated genes in close proximity to each other. If you haven't seen them yet, we recommend you explore scaffold 1 at **scaffold_1:536081..544070**.

Annotation with Helixer seems to generate longer genes than Braker3. This may be explained by its annotation method based on deep neural networks. These longer genes could correspond to extra exons or poorly predicted introns. If you'd like to see an example of this, please visit scaffold 2 at **scaffold_2:494676..508755**.

Helixer also detects a larger number of genes. OMArk analysis showed a higher quantity of proteins in the Helixer annotation (19,299 proteins) compared to Braker3 (14,209 proteins). If you wish to observe this difference, we recommend you explore scaffold 1 at **scaffold_1:585671..591035** or scaffold 13 at **scaffold_13:100856..107419**.

# Conclusion

Congratulations on reaching the end of this tutorial! You now know how to perform a structural annotation of a new eukaryotic genome, using Helixer and Braker3. And you've learned how to evaluate its quality and how to visualize it using JBrowse.

Through this comparative analysis, you've seen that both annotation tools have their strengths with this example dataset:

* Helixer offers a high level of gene completeness and avoids redundant duplications thanks to its deep learning-based approach. It's particularly effective when a consistent, duplication-free prediction is preferred.
* Braker3, leveraging RNA-Seq and protein evidence, delivers more biologically grounded annotations, with a better match to known gene families, making it a solid choice when biological accuracy and interpretability are key. As Braker3 relies on evidence data, the quality of the results depends on the quality and quantity of available data.

When annotating a new genome sequence, it is a good idea to use different methods (Helixer, Braker3, but possibly others like Funannotate, Maker, Braker2, ...), and to perform this kind of comparison to determine the best result.

tools_json:
["RNA STAR", "toolshed.g2.bx.psu.edu/repos/iuc/rgrnastar/rna_star/2.7.11a+galaxy1", "Helixer", "toolshed.g2.bx.psu.edu/repos/genouest/helixer/helixer/0.3.3+galaxy1", "Braker3", "toolshed.g2.bx.psu.edu/repos/iuc/braker3/braker3/3.0.8+galaxy0", "GFFread", "toolshed.g2.bx.psu.edu/repos/devteam/gffread/gffread/2.2.1.4+galaxy0", "Busco", "toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.7.1+galaxy0", "toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.5.0+galaxy0", "OMArk", "toolshed.g2.bx.psu.edu/repos/iuc/omark/omark/0.3.0+galaxy2", "JBrowse", "toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1"]


topic: genome-annotation
tutorial: crispr-screen
title: CRISPR screen analysis
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: ["genome editing", "CRISPR", "essential genes", "cell line"]
content_hash: e3cad506abe0ab343463795a206ff5bb
last_modified: 2026-05-15T22:28:13.499204
zenodo_link: https://zenodo.org/record/5750854
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/crispr-screen/tutorial.html

content:
The {CRISPR} system is a bacterial immune system that has been modified for genome engineering. This groundbreaking technology resulted in a Nobel Prize for Emmanuelle Charpentier and Jennifer Doudna in 2020 ({% cite Uyhazi2021 %}). CRISPR consists of two components: a guide RNA (gRNA) and a non-specific CRISPR-associated endonuclease (Cas9). The gRNA is a short synthetic RNA composed of a scaffold sequence necessary for Cas9-binding (trRNA) and ~20 nucleotide spacer or targeting sequence which defines the genomic target to be modified (crRNA). Cas9 induces double-stranded breaks (DSB) within the target DNA. The resulting DSB is then repaired by either error-prone Non-Homologous End Joining (NHEJ) pathway or less efficient but high-fidelity Homology Directed Repair (HDR) pathway. The NHEJ pathway is the most active repair mechanism and it leads to small nucleotide insertions or deletions (indels) at the DSB site. This results in in-frame amino acid deletions, insertions or frameshift mutations leading to premature stop codons within the open reading frame (ORF) of the targeted gene. Ideally, the end result is a loss-of-function mutation within the targeted gene; however, the strength of the knockout phenotype for a given mutant cell is ultimately determined by the amount of residual gene function.

The ease of generating gRNAs makes {CRISPR} one of the most scalable genome editing technologies and it has been recently utilized for genome-wide screens. These screens enable systematic targeting of 1000s of genes, with one gene targeted per cell, to identify genes driving phenotypes, such as cell survival, drug resistance or sensitivity. It is feasible for any laboratory to perform a CRISPR screen ({% cite Cluse2018 %}) and they are being increasingly used to obtain biological insight ({% cite Bock2022 %}, {% cite Przybyla2021 %}). These days, pooled whole-genome knockout, inhibition and activation CRISPR libraries and CRISPR sub-library pools are commonly screened.

![Illustration of CRISPR Screen Method](../../images/crispr-screen/crispr_screen.jpg "CRISPR knockout and activation methods (from {% cite Joung2016 %})")


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Preparing the reads

## Data upload

Here we will demonstrate analysing {CRISPR} screen using data from {% cite Fujihara2020 %}. There are 3 samples from the human esophageal cancer cell line (OACM5.1): a baseline sample taken at time zero (T0-Control), a sample treated with drug for 8 days (T8-APR-246) and a control sample treated with vehicle for 8 days (T8-Vehicle). The aim is to identify genes whose knockout increases the cancer cells sensitivity to the drug. We will use FASTQ files containing 1% of reads from the original samples to demonstrate the read processing steps.

> <hands-on-title>Retrieve CRISPR screen fastq datasets</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from Zenodo:
>
>    - Open the file {% icon galaxy-upload %} __upload__ menu
>    - Click on __Rule-based__ tab
>    - *"Upload data as"*: `Collection(s)`
>    - Copy the following tabular data, paste it into the textbox and press <kbd>Build</kbd>
>
>      ```
>      T0-Control https://zenodo.org/records/5750854/files/T0-Control.fastq.gz
>      T8-APR-246 https://zenodo.org/records/5750854/files/T8-APR-246.fastq.gz
>      T8-Vehicle https://zenodo.org/records/5750854/files/T8-Vehicle.fastq.gz
>      ```
>
>    ![Rule-based Uploader](../../images/crispr-screen/crispr_rule_uploader.png)
>
>    - From **Rules** menu select `Add / Modify Column Definitions`
>       - Click `Add Definition` button and select `List Identifier(s)`: column `A`
>
>         > <tip-title>Can't find <i>List Identifier</i>?</tip-title>
>         > Then you've chosen to upload as a 'dataset' and not a 'collection'. Close the upload menu, and restart the process, making sure you check *Upload data as*: **Collection(s)**
>         {: .tip}
>
>       - Click `Add Definition` button and select `URL`: column `B`
>
>    - Click `Apply`
>    - In the Name: box type `fastqs` and press <kbd>Upload</kbd>
>
>    ![Rule-based Editor](../../images/crispr-screen/crispr_rule_editor.png)
>
{: .hands_on}

## Raw reads QC

First we'll check the quality of the raw read sequences with [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) and aggregate the reports from the multiple samples with [MultiQC](https://multiqc.info/) ({% cite ewels2016multiqc %}). We will check if the base quality is good and for presence of adapters. For more details on quality control and what the other FastQC plots mean see the ["Quality control" tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).
With CRISPR screens we expect adapter sequence to be present, surrounding the guide. The adapters for the Brunello library vector used with this dataset ([lentiGuide-Puro](https://www.addgene.org/52963/)) are shown below. We'll get FASTQC to check for their presence by providing an adapters file with their sequences.

![Adapter sequences dataset](../../images/crispr-screen/adapter_sequences_dataset.png "Reads from the T8-APR-246 sample with one of the guide sequences highlighted in blue (controlguide1 from Brunello library file). The adapter sequences are in red, directly adjacent to the guide on the right and left. The bases in purple at the start of the reads are stagger regions of different length (0-8bp), required to maintain diversity across the flowcell during sequencing of CRISPR screens.")

> <details-title>Adapters file</details-title>
>
> The adapters file (`adapter_list.tsv`) we use here with FASTQC was created by adding the two CRISPR adapter sequences (5' and 3' of the guide) to the bottom of [the adapters file used by FASTQC](https://github.com/s-andrews/FastQC/blob/master/Configuration/adapter_list.txt). The first 12 bases of the CRISPR adapters were used, as that length is currently recommended in the FASTQC adapters file.
{: .details}


> <hands-on-title>Quality control</hands-on-title>
>
> 1. Import the adapters file from [Zenodo]({{ page.zenodo_link }}) or the Shared Data library (if available):
>    ```
>    https://zenodo.org/records/5750854/files/adapter_list.tsv
>    ```
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 2. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Short read data from your current history"*: fastqs (click "Dataset collection" button on left-side of this input field)
>    - {% icon param-file %}*"Adapter list"*: `adapter_list.tsv`
>
> 3. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy0) %} with the following parameters to aggregate the FastQC reports:
>     - In *"Results"*
>       - *"Which tool was used generate logs?"*: `FastQC`
>       - In *"FastQC output"*
>         - *"Type of FastQC output?"*: `Raw data`
>         - {% icon param-collection %} *"FastQC output"*: `Raw data` files (output of **FastQC**)
>
> 4. Add a tag (`#fastqc-untrimmed`) to the MultiQC webpage to make it easy to differentiate this dataset in the History from other MultiQC reports we will generate.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
> 5. Inspect the webpage output from MultiQC
>
>    > <question-title></question-title>
>    >
>    > 1. What is the read length?
>    > 2. What do you think of the base quality of the sequences?
>    > 3. Are adapters detected?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The read length is 75 bp.
>    > > 2. The base quality is good for the 3 files, as can be seen in the plot below.
>    > >
>    > >    ![Sequence quality](../../images/crispr-screen/fastqc_per_base_sequence_quality_plot.png)
>    > > 3. Yes. In the Adapter Content plot (below) we can see the two CRISPR adapters have been detected. The adapter 5 prime of the sgRNA starts appearing at the beginning of the reads and has been detected in nearly 100% of reads. The adapter 3 prime of the sgRNA appears after base 40. There's also a little Illumina Universal adapter detected towards the ends of some reads which indicate the presence of short fragments, possibly adapter dimers.
>    > >
>    > >    ![Adapter content](../../images/crispr-screen/fastqc_adapter_content_plot.png)
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}


## Trim adapters

We'll trim the adapters from these sequences using [Cutadapt](https://cutadapt.readthedocs.io/en/stable/guide.html) ({% cite marcel2011cutadapt %}). To trim, we'll use the 5' adapter sequence. We don't need to trim the 3' adapter as {MAGeCK} will only use the first 20bp from each read. It determines the number of bases to use automatically from the length of the sequences in the library file, in our case 20bp, or we can specify the sgRNA length.


> <details-title>Adapter trimming</details-title>
>
> In this dataset the adapters start at different positions in the reads, as was shown above. MAGeCK count can trim adapters around the guide sequences. However, the adapters need to start at the same position in each read, requiring the same trimming length, [as described on the MAGeCK website](https://sourceforge.net/p/mageck/wiki/advanced_tutorial/). An example for what MAGeCK expects is shown below. If you used MAGeCK count trimming with the dataset in this tutorial it wouldn't be able to trim the 5' adapter properly and you would only get ~60% reads mapping instead of >80%.
>
> ![Adapters MAGeCK can trim](../../images/crispr-screen/adapter_sequences_mageck.png "Example showing what MAGeCK count expects to be able to auto-detect and trim adapters. Guide sequence is higlighted in blue with the adapter sequences directly adjacent on the right and left. MAGeCK count uses the first 20 bases of each read to map so the sequence after the guide is less important to trim exactly.")
>
> For this dataset, as the adapters are not at the same position in each read, we need to trim the adapters before running MAGeCK count.
{: .details}


> <hands-on-title>Trim adapters</hands-on-title>
>
> 1. {% tool [Cutadapt](toolshed.g2.bx.psu.edu/repos/lparsons/cutadapt/cutadapt/3.5+galaxy0) %} with the following parameters:
>    - *"Single-end or Paired-end reads?"*: `Single-end`
>        - {% icon param-collection %} *"FASTQ/A file #1"*: all fastq.gz files
>        - In *"Read 1 Options"*:
>            - In *"5' (End) Adapters"*:
>                - {% icon param-repeat %} *"Insert 5' (Front) Adapters"*
>                    - *"Source"*: `Enter custom sequence`
>                        - *"Enter custom 5' adapter sequence"*: `TTGTGGAAAGGACGAAACACCG`
>    - *"Outputs selector"*:
>        - *"Report"*: tick
>
> 2. Inspect the Cutadapt report
>
>    > <question-title></question-title>
>    >
>    > For sample T8-APR-246:
>    >
>    > What % of reads contained adapter?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 99.6%
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy0) %} with the following parameters to aggregate the Cutadapt reports:
>     - In *"Results"*
>       - *"Which tool was used generate logs?"*: `Cutadapt/Trim Galore!`
>         - {% icon param-collection %} *"Output of Cutadapt"*: `Report` files (output of **Cutadapt**)
>
> 4. Add a tag (`#cutadapt-report`) to the MultiQC webpage to differentiate this report from the previous (FastQC) one
>
> 5. Inspect the MultiQC report
>
{: .hands_on}

MultiQC produces a 5' trimmed sequences plot where we can check the results are as expected. Here we can see the length of sequence trimmed from the start of the read ranges from 22bp to 30bp with a dip at 27bp. This corresponds to the length of the adapter (22bp) plus stagger sequence (0,1,2,3,4,6,7,8bp) ([see sequencing protocol](https://media.addgene.org/cms/filer_public/61/16/611619f4-0926-4a07-b5c7-e286a8ecf7f5/broadgpp-sequencing-protocol.pdf)). There is no 5bp stagger sequence so 27bp sequences are not expected to be trimmed. The trimmed sequence lengths are what we expect for this dataset and the plot looks similar for all our samples which is good.

![Plot showing lengths of sequences trimmed from 5' end of reads](../../images/crispr-screen/cutadapt_trimmed_sequences_plot_5.png){:width="70%"}


> <hands-on-title>Exercise: Quality control of the polished datasets</hands-on-title>
> Use {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0) %} and {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy0) %} like before, but using the the trimmed datasets produced by Cutadapt as input.
>
>    > <question-title></question-title>
>    >
>    > How did read trimming affect the Adapter Content plot?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > In the Adapter Content section we now don't have any 5' adapter detected. The first 20bp of the reads, what MAGeCK will use for our dataset, has no adapter detected.
>    > >
>    > > ![Plot of trimmed sequences showing no adapter detected in first 20 bases](../../images/crispr-screen/fastqc_adapter_content_plot_trimmed.png){:width="70%"}
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

# Counting

For the rest of the {CRISPR} screen analysis, counting and testing, we'll use the tool called {MAGeCK} ({% cite Li2014 %}, {% cite Li2015 %}).

To count how many guides we have for each gene, we need a library file that tells us which guide sequence belongs to which gene. The guides used here are from the [Brunello library](https://www.addgene.org/pooled-library/broadgpp-human-knockout-brunello/) ({% cite Doench2016 %}) which contains 77,441 sgRNAs, an average of 4 sgRNAs per gene, and 1000 non-targeting control sgRNAs. **The library file must be tab-separated and contain no spaces within the gene or target names**. If necessary, there are tools in Galaxy that can format the file removing spaces and converting commas to tabs.

> <hands-on-title>Count guides per gene</hands-on-title>
> 1. Import the sgRNA library file
>    ```
>    https://zenodo.org/records/5750854/files/brunello.tsv
>    ```
>
> 2. {% tool [MAGeCK count](toolshed.g2.bx.psu.edu/repos/iuc/mageck_count/mageck_count/0.5.9.2.4) %} with the following parameters:
>    - *"Reads Files or Count Table?"*: `Separate Reads files`
>        - {% icon param-collection %} *"Sample reads"*: the `Read 1 Output` (outputs of **Cutadapt**)
>    - {% icon param-file %} *"sgRNA library file"*: the `brunello.tsv` file
>    - In *"Output Options"*:
>        - *"Output Count Summary file"*: `Yes`
>        - *"Output plots"*: `Yes`
>
> 3. We have been using 1% of reads from the samples. Import the MAGeCK count files (sgRNA counts, counts summary and plots pdf) for the full dataset so you can see what results for a real dataset looks like.
>    ```
>    https://zenodo.org/records/5750854/files/kenji_mageck_sgrna_counts.tsv
>    https://zenodo.org/records/5750854/files/kenji_mageck_count_summary.tsv
>    https://zenodo.org/records/5750854/files/kenji_mageck_count_report.pdf
>    ```
>
{: .hands_on}

MAGeCK count outputs:

* a sgRNA Counts file
* a Count Summary file
* a PDF report

## Count Summary file

The contents of the count summary file is [explained on the MAGeCK website](https://sourceforge.net/p/mageck/wiki/output/#count_summary_txt), also shown below. The columns are as follows. To help you evaluate the quality of the data, recommended values from the MAGeCK authors are shown in bold.

Column | Content
--- | ---
File | The fastq (or the count table) file used.
Label | The label of that fastq file assigned.
Reads | Total number reads in the fastq file. **(Recommended: 100~300 times the number of sgRNAs)**
Mapped | Total number of reads that can be mapped to library
Percentage | Mapped percentage, calculated as Mapped/Reads **(Recommended: at least 60%)**
TotalsgRNAs | Total number of sgRNAs in the library
Zerocounts | Total number of missing sgRNAs (sgRNAs that have 0 counts) **(Recommended: no more than 1%)**
GiniIndex | The Gini Index of the read count distribution. A smaller value indicates more eveness of the count distribution. **(Recommended: around 0.1 for plasmid or initial state samples, and around 0.2-0.3 for negative selection samples)**

> <details-title>Gini index</details-title>
>
> Gini index is a measure of inequality from economics. It is used in CRISPR analysis to assess if sgRNAs are present in equal amounts. In positive selection experiments, where only some sgRNAs dominate, the index can be high. However, as discussed in {% cite Li2015 %}, in plasmid library, in early time points, or negative selection experiments, we expect fairly even
> distribution of the remaining sgRNAs that haven't been negatively selected. A high Gini index in these types of sample can indicate CRISPR oligonucleotide synthesis unevenness, low viral transfection efficiency, and overselection, respectively.
{: .details}

> <question-title></question-title>
>
> Is the data quality good for the 3 samples? Use the count summary file for the full dataset, and the recommended values in the table above, to answer these questions.
>
> 1. Have we sequenced enough reads?
> 2. Is the mapped percentage good?
> 3. Is the sgRNA zero count value good?
> 4. Is the Gini Index good?
>
> > <solution-title></solution-title>
> >
> > 1. The number of reads is ok. For example, for T0 control sample we have 17,272,052 reads mapped to guides. We have 77,441 guides so we have ~220 reads per guide (17,272,052/77,441). A minimum of 100 reads per guide, preferably 300, is recommended.
> > 2. Yes, in the summary we have >85% mapped for all 3 samples. {MAGeCK} count does not allow any base mismatches between the reads and the library file, [as described](https://sourceforge.net/p/mageck/wiki/advanced_tutorial/#tutorial-1-allow-mismatches-for-read-mapping) so we expect not all reads will map. Note that we filtered out (6-10%) reads with Cutadapt so we should include those in our unmapped % if we want an accurate count.
> > 3. T0-Control has 0.71% (546/77441 * 100) sgRNAs that have no reads mapped, which is good. The T8 samples are just slightly high at 2.3% (1752/77441 * 100) and 2.8% (2170/77441 * 100).
> > 4. The Gini Index is 0.09 for T0-Control (initial state) which is good. The T8 samples are higher at 0.13 and 0.14 but good for a negative selection experiment.
> >
> {: .solution}
>
{: .question}

MAGeCK count can also generate a PDF with plots that can help assess quality.

> <hands-on-title>Assess mageck count plots</hands-on-title>
>
> Inspect the PDF we imported above.
>
> > <question-title></question-title>
> >
> > What can you determine about the samples from the plots?
> >
> >
> > > <solution-title></solution-title>
> > >
> > > In the boxplots, we can see we have largely similiar distributions of counts for the 3 samples. The greater length of the box and between whiskers in the T8 samples compared to the control tells us we have a bit more variability of counts in those samples, more guides with low and high counts in T8 compared to the T0.
> > > ![MAGeCK count boxplots](../../images/crispr-screen/mageck_count_boxplots.png)
> > >
> > > The Distribution of read counts plot shows us how many guides we have for each count (the frequency sums to the total no. of guides 77,441). Similar to the boxplots, the wider distribution for T8 compared to T0 shows us that those samples have more guides with low and high counts. The peak for T0 appears lower just because it has more points (bins) in the plot.
> > > ![MAGeCK count histograms](../../images/crispr-screen/mageck_count_histogram.png)
> > >
> > > The PCA plot shows us the samples from the different conditions separate well. The T8 samples are a bit more similar to each other, to each other on PC1 axis than to T0. If we had more samples we could use this plot to check for clustering of replicates, batch effect or outliers.
> > > ![MAGeCK count PCA plot](../../images/crispr-screen/mageck_count_pca.png)
> > >
> > > The hierarchical clustering plot also shows us that the T8 samples are a bit more similar to each other than to T0.
> > > ![MAGeCK count hierarchical clustering plot](../../images/crispr-screen/mageck_count_clustering.png)
> > {: .solution}
> >
> {: .question}
>
{: .hands_on}


The paper by {% cite Li2015 %} has more information on MAGeCK quality control.

# Testing

Now that we've generated our guide counts, we'll use MAGeCK test to identify essential genes. Essential means positively or negatively selected sgRNAs and genes. CRISPR positive or negative selection screens can be performed. With a positive selection screen, most cells die after the treatment (selection) and we are interested in identifying genes whose sgRNAs increase and dominate, indicating loss of those genes helps cells survive that treatment. This can help identify genes essential for drug resistance. With a negative selection screen, most cells survive after the treatment. In that case, we are interested in identifying genes whose sgRNAs decrease (drop out) compared to a control (e.g. vehicle), indicating those genes are needed for the cells to survive with that treatment. This can help identify genes essential for drug sensitivity. Regardless of the type of screen performed (positive or negative), MAGeCK can identify both positively and negatively selected genes in the screen ({% cite Li2014 %}). The dataset we are using in this tutorial is from a negative selection screen where the aim is to identify genes whose knockout increases the cancer cells sensitivity to the drug.

![Positive and negative selection screens](../../images/crispr-screen/pos_neg_screen.png "Adapted from Addgene")


## Two conditions

If we want to compare the drug treatment (T8-APR-246) to the vehicle control (T8-Vehicle) we can use MAGeCK test. MAGeCK test uses a robust ranking aggregation (RRA) algorithm ({% cite Li2014 %}).

![Diagram of MAGeCK RRA algorithm](../../images/crispr-screen/mageck_rra_algorithm.png "Overview of the MAGeCK algorithm. Raw read counts corresponding to single-guided RNAs (sgRNAs) from different experiments are first normalized using median normalization and mean-variance modeling is used to capture the relationship of mean and variance in replicates. The statistical significance of each sgRNA is calculated using the learned mean-variance model. Essential genes (both positively and negatively selected) are then identified by looking for genes whose sgRNAs are ranked consistently higher (by significance) using robust rank aggregation (RRA) (from {% cite Li2014 %})")


> <hands-on-title>Test for enrichment</hands-on-title>
>
> 1. {% tool [MAGeCKs test](toolshed.g2.bx.psu.edu/repos/iuc/mageck_test/mageck_test/0.5.9.2.1) %} with the following parameters:
>    - {% icon param-file %} *"Counts file"*: `kenji_mageck_sgRNA_counts.tsv`(the counts file for the full dataset that we imported)
>    - *"Specify Treated samples or Control"*: `Treated samples`
>        - *"Treated Sample Labels (or Indexes)"*: `0`
>    - *"Control Sample Labels (or Indexes)"*: `1`
>    - In *"Output Options"*:
>        - *"Output normalized counts file"*: `Yes`
>        - *"Output plots"*: `Yes`
>
> > <details-title>MAGeCK test sample names</details-title>
> >
> > For MAGeCK test we could specify the samples using their names, which must match the names used in the columns of the counts file, but the hyphens we have in our sample names aren't allowed. We could change the hyphens in the counts file or we can specify the samples by their positions in the counts file, with the first sample column being 0. We do that here.
> >
> {: .details}
>
{: .hands_on}

> <comment-title>Replicates</comment-title>
>
> If we have biological and/or technical replicates we can handle them in a similar way to that described on the [MAGeCK website](https://sourceforge.net/p/mageck/wiki/QA/#how-to-deal-with-biological-replicates-and-technical-replicates).
> For biological replicates, we input them in MAGeCK test Treated Sample Labels/Control Sample Labels fields separated by a comma.
> For technical replicates, we could combine the fastqs for each sample/biological replicate, for example with the **Concatenate datasets** tool, before running MAGeCK count.
>
{: .comment}

> <details-title>Normalization</details-title>
>
> We are using MAGeCK's default normalization method "median" which is more robust to outliers.
> Figure M1 from {% cite Li2014 %} shows a comparison of median ("median") versus total ("total") normalization for two CRISPR screen datasets.
> The distribution of the read counts of significant sgRNAs (FDR=1%) was compared with the mean read count distribution of all sgRNAs (“all”, black). The distribution of the significant sgRNAs should be similar to the distribution of all sgRNAs if the normalization method is unbiased. The difference is small for the leukemia dataset. However, in the melanoma dataset, where a few sgRNAs have very large read counts, the difference is larger, as “total” normalization will prefer sgRNAs with higher read-counts. In contrast, the distribution after “median” normalization is closer to the distribution of all sgRNAs.
>
> ![Median versus Total normalization](../../images/crispr-screen/median_vs_total.png)
>
> If many genes have zero counts, such as after treatment in a positive selection screen, the median may be zero and we may get a message in the MAGeCK log that we have to use total normalization.
>
> Instead of median or total, we could choose to normalize using control guide sgRNAs. However, we would need to know that they're not changing due to the experiment or introducing bias ({% cite Chen2018 %}).
{: .details}

{MAGeCK} test outputs:

* a Gene Summary file
* a sgRNA Summary file
* a PDF report

### Gene Summary file

The Gene Summary file contains the columns described below and a row for each gene targeted by sgRNAs. We have >20,000 genes in the file for this dataset.
We get values for both negative and positive selection. The dataset here is from a negative selection screen so we are most interested in the negative values.
Genes are ranked by the p.neg field (by default). If you need a ranking by the p.pos, you can use the **Sort** data in ascending or descending order tool in Galaxy.

Column name | Content
--- | ---
id | Gene ID
num | The number of targeting sgRNAs for each gene
neg\|score | The RRA lo value of this gene in negative selection
neg\|p-value | The raw p-value (using permutation) of this gene in negative selection
neg\|fdr | The false discovery rate of this gene in negative selection
neg\|rank  | The ranking of this gene in negative selection
neg\|goodsgrna | The number of "good" sgRNAs, i.e., sgRNAs whose ranking is below the alpha cutoff (determined by the --gene-test-fdr-threshold option), in negative selection.
neg\|lfc | The log2 fold change of this gene in negative selection. The way to calculate gene lfc is controlled by the --gene-lfc-method option
pos\|score | The RRA lo value of this gene in positive selection
pos\|p-value | The raw p-value (using permutation) of this gene in positive selection
pos\|fdr | The false discovery rate of this gene in positive selection
pos\|rank  | The ranking of this gene in positive selection
pos\|goodsgrna | The number of "good" sgRNAs, i.e., sgRNAs whose ranking is below the alpha cutoff (determined by the --gene-test-fdr-threshold option), in positive selection.
pos\|lfc | The log fold change of this gene in positive selection


### sgRNA summary file

The sgRNA Summary file contains the columns described below. We can use the sgRNA file to check how the individual guides for genes of interest performed.

Column | Content
--- | ---
sgrna | sgRNA ID
Gene | The targeting gene
control_count | Normalized read counts in control samples
treatment_count | Normalized read counts in treatment samples
control_mean | Median read counts in control samples
treat_mean | Median read counts in treatment samples
LFC | The log2 fold change of sgRNA
control_var | The raw variance in control samples
adj_var | The adjusted variance in control samples
score | The score of this sgRNA
p.low | p-value (lower tail)
p.high  | p-value (higher tail)
p.twosided | p-value (two sided)
FDR | false discovery rate
high_in_treatment | Whether the abundance is higher in treatment samples


### PDF report

The PDF shows plots of the top 10 negatively and positively selected genes.
We can see the top genes ranked by RRA scores or p value. These values come from the gene summary file.

![MAGeCK test RRA scores](../../images/crispr-screen/mageck_test_rra_scores.png)

> <question-title></question-title>
>
> What are the top 3 negatively selected genes by P value?
>
> > <solution-title></solution-title>
> >
> > ESD, MTHFD1L and SHMT2, the same as by RRA score. RRA score is like p-value. Those genes are part of the glutathione pathway which was found to be altered in the published paper for this dataset.
> >
> > ![MAGeCK test pvalues](../../images/crispr-screen/mageck_test_pvalues.png)
> >
> {: .solution}
>
{: .question}

The PDF also shows plots with the sgRNA counts for the top 10 genes. These values are the normalized counts for each sgRNA from the sgRNA summary file. With these plots we can see if the counts of all the sgRNAs for these top genes are changing similarly.

![Plot showing guide counts for ESD gene](../../images/crispr-screen/esd_plot.png){:width="50%"}

> <question-title></question-title>
>
> 1. Are all the sgRNAs for FLI1 (the 4th top neg gene) changing similarly?
> 2. In the gene summary table, how many of the negatively selected genes have FDR < 0.05?
>
> > <solution-title></solution-title>
> >
> > 1. No. We can see in this case that, while one sgRNA is a lot lower in the APR treated sample compared to the vehicle, one increases a little, and the other two sgRNAs don't change much. So we might conclude that this gene is not strongly negatively selected.
> > 2. None. One reason for this is likely the large number of genes being tested (>20,000). You could try to [increase sensitivity with the procedures described](https://sourceforge.net/p/mageck/wiki/QA/#i-see-very-few-genes-that-are-below-the-certain-fdr-cutoff-like-010-why-it-is-that-and-what-should-i-do).
> >
> > ![Plot showing guide counts for FLI1 gene](../../images/crispr-screen/fli1_plot.png){:width="50%"}
> >
> {: .solution}
>
{: .question}


### Visualisation

In addition to the visualisations automatically generated by MAGeCK in the PDF, we can create a volcano plot to further visualise the output. With a volcano plot we plot the magnitude of change for drug treatment versus vehicle control (lfc) versus significance (p-value). As we have two columns for lfc and p-value, one for negative selection and one for positive, we first combine these into one column for each using the **awk** tool. If the `neg|p-value` is smaller than the `pos|p-value` the gene is negatively selected. If the `neg|p-value` is larger than the `pos|p-value` the gene is positively selected. Then we create the plot using the **Volcano plot** tool.


> <hands-on-title>Create volcano plot</hands-on-title>
> 1. {% tool [Text reformatting with awk](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_awk_tool/1.1.2) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `MAGeCK test Gene Summary`
>    - *"AWK Program"*: Copy and paste the text in the grey box below into this field
>
>    ```
>    # Print new header for first line
>    NR == 1 { print "gene", "pval", "fdr", "lfc" }
>
>    # Only process lines after first
>    NR > 1 {
>        # check if neg pval (column 4) is less than pos pval (column 10)
>        if ($4 < $10){
>           # if it is, print negative selection values
>            print $1, $4, $5, $8
>        } else {
>           # if it's not, print positive selection values
>            print $1, $10, $11, $14
>        }
>    }
>    ```
>  2. Inspect the file output. It should look like below.
> ```
> gene    pval        fdr       lfc
> ESD     1.9454e-05  0.324257  -1.9283
> MTHFD1L 3.226e-05   0.324257  -0.89778
> SHMT2   7.6093e-05  0.454208  -1.1328
> FLI1    9.0376e-05  0.454208  -0.08872
> ```
>
> 3. {% tool [Volcano Plot](toolshed.g2.bx.psu.edu/repos/iuc/volcanoplot/volcanoplot/0.0.5) %} to create a volcano plot
>    - {% icon param-file %} *"Specify an input file"*: the Text reformatting output file
>    - {% icon param-file %} *"File has header?"*: `Yes`
>    - {% icon param-select %} *"FDR (adjusted P value)"*: `Column 3`
>    - {% icon param-select %} *"P value (raw)"*: `Column 2`
>    - {% icon param-select %} *"Log Fold Change"*: `Column 4`
>    - {% icon param-select %} *"Labels"*: `Column 1`
>    - {% icon param-select %} *"Points to label"*: `Significant`
>        - {% icon param-text %} *"Only label top most significant"*: `10`
>
> 4. Inspect the plot in the PDF output.
>
>    > <question-title></question-title>
>    >
>    > What is the most significant gene?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ATP5E as it is the gene nearest the top of the plot.
>    > >
>    > > ![Volcano plot for APR compared to vehicle](../../images/crispr-screen/volcanoplot.png){:width="50%"}
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> For more details on using the volcano plot tool, see the [tutorial here]({% link topics/transcriptomics/tutorials/rna-seq-viz-with-volcanoplot/tutorial.md %}). For how to customise the volcano plot tool output using R, see the [tutorial here]({% link topics/transcriptomics/tutorials/rna-seq-viz-with-volcanoplot-r/tutorial.md %}).
>
{: .hands_on}


### Pathway analysis

We can perform pathway analysis on the results to identify pathways that are changing with the treatment. MAGeCK has a pathway analysis module, however, we will use the tool **fgsea** as it outputs some visualisations and the genes enriched in the pathway. For fgsea we need a ranked list of genes and a pathways file in GMT format. We can get the ranked list from our mageck results, we'll use the RRA score column for negative results. For the pathways, we'll use the Hallmark pathways from MSigDB, which can be a [good starting point for pathway exploration](http://www.gsea-msigdb.org/gsea/msigdb/collection_details.jsp).

> <hands-on-title>Perform gene set enrichment with fgsea</hands-on-title>
>
> 1. Import the Hallmark pathways file [Zenodo]({{ page.zenodo_link }}) or the Shared Data library (if available). Set the Type to tabular:
>    ```
>    https://zenodo.org/record/5750854/files/h.all.v7.4.symbols.gmt
>    ```
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
> 2. **Cut columns from a table** {% icon tool %} with
>    - {% icon param-text %} *"Cut columns"*: `c1,c3` (the gene symbols and neg score)
>    - {% icon param-select %} *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: the MAGeCK Gene Summary file
> 3. **fgsea** {% icon tool %} with
>    - {% icon param-file %} *"Ranked Genes"*: the output of **Cut** {% icon tool %}
>    - {% icon param-check %} *"File has header?"*: `Yes`
>    - {% icon param-file %} *"Gene Sets"*: `h.all.v7.4.symbols.gmt` (this should be `tabular` format, if not, see how to change it in the Tip below)
>    - {% icon param-text %} *"Minimum Size of Gene Set"*: `15`
>    - {% icon param-check %} *"Output plots"*: `Yes`
>
>    > <question-title></question-title>
>    >
>    > What is the top ranked pathway?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > Oxidative phosphorylation.
>    > > You can see some explanation of fgsea output in the [RNA-seq genes to pathways" tutorial]({% link topics/transcriptomics/tutorials/rna-seq-genes-to-pathways/tutorial.md %}).
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}


## (Optional) Multiple conditions

*It takes about 30 mins to run mageck mle on the samples below so this part of the tutorial is optional, you can try it if time allows or in your own time.*

If we have more than two conditions to compare, or a complex experimental design, we can use MAGeCK mle. MAGeCK mle uses a maximum likelihood estimation (MLE) algorithm ({% cite Li2015 %}). It outputs a single value (beta score) per gene instead of a score for both negative and positive selection. A negative beta score indicates negative selection and a positive indicates positive selection. MAGeCK mle can also be used for comparing 2 conditions instead of MAGeCK test (RRA) but it is slower.

To demonstrate using MAGeCK mle, we will compare the drug treatment (T8-APR-246) to T0 and the vehicle (T8-Vehicle) to T0.
We'll use [a design matrix file](https://sourceforge.net/p/mageck/wiki/input/#design-matrix-file).
The rules of the design matrix are:

* The design matrix file must include a header line of condition labels
* The first column is the sample labels that must match labels in read count file
* The second column must be a "baseline" column that sets all values to "1"
* The value in the design matrix is either "0" or "1"
* You must have at least one sample of "initial state" (e.g., day 0 or plasmid) that has only one "1" in the corresponding row. That only "1" must be in the baseline column
* The initial state (baseline) sample must be the 1st sample listed, under the header row, for mageck mle

The file for our samples looks like below. We have 3 conditions: baseline (T0), Vehicle and APR. The 1st row indicates that T0 is our baseline sample, the 2nd row indicates that we want to compare Vehicle to T0, and the 3rd row indicates that we want to compare APR to T0.

```
Samples     baseline  Vehicle APR
T0-Control  1         0       0
T8-Vehicle  1         1       0
T8-APR-246  1         0       1
```

Examples of more complicated design matrices, for e.g. time series experiments, can be seen on the [MAGeCK website](https://sourceforge.net/p/mageck/wiki/advanced_tutorial/#tutorial-4-make-full-use-of-mageck-mle-for-more-complicated-experimental-design-eg-paired-samples-time-series).

> <hands-on-title>Test for enrichment with MLE</hands-on-title>
> 1. Import the design matrix file from [Zenodo]({{ page.zenodo_link }}) or the Shared Data library (if available):
>    ```
>    https://zenodo.org/record/5750854/files/kenji_mageck_mle_design_matrix.tsv
>    ```
>
> 2. {% tool [MAGeCKs mle](toolshed.g2.bx.psu.edu/repos/iuc/mageck_mle/mageck_mle/0.5.9.2.1) %} with the following parameters:
>    - {% icon param-file %} *"Counts file"*: the `kenji_mageck_sgrna_counts.tsv` file
>    - *"Design matrix or sample labels"*: `Design matrix`
>        - {% icon param-file %} *"Design matrix file"*: the `mageck_mle_design_matrix` file
>
{: .hands_on}

MAGeCK [mle Gene Summary output](https://sourceforge.net/p/mageck/wiki/output/#gene_summary_txt-in-mle-subcommand) is described below.

Column | Content
--- | ---
Gene | Gene ID
sgRNA | The number of targeting sgRNAs for each gene
Condition_name\|beta | The beta score of this gene in condition (e.g. Vehicle or APR). The conditions are specified in the design matrix input to mle.
Condition_name\|p-value | The raw p-value (using permutation) of this gene
Condition_name\|fdr | The false discovery rate of this gene
Condition_name\|z  | The z-score associated with Wald test
Condition_name\|wald-p-value | The p value using Wald test
Condition_name\|wald-fdr | The false discovery rate of the Wald test

Similar to what we did with the MAGeCK test output, we can create a volcano plot to visualise the results, showing the magnitude of change (beta score) and significance (p-value).

> <hands-on-title>Visualize MLE results</hands-on-title>
> 1. Create a volcano plot to visualise the result for APR vs T0. Use the gene summary file columns `Gene` `APR|beta` `APR|wald-p-value`  `APR|wald-fdr`
>    - {% tool [Volcano Plot](toolshed.g2.bx.psu.edu/repos/iuc/volcanoplot/volcanoplot/0.0.5) %} to create a volcano plot
>        - {% icon param-file %} *"Specify an input file"*: the MAGeCK mle Gene Summary file
>        - {% icon param-select %} *"FDR (adjusted P value)"*: `Column 14`
>        - {% icon param-select %} *"P value (raw)"*: `Column 13`
>        - {% icon param-select %} *"Log Fold Change"*: `Column 9`
>        - {% icon param-select %} *"Labels"*: `Column 1`
>        - {% icon param-select %} *"Points to label"*: `Significant`
>        - {% icon param-text %} *"Only label top most significant"*: `10`
>
>   ![Volcano plot for APR compared to timepoint zero](../../images/crispr-screen/mageck_mle_apr_volcano.png){:width="50%"}
>
>    Similarly, you could create a plot for the vehicle vs T0 using the gene summary file columns `Gene` `Vehicle|beta` `Vehicle|wald-p-value`  `Vehicle|wald-fdr`
>
{: .hands_on}


> <tip-title>Getting help</tip-title>
>
> For questions about using Galaxy, you can ask in the [Galaxy help forum](https://help.galaxyproject.org/). For questions about MAGeCK, you can ask in the [MAGeCK Google group](https://groups.google.com/g/mageck).
>
{: .tip}


# Conclusion


{CRISPR} Screen reads can be assessed for quality using standard sequencing tools such as FASTQC, MultiQC and trimmed of adapters using Cutadapt. The detection of enriched guides can be performed using {MAGeCK}. Downstream analysis can include visualisations, such as volcano plot, and pathway analysis with tools like fgsea.

# Acknowledgements


Thanks to Mehmet Tekman for suggesting the awk tool. Thanks also to Jennifer Devlin, Lydia Lim and Sylvia Mahara for comments and feedback on the tutorial.

tools_json:
["FastQC", "toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy0", "Cutadapt", "toolshed.g2.bx.psu.edu/repos/lparsons/cutadapt/cutadapt/3.5+galaxy0", "MAGeCK count", "toolshed.g2.bx.psu.edu/repos/iuc/mageck_count/mageck_count/0.5.9.2.4", "MAGeCKs test", "toolshed.g2.bx.psu.edu/repos/iuc/mageck_test/mageck_test/0.5.9.2.1", "Text reformatting with awk", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_awk_tool/1.1.2", "Volcano Plot", "toolshed.g2.bx.psu.edu/repos/iuc/volcanoplot/volcanoplot/0.0.5", "MAGeCKs mle", "toolshed.g2.bx.psu.edu/repos/iuc/mageck_mle/mageck_mle/0.5.9.2.1"]


topic: genome-annotation
tutorial: hpc-for-lsgc
title: From small to large-scale genome comparison
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: ["plants", "prokaryote"]
content_hash: c9938a00c03a974569c4ea75876242df
last_modified: 2026-05-15T22:28:13.510274
zenodo_link: https://doi.org/10.5281/zenodo.4485547
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/hpc-for-lsgc/tutorial.html

content:
Sequence comparison is a core problem in bioinformatics. It is used widely in evolutionary studies, structural and functional analyses, assembly, metagenomics, etc. Despite its regular presence in everyday Life-sciences pipelines, it is still not a trivial step that can be overlooked. Therefore, understanding how sequence comparison works is key to developing efficient workflows that are central to so many other disciplines.

In the following tutorial, we will learn how to compare both small and large sequences using both seed-based alignment methods and alignment-free methods, and how to post-process our comparisons to refine our results. Besides, we will also learn about the theoretical background of sequence comparison, including why some tools are suitable for some jobs and others are not.

This tutorial is divided into two large sections:

 - Fine-grained sequence comparison: In this part of the tutorial we will use `GECKO` ({% cite GECKO %}) to perform sequence alignment between small sequences. We will also identify, extract and re-align regions of interest.
 - Coarse-grained sequence comparison: In this part of the tutorial, we will tackle on how to compare massive sequences using `CHROMEISTER` ({% cite perez2019ultra %}), an alignment-free sequence comparison tool. We will generate visualization plots for the comparison of large plant genomes and automatically detect large-scale rearrangements.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Fine-grained sequence comparison

Imagine you are working on an evolutionary study regarding the species *Mycoplasma hyopneumoniae*. In particular, you are interested in the strains `232` and `7422` and wish to compare their DNA sequence to know more about the evolutionary changes that took place between both. The workflow you will follow starts with (1) acquiring the data, (2)  getting it ready for Galaxy, (3) running the comparison and (4) inspecting and working with the resulting alignments. Let's go!

## Preparing the data

First we will be uploading the data to Galaxy so that we can run our tools on it.

> <hands-on-title>Mycoplasma data upload</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a descriptive name (e.g. "Mycoplasma comparison hands-on")
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import `mycoplasma-232.fasta` and `mycoplasma-7422.fasta` from [Zenodo](https://zenodo.org/record/4485547#.YBj8XHmCGUk).
>
>    ```
>    https://zenodo.org/record/4485547/files/mycoplasma-232.fasta
>    https://zenodo.org/record/4485547/files/mycoplasma-7422.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    As default, Galaxy takes the link as name, so rename them.
>
> 3. Rename the files to `232.fasta` and `7422.fasta` and change the datatype if needed to `fasta` (Galaxy will auto-discover the format of the files).
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
{: .hands_on}

If you were successful, both sequences should now be available as `.fasta` datasets in your history.


> <question-title></question-title>
>
> 1. What do you think about the size of the sequences in regards to the difficulty of comparing them? (You can check the size of the files by clicking on the information icon)
>
> > <solution-title></solution-title>
> > 1. It always depends on the focus of our study. For instance, if we were looking for optimal alignments, two 1 MB sequences are indeed large enough to make most approaches either fail or take a decent amount of time and resources. On the other hand, if we were looking for seed-based local alignments (e.g. `GECKO` or `BLAST` ({% cite BLAST %}) ), the comparison would require merely seconds (check the slides for more information).
> >
> {: .solution }
>
{: .question}

As we discussed in the previous section, running optimal aligning tools on such "small" data can be difficult (in fact, tools such as the well-known `EMBOSS needle` will require cuadratic amounts of time and memory, whereas tools such as `EMBOSS strecher` will require quadratic time ({% cite myers1988optimal %})). These limitations can make it impractical in many situations. Therefore, we will now learn how to overcome these limitations by employing seed-based methods, particularly `GECKO`.

## Running the comparison

We will now run a comparison between *Mycoplasma hyopneumoniae 232* and *Mycoplasma hyopneumoniae 7422* in Galaxy using `GECKO`.

> <hands-on-title>Comparing two mycoplasmas with GECKO</hands-on-title>
> 1. {% tool [GECKO](toolshed.g2.bx.psu.edu/repos/iuc/gecko/gecko/1.2) %} with the following parameters
>    - {% icon param-file %} *"Query sequence"*: `232.fasta`
>    - {% icon param-file %} *"Reference sequence"*: `7422.fasta`
>    - *"K-mer seed size"*: `16`
>    - *"Minimum length"*: `50`
>    - *"Minimum similarity"*: `60`
>    - *"Generate alignments file?"*: `Extract alignments (CSV and alignments file)`
> 2. Check out the files that have been generated, i.e. the `CSV` and the `Alignments` file.
>    > <question-title></question-title>
>    >
>    > 1. What information is provided in the `CSV` file?
>    > 2. And in the `Alignments` file?
>    > 3. What happens if we re-run the experiment with other parameters (e.g. change `Minimum length` to `5000` and `Minimum similarity` to `95`)?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The `CSV` file contains a summary of the detected High-Scoring Segment Pairs (HSPs) or alignments. The file is divided in a few rows of metadata (e.g. containing the sequence names) and one row per alignment detected. Check out the `GECKO` help (bottom of the tool {% icon tool %} page to know what each column does!
>    > > 2. The `Alignments` file contains the actual regions of the query and reference sequence for each alignment detected. With this file, you can investigate individual alignments, find mutations and differences between the sequences, extract the aligned part, etc.
>    > > 3. Changing the parameters affect the number of alignments GECKO will detect. In fact, if we use parameters that are too restrictive, then we might not get any alignments at all! On the other hand, if we use parameters that are too permissive, then we might get a lot of noise in the output. Thus, it is very important to understand what the parameters do. Never leave your parameters default, always know what they do! Check out the help section to get information about the parameters.
>    >  {: .solution }
>    {: .question}
>
>    > <comment-title>Note about parameters</comment-title>
>    > 1. Besides the `minimum length` and `minimum similarity` parameters, which are applied as a filter after the comparison is completed, the `k-mer seed size` is an internal parameter that regulates the number of seeds that can be found. A smaller `k-mer` size will translate into detecting more seeds, which are starting points for alignments.
>    > 2. Notice that this parameter must be taken into account depending on the type of sequence that we are working with. For example, if we set `k-mer seed size` to `32`, then `GECKO` will attempt to find alignments that start with 32 consecutive, equally aligned base pairs between the two sequences. If your sequences are far away from each other (in terms of evolutionary distance) then it will be very hard to find such 32 consecutive base pairs!
>    > 3. Some advice: this parameter can affect performance greatly. For instance, setting `k-mer seed size` to `8` in the case of chromosomes can require much more computing time than using for instance `32`. On the other hand, if we set it to `32` in the case of small bacterial sequences, we might not find any alignments!
>    {: .comment}
>
{: .hands_on}

## Post-processing: extracting alignments

We will now use additional tools to post-process our alignments. Consider the case where you are interested in a set of DNA repeats located at a given position in the sequences.

> <comment-title>Note about interactive exploration of alignments</comment-title>
> In a different scenario where e.g. you do not know what to look for or need to find the location of particular alignments, and you need an interactive exploration, you can also use `GECKO-MGV` to ease the process. `GECKO-MGV`generates a visual representation where you can zoom in and out of the alignments, select and export as FASTA, etc., find more in the `GECKO-MGV` article ({% cite diaz2019combining %}) or run it with a dockerized container, as described in the [GitHub repository](https://github.com/estebanpw/docker-geckomgv).
{: .comment}


For our current experiment, we will be looking for the following set of repeats:

![Mycoplasma hyopneumoniae comparison example](../../images/hpc-for-lsgc/GeckoMGV01.PNG "(Left) Sequence comparison between <em>Mycoplasma hyopneumoniae</em> 232 and 7422. (Right) Zoomed-in region where the repeats of interest are located.")

Let's extract the repeats highlighted in red (Figure 1, right) which are aligned to the position 19,610 in the query sequence and perform a multiple sequence alignment on them to check if there are any evolutionary differences. The sequences will be extracted from the reference (i.e. *Mycoplasma hyopneumoniae 7422*) since this is where the repeats duplicate in respect to the query sequence (notice that in Figure 1 we are selecting the ones stacked vertically).

### Extracting repeat alignments and running Multiple Sequence Alignment

> <hands-on-title>Multiple Sequence Alignment of a set of repeats</hands-on-title>
> 1. Search for the tool {% tool [Text reformatting with AWK](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_awk_tool/1.1.2) %} and run it with the following parameters
>    - {% icon param-file %} *"File to process"*: `Gecko on data 2 and 1: Alignments` (Note: if you have run the previous experiment with different parameters, make sure to select here the alignments file corresponding to the first experiment, and make sure you ran it with the same parameters!)
>    - *"AWK Program"*: `BEGIN{FS=" "} /@\(196[0-9][0-9]/ { printf(">sequence%s%s\n", $(NF-1), $NF); getline; while(substr($0,1,1) != ">"){ if(substr($0,1,1) =="Y"){ print $2; } getline; } } ` (Paste this code into the text box)
>
>    > <comment-title>Note about AWK</comment-title>
>    > Although the AWK script looks a bit threatening, it is very simple:
>    >  - The `BEGIN{FS=" "}` tells it to separate fields by a space.
>    >  - The `/@\(196[0-9][0-9]/` is a regular expression that indicates we are looking for a line containing the string `@(196xx` where `xx` are two random digits between 0 and 9. This is used to identify the repeats that start at coordinates 19,600 to 19,699.
>    >  - The previous regular expression triggers an action. The first part `printf(">sequence%s%s\n", $(NF-1), $NF); getline` prints out the sequence ID along with the coordinates and moves on to the next line.
>    >  - The second part `while(substr($0,1,1) != ">"){ if(substr($0,1,1) =="Y"){ print $2; } getline;` scans the next lines and prints the nucleotide sequence corresponding to the `Y` sequence (the reference), up until a `>` is found, which means we are done with the sequence.
>    {: .comment}
>
> 2. Change the name of the output file `Text reformatting on data ...` to `repeats` Change the datatype to `.fasta`.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
> 3. {% tool [ClustalW](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_awk_tool/1.1.2) %} with the following parameters
>    - {% icon param-file %} *"FASTA file"*: `repeats.fasta`
>    - *"Data type"*: `DNA nucleotide sequences`
>    - *"Output alignment format"*: `Native Clustal output format`
>    - *"Show residue numbers in clustal format output"*: `No`
>    - *"Output order"*: `Aligned`
>    - *"Output complete alignment"*: `Complete alignment`
>
> 4. Inspect the output file `ClustalW on data ... clustal`.
>    - The file is divided into several blocks.
>    - Each block contains the next 50 contiguos nucleotides of each alignment in one line per sequence.
>    - Gaps are included.
>    - The asterisk symbol `*` indicates that there is consensus across all sequences in the current position.
> 5. Another file `ClustalW on data ... dnd` is also generated which can be used to view the output alignment as a tree.
>
{: .hands_on}

Your output alignment file `ClustalW on data ... clustal` should look similar to the following one (click on {% icon galaxy-eye %}) :

```
CLUSTAL 2.1 multiple sequence alignment


sequence@_19610_31252_      AAAATAAAATCAGAATTTCGTGAAAAAGCACGTAAAATAGCGACTCTTTG
sequence@_19610_28814_      AAAATAAAATCAGAATTTCGTGAAAAAGCACGTAAAATAGCGACTCTTTG
sequence@_19610_25551_      AAAATAAAATCTGAATTTCGCGAAAAAGCACGTAAAATACCGACTCTTTG
                            *********** ******** ****************** **********

sequence@_19610_31252_      TTTTTCACCACCGGATAAATCTTTGACTTGTTGGTTTAGTTTGTCATTTT
sequence@_19610_28814_      TTTTTCACCACCGGATAAATCTTTGACTTGTTGGTTTAGTTTGTCATTTT
sequence@_19610_25551_      TTTCTCGCCACCCGATAAATCTTTGACTTGTTGATTTAGTTTTTCATTTT
                            *** ** ***** ******************** ******** *******

sequence@_19610_31252_      CGATATTGAGGAAATTTGCACTTTTTTCAAGTAAGACTTGGTCAAATTCC
sequence@_19610_28814_      CGATATTGAGGAAATTTGCACTTTTTTCAAGTAAGACTTGGTCAAATTCC
sequence@_19610_25551_      CAATATTAAGAAAATTTGCTCTTTTTTCAAGCAAATTTTGATCAAATTCT
                            * ***** ** ******** *********** **   *** ********

sequence@_19610_31252_      CTTTTAATTATGTTATTTCCAATTAAAATATTATTTTTAACCGATAAATT
sequence@_19610_28814_      CTTTTAATTATGTTATTTCCAATTAAAATATTATTTTTAACCGATAAATT
sequence@_19610_25551_      TTTTGAATTAAGCTATTTCCAATTAAAATATTATTTTTAACAGAAAGATT
                             *** ***** * **************************** ** * ***

sequence@_19610_31252_      CTCAATCAAATTAAAATCTTGAAAAACTACATCAATTAAAGGGTTTTTTT
sequence@_19610_28814_      CTCAATCAAATTAAAATCTTGAAAAACTACATCAATTAAAGGGTTTTTTT
sequence@_19610_25551_      TTCAATTAGGTTAAAATCCTGAAAAACTACATCAATTAAAGGATTTTTTT
                             ***** *  ******** *********************** *******

sequence@_19610_31252_      CTTCTTTACCTT
sequence@_19610_28814_      CTTCTTTACCTT
sequence@_19610_25551_      CTTCTTT-----
                            *******

```
We can see that there are several locus along the duplicated repeats where single point mutations have taken place, and that the ending of the last repeat is missing.



So far we have learnt how to run our own custom sequence comparison, extract a set of DNA repeats and perform multiple sequence alignment on them (MSA). While this represents an common example of a bioinformatics pipeline, in the next part of the tutorial we will jump to a large-scale comparison scenario where we do not need fine-grained alignments, but instead we will be looking at the "bigger picture" using alignment-free methods.

# Coarse-grained sequence comparison

In this second part of the tutorial, we are going to work with chromosome-sized sequences. While we can still use `GECKO` for chromosomes, it will take some more time and resources. However, consider now the case where you do not need alignments, but rather are in one of the following cases:

- You are working with a very large *de novo* assembly genome for which you do not know the strain and thus you need to compare it to several candidates.
- You want to study large evolutionary rearrangements at a block level rather than alignment level at several species at once.
- You want to generate a dotplot between several massive chromosomes from e.g. plants to find the main syntenies, but your sequences are full of repeats and other computational approaches fail.

As you can imagine, all three scenarios require several large-scale sequence comparisons. While there are also other ways to approach these situations, in this tutorial we will learn how we can use `CHROMEISTER` which is specifically designed for large-scale genome comparison without alignments (alignment-free method).

## Introduction

Before jumping on the hands-on, let us see an example of the second case: say we wanted to study the evolutionary rearrangements between the grass and common wheat genomes (*Aegilops tauschii* and *Triticum aestivum*). The following plot shows the comparison:

![Large-scale genome comparison example](../../images/hpc-for-lsgc/Chrom07.png "Chromosome comparison for <em>Aegilops tauschii</em> and <em>Triticum aestivum</em>. Since both genomes have seven chromosomes, we would require seven times seven comparisons, i.e. 49 chromosome comparisons with each sequence being over 500 MBs in size!")

Now there are some things to note here: (1) these alignment-free comparisons can be performed very quickly, (2) we can visually see the large rearrangements and (3) we can identify which chromosome comparisons actually have any signal. This last point is interesting because it allows us to use `CHROMEISTER` to quickly identify which comparisons are worth to perform with more accurate tools and avoid lots of unnecessary computation (in this case only the diagonal has similarity, but it might not be the case, *e.g.* *Homo sapiens* and *Mus musculus*).

Let us now jump into the hands-on! We will learn how to compare chromosomes with `CHROMEISTER`, particularly from the grass and common wheat genomes, which are nearly 5 times larger than the average human chromosome!

## Preparing the data

> <hands-on-title>Chromosome data upload</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a descriptive name (e.g. "Chromosome comparison hands-on")
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import `aegilops_tauschii_chr1.fasta` and `triticum_aestivum_chr1.fasta` from [Zenodo](https://zenodo.org/record/4485547#.YBj8XHmCGUk).
>
>    ```
>    https://zenodo.org/record/4485547/files/aegilops_tauschii_chr1.fasta
>    https://zenodo.org/record/4485547/files/triticum_aestivum_chr1.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    As default, Galaxy takes the link as name, so rename them.
>
> 3. Rename the files to `aegilops.fasta` and `triticum.fasta` and change the datatype to `fasta` if needed.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
>    > <comment-title>Note on the name of the chromosomes</comment-title>
>    > Please notice that these chromosomes are not labelled as "chromosome 1" at their original sources. We have renamed them for simplicity.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. We already discussed the mycoplasma sequences in regards to their size. What do you think about the size of the two chromosomes that we will be comparing now from a computational standpoint?
>
> > <solution-title></solution-title>
> > 1. Chromosome-like sequences are arguably some of the largest DNA sequences you can find. Regarding the comparison, optimal chromosome comparison typically requires either large clusters to be run or special hardware accelerators (such as GPUs). On the other hand, for seed-based local alignment we can still use common approaches, but they will take a considerable amount of time. Finally, if we use alignment-free methods such as `CHROMEISTER`, we can perform a comparison in few minutes (check the slides for more information) instead of hours.
> >
> {: .solution }
>
{: .question}

## Running the comparison

> <hands-on-title>Comparing two plant chromosomes with CHROMEISTER</hands-on-title>
> 1. {% tool [CHROMEISTER](toolshed.g2.bx.psu.edu/repos/iuc/chromeister/chromeister/1.5.a) %} with the following parameters
>    - {% icon param-file %} *"Query sequence"*: `aegilops.fasta`
>    - {% icon param-file %} *"Reference sequence"*: `triticum.fasta`
>    - *"Output dotplot size"*: `1000`
>    - *"K-mer seed size"*: `32`
>    - *"Diffuse value"*: `4`
>    - *"Add grid to plot for multi-fasta data sets"*: `No`
>    - *"Generate image of detected events"*: `Yes`
> 2. Run the job and wait for the results. It should take around ~3 minutes.
> 3. Let's inspect the output files:
>    - *"Comparison matrix"*: This file is the "core" of the comparison. It contains the heuristically matched seeds sampled per section of the chromosomes. This file is used for custom post-processing.
>    - *"Comparison dotplot"*: This file is a `.png` image representing the pairwise comparison.
>    - *"Comparison metainformation"*: This CSV file contains information about the comparison such as name of the sequences compared, length, etc. It is particularly useful when using multi-fasta inputs.
>    - *"Detected events"*: This CSV file contains information regarding rearrangements automatically detected.
>    - *"Detected events plot"*" This file is a `.png` image showing the detected events.
>    - *"Comparison score"*: This text file contains the scoring distance between the query and the reference.
>
>    > <comment-title>About parameters</comment-title>
>    >
>    > Some of the parameters used in `CHROMEISTER` are similar to those in `GECKO`, but others are new, in particular:
>    > 1. *"Output dotplot size"*: This number represents the width and height of the resulting comparison plot, but it also affects the degree of precision with which alignment-free blocks are calculated. A value of `1000` is adequate for mostly everything (resulting in a `1000x1000` output doplot), but for extremely large sequences a value of `2000` will yield more resolution since less blocks will be grouped together.
>    > 2. *"K-mer seed size"*: This value is the same as in `GECKO`. Generally, `32` is a good choice for any sequence that is not very small such as the mycoplasmas from the previous part of the tutorial, which were around 1 Mbp in size.
>    > 3. *"Diffuse value"*: This parameter regulates the stringency with which alignment-free seeds are matched. A diffuse value of `1` means perfect matching, whereas a value of `4` means match only a fourth of the seed (taken in uniform steps). The value of `4` is recommended for everything except when signals start to deteriorate in large comparisons with plenty of repeats.
>   > 4. The *"Add grid to plot for multi-fasta data sets"* parameter adds grid lines to the plot to separate multiple sequences. This is useful when using multi-fasta inputs.
>   > 5. The *"Generate image of detected events"* parameter will include, if enabled, a plot of the rearrangements detected with colors, one per type of rearrangement. More on that later!
>    {: .comment}
>
>
{: .hands_on}



Your `Comparison dotplot` file (remember to inspect each using the {% icon galaxy-eye %} icon) should look like this:

![Single chromosome comparison](../../images/hpc-for-lsgc/aegilops-triticum.png "Chromosome comparison corresponding to <em>Aegilops tauschii</em> and <em>Triticum aestivum</em> as generated with CHROMEISTER. Each axis corresponds to one sequence, from origin to end. ")

Figure 3 shows the comparison plot for the plant chromosomes. Notice that the origin of both sequences is at the top-left and the ending is at the bottom-right of the plot. Noise and small repeats are filtered out automatically by `CHROMEISTER` to allow focusing on where the main similarity blocks are located. We can see that there three types of blocks:

- Blocks that are "conserved" are on the main diagonal. This means that the sequences are similar (in a coarse-grained fashion) in such regions.
- Blocks that are inverted are on the antidiagonals, i.e. rotated 90 degrees. These blocks show sequence regions that have been reversed and rearranged into the complementary strand.
- Blocks that are not in the main diagonal are "transposed", meaning that they have been rearranged in one of the sequences but not in the other. These can also be inverted.

> <comment-title>Note on the comparison plot</comment-title>
> Notice that the comparison plot is only an approximation. It is aimed at showing the general location and direction of syntenies. For example, if `CHROMEISTER` was run with parameter **Output dotplot size** equal to `1000`, then each pixel in the plot contains the averaged information of nearly 500,000 base pairs! Thus, any block can contain lots of smaller rearrangements, mutations, inversions, etc., that are ignored for the sake of providing a clean overview of the general alignment direction in a pairwise comparison.
{: .comment}

Also note that a `score` value can be seen in the title of the plot (this value is also available in the **Comparison score** file). This value is calculated based on the alignment coverage and number of rearrangements and can be used to automatically filter out similar from dissimilar sequence comparisons. A value of `0` means that the sequences are nearly equal (rearrangement-wise), whereas a value closer to `1` means that the sequences are more dissimilar.

## Post-processing: detecting rearrangements

Let us now see how the blocks detected in the comparison are automatically classified for further processing. Figure 4 shows the blocks detected heuristically from the comparison and classified according to their position and orientation. This plot is available in the file **Detected events plot**.

![Events in chromosome comparison](../../images/hpc-for-lsgc/aegilops-triticum-events.png "Automatically detected events in the chromosome comparison corresponding to <em>Aegilops tauschii</em> and <em>Triticum aestivum</em> as generated with CHROMEISTER. Legend: conserved blocks (red), inverted blocks (green) and transposed blocks (blue)")

In Figure 4 we can see a similar plot to that in Figure 3 but with the addition of colors. Each detected block is labelled according to whether it is in the main diagonal (red), transposed (blue) or inverted (green). Inverted transpositions are also labelled green. Notice that this automatic classification process is based on the Hough transform ({% cite duda1972use %}) which is governed by a set of parameters which can therefore affect the sensitivity of the detection of blocks, and hence some blocks can be missed.

Each of these blocks is also available in the `CSV` file **Detected events**. Let us take a peek:

```
xStart,yStart,xEnd,yEnd,strand,approximate length,description
189935556,218167395,202237082,205056373,f,12301526,conserved block
...
156475406,165723309,175173725,185127621,r,18698319,inversion
...
208633875,239669470,213062425,234949502,f,4428550,transposition
...
86602740,73946160,103332815,91252708,r,16730075,inverted transposition
```

The header line describes each column. `xStart` and `xEnd` are the starting and ending coordinates in the query sequence, wheras `yStart` and `yEnd` are the same but for the reference sequence. The strand can be either `f` for the forward strand or `r` for the reverse strand. Notice that the length of the block is approximate, since it is an estimation based on the length of the line and transformed by the length of each sequence. The `description` column contains the label of the detected event. The four possible types (conserved block, inversion, transposition or inverted transposition) are shown above with examples.

Besides providing a visual understanding of the large rearrangements that took place between two sequences, the detection of blocks can be further used to research evolutionary distances, for instance by incorporating the number of Large Scale Genome Rearrangements to generate richer metrics in phylogenetic studies.

# Conclusion


From a pratical perspective, we have shown how to run pairwise genome comparisons using Galaxy, from small to large-scale sequences (in particular bacterial mycoplasmas and plant chromosomes) throughout two levels of precision:

- Using `GECKO` to detect High-scoring Segment Pairs, which can be used to find fine-grained alignments such as repeats, genes, synteny blocks, etc.
- Using `CHROMEISTER` to find large-scale conserved blocks, which enables us to look at the "big picture" of a comparison as well as study its large-scale rearrangements.

Moreover, we have described how we can use additional tools to post-process our comparisons, including extraction of alignments, multiple sequence alignment, automatic detection of rearrangements, dotplot visualization, etc. Additionally, we recommend taking a look at `GECKO-MGV` ({% cite diaz2019combining %}), a tool for the interactive exploration of sequence comparisons produced with `GECKO`.

On the other hand, from a theoretical standpoint, we have discussed why sequence comparison is a difficult problem, how the length of the sequences can affect runtimes, the complexity of the algorithms, impact of parameters, etc., in order to give researchers a broader understanding of how to use sequence comparison algorithms for their scientific experiments (more details in the slides).

tools_json:
["GECKO", "toolshed.g2.bx.psu.edu/repos/iuc/gecko/gecko/1.2", "Text reformatting with AWK", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_awk_tool/1.1.2", "ClustalW", "CHROMEISTER", "toolshed.g2.bx.psu.edu/repos/iuc/chromeister/chromeister/1.5.a"]


topic: genome-annotation
tutorial: annotation-with-prokka
title: Genome annotation with Prokka
difficulty: introductory
time_estimation: 1h
requirements_json: []
tags_json: ["gmod", "prokaryote", "microgalaxy", "jbrowse1"]
content_hash: c9d01aabd43bf1dd0fe0cde75af1c781
last_modified: 2026-05-27T09:01:48.732094
zenodo_link: https://doi.org/10.5281/zenodo.1156405
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/annotation-with-prokka/tutorial.html

content:
In this section we will use a software tool called Prokka to annotate a draft genome sequence. Prokka is a “wrapper”; it collects together several pieces of software (from various authors), and so avoids “re-inventing the wheel”.

Prokka finds and annotates features (both protein coding regions and RNA genes, i.e. tRNA, rRNA) present on on a sequence. Note, Prokka uses a two-step process for the annotation of protein coding regions: first, protein coding regions on the genome are identified using [Prodigal](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2848648/); second, the *function* of the encoded protein is predicted by similarity to proteins in one of many protein or protein domain databases. Prokka is a software tool that can be used to annotate bacterial, archaeal and viral genomes quickly, generating standard output files in GenBank, EMBL and gff formats. More information about Prokka can be found in [Prokka's github link](https://github.com/tseemann/prokka).

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Import the data

Prokka requires assembled contigs.

> <hands-on-title>Obtaining our data</hands-on-title>
>
> 1. Make sure you have an empty analysis history. Give it a name.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the following files from [Zenodo](https://doi.org/10.5281/zenodo.1156405) or from the shared data library
>
>    ```
>    https://zenodo.org/record/1156405/files/contigs.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

## Annotate the genome

Now we will run the tool called Prokka.

> <hands-on-title>Annotate genome</hands-on-title>
>
> 1. {% tool [Prokka](toolshed.g2.bx.psu.edu/repos/crs4/prokka/prokka/1.14.5+galaxy0) %} with the following parameters (leave everything else unchanged)
>    - {% icon param-file %} *"contigs to annotate"*: `contigs.fasta`
{: .hands_on}

## Examine the output

Once Prokka has finished, examine each of its output files.

- The GFF and GBK files contain all of the information about the features annotated (in different formats.)
- The .txt file contains a summary of the number of features annotated.
- The .faa file contains the protein sequences of the genes annotated.
- The .ffn file contains the nucleotide sequences of the genes annotated.


## View annotated features in JBrowse

Now that we have annotated the draft genome sequence, we would like to view the sequence in the JBrowse genome viewer. First, we have to make a JBrowse file. Then, we can view it within Galaxy.

> <hands-on-title>Visualize the annotation</hands-on-title>
>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1) %} with the following parameters
>    - *"Reference genome to display"*: `Use a genome from history`
>       - {% icon param-file %} *"Select the reference genome"*: `fna` output of {% tool [Prokka](toolshed.g2.bx.psu.edu/repos/crs4/prokka/prokka/1.14.5+galaxy0) %}
>
>       This sequence will be the reference against which annotations are displayed
>
>    - *"Output JBrowse"*: `Minimal for viewing (Documentation removed)`
>    - *"Genetic Code"*: `11: The Bacterial, Archaeal and Plant Plastid Code`
>    - Click on `Insert Track Group`
>
>      We will now set up one track - each track is a dataset displayed underneath the reference sequence (which is displayed as nucleotides in FASTA format).
>      We will choose to display the annotations (the Prokka.gff file).
>
>      - In **1: Track Group**
>           - *"Track Category"*: `gene annotations`
>           - Click on `Insert Annotation Track` and fill it with:
>               - *"Track Type"*: `GFF/GFF3/BED Features`
>               - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `gff` output of {% tool [Prokka](toolshed.g2.bx.psu.edu/repos/crs4/prokka/prokka/1.14.5+galaxy0) %}
>
>    A new file will be created in your history, this contains the JBrowse interactive visualisation. We will now view its contents and play with it
>
> 2. Inspect the `JBrowse on data XX and data XX - Complete` file by clicking on the {% icon galaxy-eye %} (eye) icon
>
>    The JBrowse window will appear in the centre Galaxy panel.
>
> 3. Display all the tracks and practice maneuvering around
>    1. Click on the tick boxes on the left to display the tracks
>    2. Select contig 1 in the drop down box. You can only see one contig displayed at a time.
>    1. Zoom by clicking on the `plus` and `minus` buttons.
>    1. JBrowse displays the sequence and a 6-frame amino acid translation.
>    1. Right click on a gene/feature annotation (the bars on the annotation track), then select View Details to see more information.
>      - gene name
>      - product name
>      - you can download the FASTA sequence by clicking on the disk icon
> ![JBrowse](../../images/jbrowse6.png)
{: .hands_on}

## What's Next

After automatic annotation of prokaryotic genome, if inspection of predicted genes with JBrowse introduced mistakes, e.g. wrong exon/intron limits, splitted genes, or merged genes -- or simply if you wish to rename genes or provide additional functional (e.g., Gene Ontology) data, setting up a manual curation project using [Apollo](http://genomearchitect.org/) helps a lot to manually fix these errors.

The [Apollo training]({% link topics/genome-annotation/tutorials/apollo/tutorial.md %}) should provide additional guidance.

tools_json:
["Prokka", "toolshed.g2.bx.psu.edu/repos/crs4/prokka/prokka/1.14.5+galaxy0", "JBrowse", "toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1"]


topic: genome-annotation
tutorial: secondary-metabolite-discovery
title: Secondary metabolite discovery
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: []
content_hash: ed2f6f2b67d9c63874a3adef62a5f0bb
last_modified: 2026-05-15T22:28:13.516014
zenodo_link: https://zenodo.org/records/10652998
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/secondary-metabolite-discovery/tutorial.html

content:
Genome mining is an important source for various bio-active compounds such as antibiotics and fungicides ({% cite adamek_mining_2017 %}).
One challenge in secondary metabolite discovery using genome mining requires to annotate biosynthetic gene clusters (BGCs) using dedicated tools such as antiSMASH ({% cite blin_antismash_2023 %}) as well as to query the discovered secondary metabolites against compound libraries ({% cite zierep_sempi_2020 %}) in order to identify weather they might posses bio-active properties.

In this toturial we will:
* Annotate biosynthetic gene clusters (BGCs) using antiSMASH. 
* Extract the predicted compounds  as `SMILES` using a custom script. 
* Query the predicted compounds agains the MIBiG ({% cite terlouw_mibig_2023 %}) which is a dedicated BGC database.
* Characterize the predicted compounds using dedicated cheminformatic tools.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Galaxy and data preparation

Any analysis should get its own Galaxy history. So let’s start by creating a new one and get the data into it.

## Get data

> <hands-on-title> History Creation and Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 3. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/record/10652998/files/MIBiG_compounds_3.0.sdf
>    https://zenodo.org/record/10652998/files/gbk2features.ipynb
>    ```
>   
>    For this training we need a custom jupyter notebook and a compound library.
>   
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 4. Rename the datasets
> 5. Check that the datatype is correct
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 6. Add useful tags to your datasets 
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Biosynthetic gene clusters (BGCs) detection and compound extraction

The general idea of this tutorial is described in the flowchart below. 
Zierep et. al. previously created a 
workflow that combines BGC prediction with natural compound screening in their webserver [SeMPI 2.0](http://sempi.pharmazie.uni-freiburg.de/)
({% cite zierep_sempi_2020 %}). 
Here, we want to reproduce the workflow using Galaxy, which allows to adjust the workflow and combine it with other
tools and databases. 
E.g. the workflow could be combined with metagenomic workflows, that allow to screen a vast amount of metagenome assembly genomes (MAGs) for potential novel bioactive compounds.

![Overview flowchart](../../images/bgc_screening/flowchart.drawio.png "Overview flowchart of the workflow.")

## Download a bacterial genome with **NCBI Accession Download**

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [NCBI Accession Download](toolshed.g2.bx.psu.edu/repos/iuc/ncbi_acc_download/ncbi_acc_download/0.2.8+galaxy0) %} with the following parameters:
>    - *"Select source for IDs"*: `Direct Entry`
>        - *"ID List"*: `AL645882.2`
>    - *"Molecule Type"*: `Nucleotide`
>
>    > <comment-title> Genome download </comment-title>
>    >
>    > This downloads the `Streptomyces coelicolor A3(2) complete genome`, 
>    > which should be a great source for biosynthetic gene clusters (BGCs).
>    {: .comment}
>
{: .hands_on}

## Detect BGCs and predict NRPS / PKS metabolite structures with **Antismash**

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Antismash](toolshed.g2.bx.psu.edu/repos/bgruening/antismash/antismash/6.1.1+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Sequence file in GenBank,EMBL or FASTA format"*: `Downloaded Files` (output of **NCBI Accession Download** {% icon tool %})
>    - *"Taxonomic classification of input sequence"*: `Bacteria`
>
>    > <comment-title> BGC detection </comment-title>
>    >
>    > This step uses Antismash for BGC detection. Another tool that could be used is [prism](https://prism.adapsyn.com/).
>    > But prism can only be used via a web server and is therefore not integrable into Galaxy workflows.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many BGCs are detected for the `Streptomyces coelicolor A3(2) complete genome` ?
> 2. How many BGCs are of type NRPS ? 
>
> > <solution-title></solution-title>
> >
> > 1. In the HTML report of Antismash you can find 27 BGCs of various types.
> > 2. Of those 27 BGCs 4 are pure NRPS and one more is  classified as NRPS-like.
> >
> {: .solution}
>
{: .question}

## Collapse single BGC Genbank files into one with **Collapse Collection**

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0) %} with the following parameters:
>    - {% icon param-file %} *"Collection of files to collapse into single dataset"*: `Genbank` (output of **Antismash** {% icon tool %})
>    - *"Prepend File name"*: `Yes`
>
>    > <comment-title> Collapse Genbank files </comment-title>
>    >
>    > Antismash produces one Genbank file for each cluster. Genbank files can contain multiple records. For downstream analysis it is more convenient to combine all clusters into one file.   
>    {: .comment}
>
{: .hands_on}

# Use custom scripts in Galaxy to bridge between workflow steps

## Integrate a custom script into the workflow using **Interactive JupyTool and notebook**

For the next steps we need to extract the `SMILES` representations of the predicted BGC metabolites.
And convert it into the `.smi` format (basically a tabular format that contains names and structures of chemical compounds).

The  `SMILES` representations of the predicted BGC metabolites are stored as features of the clusters in the Genbank files.

```
     cand_cluster    1..49828
                     /SMILES="NC(CCCN)C(=O)NC(C(O)C)C(=O)NC(CCCN)C(=O)O"
                     /candidate_cluster_number="1"
                     /contig_edge="False"
                     /detection_rules="cds(Condensation and (AMP-binding or
                     A-OX))"
                     /kind="single"
                     /product="NRPS"
                     /protoclusters="1"
                     /tool="antismash"
```

There are various tools that can convert Genbank files into GFF (like {% tool [customGbkToGff](toolshed.g2.bx.psu.edu/repos/cpt/cpt_gbk_to_gff/edu.tamu.cpt.gff3.customGbkToGff/20.1.0.0) %} and  {% tool [genbank2gff3](toolshed.g2.bx.psu.edu/repos/iuc/bp_genbank2gff3/bp_genbank2gff3/1.1) %}). From the GFF one could extract the SMILES string via regex.
However, unfortunately both tools seem to have difficulty to parse SMILES and introduce artefacts like: 

```
SMILES=CC(%3DO)C(%3DO)O
```

Therefore we need to quickly prototype our own script, that allows to generate the desired `SMILES` representations.
This is in fact a great chance to demonstrate how the interactive {% tool [Interactive JupyTool and notebook](interactive_tool_jupyter_notebook) %} can be used to connect workflow steps, for which there is no dedicated tool yet available in Galaxy, showing the flexible nature of Galaxy.

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Interactive JupyTool and notebook](interactive_tool_jupyter_notebook) %} with the following parameters:
>    - *"Do you already have a notebook?"*: `Load a previous notebook`
>        - {% icon param-file %} *"IPython Notebook"*: `gbk2features.ipynb` (from your history)
>        - *"Execute notebook and return a new one."*: `Yes`
>    - In *"User inputs"*:
>        - {% icon param-repeat %} *"Insert User inputs"*
>            - *"Name for parameter"*: `dataset` (a file called `dataset` will be created in the notebook in the folder `galaxy_inputs`)
>            - *"Choose the input type"*: `Dataset`
>                - {% icon param-file %} *"Select value"*: `Collapse Genbank files` (output of **Collapse Collection** {% icon tool %})
>
>
>    > <comment-title> Interactive JupyTool and notebook </comment-title>
>    >
>    > JupyTool notebooks can be executed interactively, which allows to run any code (python / R / bash) dynamically or as a normal tool itself, that executes a provide script and return the output. More information about the JupyTool can be found in the [Interactive JupyTool Tutorial](https://training.galaxyproject.org/topics/galaxy-interface/tutorials/jupyterlab/tutorial.html).
>    {: .comment}
>
{: .hands_on}

In this workflow the jupyter notebook is executed and the output collected. If you want to run the notebook interactively. Rerun the tool and set the option *"Execute notebook and return a new one."*: `No`. Now the notebook will start up interactively and you can play around the the python code.

The code we provide for the notebook looks like this:

```python
from Bio import SeqIO
import pandas as pd
import numpy as np

input_path = "galaxy_inputs"

use_qualifiers = ["SMILES"] 

feature_list = []

for folder in os.listdir(input_path):

    folder_path = os.path.join(input_path, folder)

    if os.path.isdir(folder_path):

        for file in os.listdir(folder_path):
            file_path = os.path.join(folder_path, file)
            if '.genbank' in file:
                for record in SeqIO.parse(file_path, "genbank"):
                    #print(record)
                    
                    for feat in record.features:

                        combined_features = {}
                        combined_features['ID'] = record.id
                        combined_features['Name'] = record.name
                        combined_features['Type'] = feat.type
                        combined_features['Start'] = feat.location.start
                        combined_features['End'] = feat.location.end
                        combined_features['Strand'] = feat.location.strand

                        for qualifier in feat.qualifiers.items():
                                combined_features[qualifier[0]] = qualifier[1][0]
                        
                        feature_list.append(combined_features)
                
df = pd.DataFrame(feature_list)

# df trimming is simpler with pandas
df.replace('', np.nan, inplace=True)
df = df.dropna(subset=use_qualifiers)

#remove extra space antiSMASH bug (https://github.com/antismash/antismash/issues/694)
df['SMILES'] = df['SMILES'].str.replace(' ', '')


#only keep what is needed downstream
df = df.loc[:,['ID', 'Name', 'Type', 'Start', 'End', 'Strand', 'SMILES']]

df.to_csv("outputs/collection/feature_table.tsv", sep="\t")
```

> <question-title></question-title>
>
> 1. What does the notebook do ?
>
> > <solution-title></solution-title>
> >
> > 1. The notebook creates a table with a row for each `SMILES` qualifier.
> >
> {: .solution}
>
{: .question}

# Cheminformatics

Now, that we have predicted the products of the BGCs found in the `Streptomyces coelicolor A3(2) complete genome`,
we can compare the chemical structures of the prediction with natural compound libraries to see if some of the compounds 
are similar to other compounds with known bio-activte properties. Those compounds should be further investigated.
The natural compound library we are using here comprises all the compounds found in the MIBiG ({% cite terlouw_mibig_2023 %}).

## Convert the table into a two column table 

The next two steps convert the generated table into the `.smi` format. The `.smi` format is basically a specific table format where the first column is a chemical structure and the second column the ID or name of the compound. The table must
not have a header.

> <hands-on-title> Table to smi format conversion</hands-on-title>
>
> 1. {% tool [Text reformatting](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_awk_tool/1.1.2) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `output_collection` (output of **Interactive JupyTool and notebook** {% icon tool %})
>    - *"AWK Program"*: `{print $8, $2-$5-$6}`
>
> 2. {% tool [Remove beginning](Remove beginning1) %} with the following parameters:
>    - {% icon param-file %} *"from"*: `outfile` (output of **Text reformatting** {% icon tool %})
>
> 3. Now, that the table is indeed the same as the `.smi` format it can also be converted into `.smi`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="smi" %}
>
{: .hands_on}


## Remove duplicated molecules

> <hands-on-title> Remove duplicated molecules </hands-on-title>
>
> 1. {% tool [Remove duplicated molecules](toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_remduplicates/openbabel_remDuplicates/3.1.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Molecular input file"*: (output of **Remove beginning** {% icon tool %})
>    - *"Select descriptor for molecule comparison"*: `Canonical SMILES`
>
>    > <comment-title> short description </comment-title>
>    >
>    > For the downstream analysis deduplicated molecules are not required. Especially short, hybrid or uncompleted cluster can produce short molecular structures, that appear multiple times in an organisms. These structures can be regarded as artefacts and could probably be removed as well.
>    {: .comment}
>
{: .hands_on}

## Generate fingerprints for the predicted secondary metabolites and the target database with **Molecule to fingerprint**

Various molecular fingerprint options can be used to convert the compounds. Details about molecular fingerprint can be found in {% cite muegge_overview_2016 %} and {% cite capecchi_one_2020 %}.

> <hands-on-title> Task description </hands-on-title>
>
>
> 1. {% tool [Molecule to fingerprint](toolshed.g2.bx.psu.edu/repos/bgruening/chemfp/ctb_chemfp_mol2fps/1.5) %} with the following parameters:
>    - {% icon param-file %} *"Molecule file"*: (output of **Remove duplicated molecules** {% icon tool %})
>    - *"Type of fingerprint"*: `Open Babel FP2 fingerprints`
>
> 2. {% tool [Molecule to fingerprint](toolshed.g2.bx.psu.edu/repos/bgruening/chemfp/ctb_chemfp_mol2fps/1.5) %} with the following parameters:
>    - {% icon param-file %} *"Molecule file"*: `output` (MIBiG_compounds_3.0.sdf)
>    - *"Type of fingerprint"*: `Open Babel FP2 fingerprints`
>
>    > <comment-title> Be consitent </comment-title>
>    >
>    > A major application of molecular fingerprints is to determine the similarity of compounds. For the downstream tool
>    > {% tool [Similarity Search](toolshed.g2.bx.psu.edu/repos/bgruening/simsearch/ctb_simsearch/0.2) %} it is **important** that the same fingerprint are used for the query and target compounds!
>    {: .comment}
>
{: .hands_on}

Now that we have converted our compounds into molecular fingerprints, we can compare them to find the closest match between our prediction and the target database.

## Compare prediction with target DB using **Similarity Search**

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Similarity Search](toolshed.g2.bx.psu.edu/repos/bgruening/simsearch/ctb_simsearch/0.2) %} with the following parameters:
>    - *"Subject database/sequences"*: `Chemfp fingerprint file`
>        - *"Query Mode"*: `Query molecules are stores in a separate file`
>            - {% icon param-file %} *"Query molecules"*: (output of **Molecule to fingerprint** for `MIBiG_compounds_3.0.sdf` {% icon tool %})
>            - {% icon param-file %} *"Target molecules"*: (output of **Molecule to fingerprint** from **Remove duplicated molecules** {% icon tool %})
>        - *"select the k nearest neighbors"*: `1`
>
{: .hands_on}


## Determine the novelty of the compounds using **Natural Product** likeness calculator

Normally, this tool allows to estimate how similar any kind of compound is to a natural product.
In our case we know that the compounds are natural compounds. However, by comparing to a large source of known 
natural products we can estimate the novelty of our predicted structure. This could for example be used to 
mine metagenomic datasets for likely sources of uncharacterized molecular scaffolds. 

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Natural Product](toolshed.g2.bx.psu.edu/repos/bgruening/natural_product_likeness/ctb_np-likeness-calculator/2.1) %} with the following parameters:
>    - {% icon param-file %} *"Molecule file"*: (output of **Remove duplicated molecules** {% icon tool %})
>
>
{: .hands_on}

## Check **Drug-likeness** of the predicted compounds

Estimates the drug-likeness of molecules, based on eight commonly used molecular properties, and reports a score between 0 (all properties unfavourable) to 1 (all properties favourable). Two possible methods to weight the features are available (QEDw,mo, QEDw,max), as well as an option to leave features unweighted (QEDw,u).

The eight properties used are: molecular weight (MW), octanol–water partition coefficient (ALOGP), number of hydrogen bond donors (HBDs), number of hydrogen bond acceptors (HBAs), molecular polar surface area (PSA), number of rotatable bonds (ROTBs), number of aromatic rings (AROMs) and number of structural alerts (ALERTS).

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Drug-likeness](toolshed.g2.bx.psu.edu/repos/bgruening/qed/ctb_silicos_qed/2021.03.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Molecule data in SDF or SMILES format"*: (output of **Remove duplicated molecules** {% icon tool %})
>
{: .hands_on}

# Conclusion

In this simple use case the combination of BGC prediction software and cheminformatics was used to detect and characterize novel secondary metabolites.

tools_json:
["NCBI Accession Download", "toolshed.g2.bx.psu.edu/repos/iuc/ncbi_acc_download/ncbi_acc_download/0.2.8+galaxy0", "Antismash", "toolshed.g2.bx.psu.edu/repos/bgruening/antismash/antismash/6.1.1+galaxy1", "Collapse Collection", "toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0", "customGbkToGff", "toolshed.g2.bx.psu.edu/repos/cpt/cpt_gbk_to_gff/edu.tamu.cpt.gff3.customGbkToGff/20.1.0.0", "genbank2gff3", "toolshed.g2.bx.psu.edu/repos/iuc/bp_genbank2gff3/bp_genbank2gff3/1.1", "Interactive JupyTool and notebook", "interactive_tool_jupyter_notebook", "Text reformatting", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_awk_tool/1.1.2", "Remove beginning", "Remove beginning1", "Remove duplicated molecules", "toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_remduplicates/openbabel_remDuplicates/3.1.1+galaxy0", "Molecule to fingerprint", "toolshed.g2.bx.psu.edu/repos/bgruening/chemfp/ctb_chemfp_mol2fps/1.5", "Similarity Search", "toolshed.g2.bx.psu.edu/repos/bgruening/simsearch/ctb_simsearch/0.2", "Natural Product", "toolshed.g2.bx.psu.edu/repos/bgruening/natural_product_likeness/ctb_np-likeness-calculator/2.1", "Drug-likeness", "toolshed.g2.bx.psu.edu/repos/bgruening/qed/ctb_silicos_qed/2021.03.4+galaxy0"]


topic: genome-annotation
tutorial: functional
title: Functional annotation of protein sequences
difficulty: introductory
time_estimation: 1h
requirements_json: []
tags_json: ["eukaryote", "biodiversity"]
content_hash: 0a4ea2bafe61e0ad4421c286f04938ef
last_modified: 2026-05-27T09:01:48.985723
zenodo_link: https://zenodo.org/record/6861851
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/functional/tutorial.html

content:
When performing the structural annotation of a genome sequence, you get the position of each gene, but you don't have information about their name of their function. That's the goal of **functional annotation**.

In this short tutorial, we will run the most commonly used tools to perform functional annotation, starting from the predicted protein sequences of a few example genes.

For a more complete view of how this step integrates into a whole genome sequencing and annotation process, you can have a look at the [Funannotate tutorial]({% link topics/genome-annotation/tutorials/funannotate/tutorial.md %}).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data upload

We will annotate a small set of **protein sequences**. These sequences were predicted from the gene structures obtained in the [Funannotate tutorial]({% link topics/genome-annotation/tutorials/funannotate/tutorial.md %})? Though these sequences from from a fungal species, you can run the same tools on proteins from any organisms, including prokaryotes.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/record/6861851/files/proteins.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

{% include topics/genome-annotation/tutorials/functional/content.md short=true %}

# Conclusion

Congratulations for reaching the end of this tutorial! Now you know how to perform the functional annotation of a set of protein sequences, using EggNOG mapper and InterProScan.

If you want to collect more functional annotation, you can try to run the {% tool [NCBI BLAST+ blastp](toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastp_wrapper/2.10.1+galaxy2) %} or {% tool [Diamond](toolshed.g2.bx.psu.edu/repos/bgruening/diamond/bg_diamond/2.0.15+galaxy0) %} tools against the UniProt or NR databases (Diamond runs much faster on big datasets). These tools will search for similarities between your protein sequences and the ones already described in big international databases.

Also note that many other more specialised tools exist to collect even more functional annotation, in particular for certain species (prokaryotes forexample), or enzyme/protein families.

tools_json:
["NCBI BLAST+ blastp", "toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastp_wrapper/2.10.1+galaxy2", "Diamond", "toolshed.g2.bx.psu.edu/repos/bgruening/diamond/bg_diamond/2.0.15+galaxy0"]


topic: microbiome
tutorial: qiime2-moving-pictures
title: QIIME 2 Moving Pictures
difficulty: intermediate
time_estimation: 10H
tools_json: []
requirements_json: []
tags_json: ["metagenomics", "taxonomic profiling", "diversity", "microgalaxy"]
content_hash: d41d8cd98f00b204e9800998ecf8427e
last_modified: 2026-05-27T09:01:49.385308
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/qiime2-moving-pictures/tutorial.html


topic: microbiome
tutorial: mags-building
title: Building and Annotating Metagenome-Assembled Genomes (MAGs) from Short Metagenomics
difficulty: intermediate
time_estimation: 6H
requirements_json: []
tags_json: ["assembly", "binning", "metagenomics", "microgalaxy"]
content_hash: e9824ec09538013dac8a5fd19aab5b25
last_modified: 2026-05-27T09:01:49.351375
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/mags-building/tutorial.html

content:
Metagenomics has revolutionized our understanding of microbial communities by enabling the study of genetic material directly from environmental samples. One of the most powerful applications of metagenomics is the reconstruction of **Metagenome-Assembled Genomes (MAGs)**, i.e. near-complete or complete genomes of individual microorganisms recovered from complex microbial communities. MAGs provide invaluable insights into microbial diversity, function, and ecology, without the need for laboratory cultivation.

This tutorial is designed for anyone interested in reconstructing MAGs from paired short metagenomic reads. You will learn the essential steps, from quality control and read preprocessing to assembly, binning, refinement, and annotations of MAGs. By the end of this tutorial, you will be equipped with the knowledge, tools, and workflows to confidently generate high-quality MAGs from your own metagenomic datasets.

To guide through this process, we will use **Galaxy workflows** developed within the [FAIRyMAGs project](https://elixir-europe.org/how-we-work/scientific-programme/commissioned-services/science/bfsp/fairymags) and maintained by the [Intergalactic Workflow Commission](https://iwc.galaxyproject.org/). These workflows are available via the [IWC Workflow Library](https://iwc.galaxyproject.org/), as well as two workflow registries: [WorkflowHub](https://workflowhub.eu/) and [Dockstore](https://dockstore.org/).

### Case Study: Honey Bee Gut Microbiome

To illustrate the workflows, we will use public data from a study investigating the response of the **honey bee gut microbiota** to *Nosema ceranae* (a highly prevalent microsporidian parasite) under the influence of a probiotic and/or a neonicotinoid insecticide ({% cite sbaghdi2024response %}). The honey bee gut microbiome comprises a relatively simple core community dominated by **five bacterial lineages**, including *Snodgrassella alvi* and *Gilliamella apicola*, which play essential roles in digestion, immunity, and pathogen defense ({% cite motta2024honeybee %}). However, this microbiome is highly sensitive to abiotic stressors, such as pesticides, pollutants, and climate change ({% cite motta2024honeybee %}, {% cite ramsey2019varroa %}, {% cite alberoni2021neonicotinoids %}). For example, exposure to neonicotinoids has been linked to shifts in microbial diversity and increased susceptibility to opportunistic pathogens {% cite alberoni2021neonicotinoids %}.

While research often examines stressors in isolation, their synergistic effects remain understudied ({% cite paris2020honeybee %}, {% cite meixner2010historical %}). In the {% cite sbaghdi2024response %} study, western honey bees were experimentally infected with *N. ceranae* spores, exposed to the neonicotinoid thiamethoxam, and/or treated with the probiotic bacterium *Pediococcus acidilactici*, which is thought to enhance tolerance to *N. ceranae*. The study used deep shotgun metagenomic Illumina sequencing to analyze 21 samples, with taxonomic composition explored using MetaPhlAn4 ({% cite blanco2023extending %}).

> <comment-title></comment-title>
> Learn more about taxonomic profiling in our dedicated tutorial: [**Taxonomic Profiling and Visualization of Metagenomic Data**]({% link topics/microbiome/tutorials/taxonomic-profiling/tutorial.md %})
{: .comment}

In this tutorial, we will focus on **two representative samples** from this dataset to demonstrate the MAG reconstruction process.

Sample Alias | Treatment | Replicate | BioSample | Run
--- | --- | --- | --- | ---
IP2 | `NP`: infected with *N. ceranae* spores and treated with the probiotic | 2 | [SAMN35523843](https://www.ebi.ac.uk/ena/browser/view/SAMN35523843) | [SRR24759598](https://www.ebi.ac.uk/ena/browser/view/SRR24759598)
I1 | `N`: infected with *N. ceranae* spores | 1 | [SAMN35523839](https://www.ebi.ac.uk/ena/browser/view/SAMN35523839) | [SRR24759616](https://www.ebi.ac.uk/ena/browser/view/SRR24759616)

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Prepare Galaxy and Data

A well-organized workspace is essential for efficient and reproducible metagenomic analysis. In Galaxy, each analysis should be conducted within its own **dedicated history** to ensure clarity, avoid data mixing, and streamline collaboration or future reference.

> <hands-on-title>Prepare history</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

To proceed with our analysis, we need to **import the raw metagenomic sequencing data** from the NCBI Sequence Read Archive (SRA). For this tutorial, we use two representative datasets, identified by their SRA run accession numbers: `SRR24759598` and `SRR24759616`. These datasets will serve as the starting point for our quality control, preprocessing, and MAG reconstruction workflows. Let’s retrieve and prepare these reads for downstream analysis.

> <hands-on-title>Get data from NCBI SRA</hands-on-title>
>
> 1. Create a new file with one SRA per line
>
>    ```text
>    SRR24759598
>    SRR24759616
>    ```
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md %}
>
> 2. {% tool [Faster Download and Extract Reads in FASTQ](toolshed.g2.bx.psu.edu/repos/iuc/sra_tools/fasterq_dump/3.1.1+galaxy1) %}
>   - *"select input type"*: `List of SRA accession, one per line`
>      - {% icon param-file %}*"sra accession list"*: created dataset
>
> 3. Rename `Pair-end data (fasterq-dump)` collection to `Raw reads`
>
> > <comment-title></comment-title>
> > 
> > Download from NCBI SRA can take time. If it takes too long, you can import the data from Zenodo.
> > 
> > > <hands-on-title>Import raw data</hands-on-title>
> > >
> > > 1. Import the two files from [ENA](https://www.ebi.ac.uk/ena/browser/view/PRJNA977416) or the Shared Data library:
> > >
> > >    ```text
> > >    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR247/098/SRR24759598/SRR24759598_1.fastq.gz
> > >    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR247/098/SRR24759598/SRR24759598_2.fastq.gz
> > >    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR247/016/SRR24759616/SRR24759616_1.fastq.gz
> > >    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR247/016/SRR24759616/SRR24759616_2.fastq.gz
> > >    ```
> > >
> > > 2. Create a collection named `Raw reads`, rename the pairs with the sample name (`SRR24759598` and `SRR24759616`)
> > >
> > {: .hands_on}
> {: .comment}
{: .hands_on}

# Preprocess the Reads

The quality and composition of our metagenomic dataset directly influence the success of Metagenome-Assembled Genome (MAG) reconstruction. To ensure accurate, high-resolution results, it is essential to preprocess the reads through a series of critical steps:

- **Quality Control**: Assess and refine sequencing data to remove low-quality bases, adapters, and short reads that could compromise downstream analyses.
- **Contamination Removal**: Filter out host-derived sequences (e.g., honey bee DNA) and potential human contaminants, which can obscure microbial signals and introduce bias.

By meticulously preparing the reads, we create a clean, microbial-enriched dataset—the optimal foundation for robust MAG reconstruction. Let's walk through these essential preprocessing steps.

## Control Raw Data Quality

**Poor-quality reads**, such as those with low base-calling accuracy, adapter contamination, or insufficient length, can introduce errors, bias assemblies, and compromise the integrity of your results. Before proceeding with any analysis, it is essential to **assess, trim, and filter** our raw sequencing data to ensure only high-quality reads are retained.

We will use a [versatile quality control workflow](https://iwc.galaxyproject.org/workflow/short-read-qc-trimming-main/) to process our FASTQ files. This workflow is designed to:
1. **Trim and filter reads** using **fastp** ({% cite Chen_2018 %}), removing low-quality bases (default: minimum quality score of 15) and short reads (default: minimum length of 15 bases). These parameters can be adjusted based on your specific dataset and requirements.
2. Aggregate quality reports for all samples into a single, comprehensive overview using **MultiQC** ({% cite ewels2016multiqc %}), allowing to quickly visualize and compare the quality metrics across the datasets.

> <comment-title></comment-title>
> Learn more about sequencing data quality control in our dedicated tutorial: [**Quality Control**]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %})
{: .comment}

Let's import the workflow from WorkflowHub:

{% snippet faqs/galaxy/workflows_run_wfh.md title="short-read-qc-trimming/main" wfhub_id="2025" version="1"%}

We can now launch it:

> <hands-on-title>Control raw data quality</hands-on-title>
>
> 1. Run the workflow using the following parameters
>    - {% icon param-collection %} *"Raw reads"*: `Raw reads` collection
>    - *"Qualified quality score"*: `15`
>    - *"Minimal read length"*: `15`
> 2. Inspect the MultiQC report
{: .hands_on}

> <question-title></question-title>
>
> 1. How many reads are in each dataset?
> 2. What are the length of the reads before trimming?
> 3. What are the miminum values for average bp quality score for forward and reverse reads, before and and after trimming and filtering?
>
> > <solution-title></solution-title>
> >
> > 1. 43.9 millions reads for SRR24759598 and 42.8 millions for SRR24759616
> > 2. 150 bp
> > 3. Minimum average bp quality scores given the "Sequence Quality" graph:
> > 
> >     | SRR24759598 | SRR24759616
> >    --- | --- | ---
> >    Forward (Read 1) - Before | 34.5 | 34.6
> >    Forward (Read 1) - After | 34.5 | 34.6
> >    Reverse (Read 2) - Before | 33.4 | 32.7
> >    Reverse (Read 2) - After | 33.4 | 32.7
> >
> {: .solution}
>
{: .question}

## Remove Contamination and Host Reads

Metagenomic datasets often contain **non-microbial sequences** that can compromise the accuracy and quality of downstream analyses, including MAG reconstruction. These sequences may originate from **host DNA** (in this case, the honey bee) or **external contamination**, such as human DNA introduced during sample handling or sequencing. Failing to remove these sequences can lead to misassembly, incorrect taxonomic assignments, and skewed functional interpretations.

We will now focus on filtering out both host (bee) and potential human contamination from our metagenomic reads. This critical step ensures that our dataset is enriched for microbial sequences, improving the reliability of our MAGs and enabling more precise biological insights. We will use a dedicated [workflow](https://iwc.galaxyproject.org/workflow/host-contamination-removal-short-reads-main/) to:
1. **Map reads against reference genomes** for the host (bee) and common contaminants using  **Bowtie2** ({% cite Langmead_2009 %}, {% cite Langmead_2012 %}), enabling precise identification and removal of non-microbial sequences.
2. Aggregate and visualize mapping results with **MultiQC** ({% cite ewels2016multiqc %}), providing a comprehensive overview of the filtering process and ensuring transparency in our data cleaning efforts.

Let's import the workflow from WorkflowHub:

{% snippet faqs/galaxy/workflows_run_wfh.md title="host-contamination-removal-short-reads/main" wfhub_id="2026" version="1" %}

We will run the workflow twice:
1. **First**, we focus on **filtering out host (honey bee) reads**, i.e. sequences originating from the bee itself, which can dominate the dataset and obscure microbial signals
2. **Second**, we will target **potential human contamination**, which may have been introduced during sample handling or sequencing. 

Let's begin by launching the workflow to remove honey bee-derived sequences. 

> <hands-on-title>Remove host reads</hands-on-title>
>
> 1. Run the workflow using the following parameters
>    - {% icon param-collection %} *"Short-reads"*: `Trimmed reads` collection
>    - *"Host/Contaminant Reference Genome"*: `A. mellifera genome (apiMel3, Baylor HGSC Amel_3.0)`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
> 2. Rename `Reads without host or contaminant reads` collection to `Reads without bee reads`
> 3. Inspect the MultiQC report
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Which percentage of reads mapped to the honey bee reference genome?
> 2. How many reads have been removed?
>
> > <solution-title></solution-title>
> >
> > 1. 2.8% for SRR24759598 and 1.1% for SRR24759616
> > 2. 2.8% for SRR24759598 and 1.1% for SRR24759616
> >
> {: .solution}
>
{: .question}

With the host (honey bee) sequences successfully filtered out, we now turn our attention to removing potential human contamination from the dataset. We run the workflow a second time, this time aligning the reads against a human reference genome:
 
> <hands-on-title>Remove human reads</hands-on-title>
>
> 3. Run the workflow using the following parameters
>    - {% icon param-collection %} *"Short-reads"*: `Reads without bee reads` collection
>    - *"Host/Contaminant Reference Genome"*: `Human (Homo sapiens): hg38 Full`
> 4. Rename `Reads without host or contaminant reads` collection to `Reads without bee and human reads`
> 3. Inspect the newly generated MultiQC report
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Which percentage of reads mapped to the honey bee reference genome?
> 2. How many reads have been removed?
>
> > <solution-title></solution-title>
> >
> > 0.1% in both samples
> >
> {: .solution}
>
{: .question}

We have now completed the essential preprocessing steps to ensure our metagenomic dataset is **clean, high-quality, and enriched for microbial sequences**. By performing **rigorous quality control** and systematically **removing host and contaminant reads**, we have minimized potential biases and maximized the reliability of our data. With these preparations complete, our dataset is now optimized for the next phase: **assembling and reconstructing Metagenome-Assembled Genomes (MAGs)**. 

# Build, Refine, and Annotate Metagenome-Assembled Genomes (MAGs)

Now that our metagenomic reads are **clean, high-quality, and free of contamination**, we are ready to embark on the core phase of our analysis: **reconstructing and annotating Metagenome-Assembled Genomes (MAGs)**. This process transforms our processed sequencing data into **near-complete or complete microbial genomes**, enabling deeper insights into the functional potential, taxonomy, and ecological roles of the microorganisms in our samples.

We will use a comprehensive workflow to go through each step of MAG reconstruction and annotation:

![This flowchart illustrates the comprehensive workflow for building and annotating Metagenome-Assembled Genomes (MAGs) from metagenomic reads. The process begins with sample preparation, including quality control and host removal sub-workflows, followed by grouping samples. Reads are assembled into contigs using MEGAHIT and metaSPADES, and then aligned with Bowtie2. Contigs are binned using four tools: MaxBin2, SemiBin, MetaBAT2, and CONCOCT. The resulting bins are refined using Binette and dereplicated with dRep based on CheckM2 quality metrics. The best quality bins are assessed for completeness and contamination using QUAST, CheckM, and CheckM2, while abundance is estimated with CoverM. Taxonomic classification is performed using GTDB-Tk, and functional annotation is carried out with Bakta. Finally, all results are consolidated into a MultiQC report for easy visualization and analysis. The diagram uses color-coded boxes and arrows to represent each step and data flow, providing a clear and structured overview of the entire MAG reconstruction and annotation process.](images/fairymags_workflow.png)

1. **Assembly**: Reconstruct genomic sequences (contigs) from our metagenomic reads using **MEGAHIT** ({% cite Li_2015 %}) or **metaSPADES** ({% cite Nurk_2017 %}).
2. **Binning**: Group contigs into discrete genomic bins using four complementary tools: **MetaBAT2** ({% cite Kang_2019 %}), **MaxBin2** ({% cite Wu_2015 %}), **SemiBin** ({% cite Pan_2022 %}), and **CONCOCT** ({% cite Alneberg_2014 %}), each offering unique strengths for capturing microbial diversity.
3. **Refinement**: Improve the quality and completeness of the MAGs by refining bins with **Binette** ({% cite Mainguy2024 %}), the successor to metaWRAP, which removes contamination and merges related bins.
4. **Dereplication**: Consolidate the MAGs across all input samples using **dRep** ({% cite olm2017drep %}), which clusters genomes based on CheckM2 quality metrics to eliminate redundancy.
5. **Quality Assessment**: Evaluate the integrity of the MAGs with **CheckM** ({% cite Parks2015 %}) and **CheckM2** ({% cite Chklovski2023CheckM2 %}) for completeness and contamination estimates.
6. **Abundance Estimation**: Quantify the relative abundance of each MAG in our samples using **CoverM** ({% cite aroney2025coverm %}), providing insights into microbial population dynamics.
7. **Taxonomic Assignment**: Classify the MAGs using **GTDB-Tk** ({% cite Chaumeil2019 %}), aligning them with the Genome Taxonomy Database for accurate taxonomic placement.
8. **Functional Annotation**: Identify and annotate genes within our MAGs using **Bakta** ({% cite Schwengers_2021 %}), revealing their functional potential and biological roles.

All results are consolidated into a single **MultiQC** ({% cite ewels2016multiqc %}) report, allowing us to easily visualize and interpret the quality and characteristics of our MAGs.

Let's import the workflow from WorkflowHub:

{% snippet faqs/galaxy/workflows_run_wfh.md title="mags-building/main" wfhub_id="1352" version="4"%}

> <hands-on-title>Build, refine, and annotate MAGs</hands-on-title>
>
> 1. Run the workflow using the following parameters
>    - *"Trimmed reads from grouped samples"*: `Reads without bee and human reads`
>    - *"Trimmed reads"*: `Reads without bee and human reads`
>    - *"Choose Assembler"*: `MEGAHIT`
>
>      > <comment-title></comment-title>
>      > metaSPAdes is an alternative assembler.
>      >
>      > MEGAHIT is less computationally intensive and generate higher quality single and shorter contigs but shorter. metaSPAdes is very computationally intensive, but generates longer/more complete assemblies.
>      {: .comment}
>
>    - *"Minimum length of contigs to output"*: `200`
>
>    - *"Environment for the built-in model (SemiBin)"*: `global`
>
>      > <comment-title></comment-title>
>      > Environment for the built-in model (SemiBin), options are: 
>      > - `human_gut`, 
>      > - `dog_gut`,
>      > - `ocean`,
>      > - `soil`,
>      > - `cat_gut`,
>      > - `human_oral`,
>      > - `mouse_gut`,
>      > - `pig_gut`,
>      > - `built_environment`,
>      > - `wastewater`,
>      > - `chicken_caecum`,
>      > - `global`
>      {: .comment}
>
>    - *"Minimum MAG completeness percentage"*: `75`
>
>      {% snippet topics/microbiome/faqs/minimum_mag_completeness_percentage.md %}
>
>    - *"Contamination weight (Binette)"*: `2`
>
>      > <comment-title></comment-title>
>      > This weight is used for the scoring the bins. A low weight favor complete bins over low contaminated bins.
>      {: .comment}
>
>    - *"Read length (CONCOCT)"*: `150`
>
>      > <comment-title></comment-title>
>      > CONCOCT requires the read length for coverage. Best use fastQC to estimate the mean value
>      {: .comment}
>
>    - *"CheckM2 Database"*: most recent one
>
>      > <comment-title></comment-title>
>      > This database is used for quality assessment for Binette, dRep, and quality assessment of the final bins.
>      {: .comment}
>
>    - *"Minimum MAG length"*: `50000`
>
>      {% snippet topics/microbiome/faqs/minimum_mag_length.md %}
>
>    - *"ANI threshold for dereplication"*: `0.95`
>
>      {% snippet topics/microbiome/faqs/ani.md %}
>
>      {% snippet topics/microbiome/faqs/ani_dereplication_threshold.md %}
>
>    - *"Maximum MAG contamination percentage"*: `25`
>
>      {% snippet topics/microbiome/faqs/maximum_mag_contamination_percentage.md %}
>
>    - *"Run GTDB-Tk on MAGs"*: `Yes`
>    - *"GTDB-tk Database"*: most recent one
>    - *"Bakta Database"*: most recent one
>    - *"AMRFinderPlus Database for Bakta"*: most recent one
{: .hands_on}

To ensure clarity and efficiency, we will **walk through each step of the workflow** to carefully inspect the generated results. Given that executing the entire workflow can be time-consuming, we will also **provide the option to import an history with pre-computed results for each step**. This approach allows to focus on understanding the outputs and interpreting the findings without waiting for lengthy computations.

> <details-title>Import history with pre-computed results</details-title>
>
> > <hands-on-title>Import history with pre-computed results</hands-on-title>
> >
> > 1. Import the history in one of the following URLs:
> >    - [{{ page.answer_histories[0].label }}]({{ page.answer_histories[0].history }})
> >    - [{{ page.answer_histories[1].label }}]({{ page.answer_histories[1].history }})
> >    - [{{ page.answer_histories[2].label }}]({{ page.answer_histories[2].history }})
> >    - [{{ page.answer_histories[3].label }}]({{ page.answer_histories[3].history }})
> >
> >    {% snippet faqs/galaxy/histories_import.md %}
> >
> {: .hands_on}
{: .details}

## Assembly: Reconstructing Genomic Sequences from Metagenomic Reads

The first critical step in the MAG reconstruction workflow is **assembly**: the computational process of reconstructing genomic sequences, known as **contigs**, from fragmented metagenomic reads. Assembly can be likened to solving a complex jigsaw puzzle: the goal is to identify reads that "fit together" by detecting overlapping sequences. However, unlike a traditional puzzle, this task is complicated by several challenges inherent to genomics and metagenomics:

- **Genomic Complexity:** Repeated sequences, missing fragments, and errors introduced during sequencing make accurate reconstruction difficult.
- **Data Volume:** Metagenomic datasets are often massive, requiring significant computational resources.
- **Community Diversity:** Unequal representation of microbial community members—ranging from dominant to rare species—can skew assembly results.
- **Genomic Similarity:** The presence of closely related microorganisms or multiple strains of the same species adds layers of complexity.
- **Data Limitations:** Minor community members may be underrepresented, leading to incomplete or fragmented assemblies.

To address these challenges, a variety of **metagenomic assemblers** have been developed, including **metaSPAdes** ({% cite Nurk_2017 %}) and **MEGAHIT** ({% cite Li_2015 %}). Each assembler has unique computational characteristics, and their performance can vary depending on the microbiome being studied. As demonstrated by the **Critical Assessment of Metagenome Interpretation (CAMI) initiative** ({% cite sczyrba2017 %}, {% cite meyer2021 %}, {% cite meyer2022 %}), the choice of assembler often depends on the specific goals of the analysis, such as maximizing contiguity, accuracy, or computational efficiency. Selecting the right tool is essential for achieving high-quality assemblies tailored to your research objectives.

> <comment-title></comment-title>
> Learn more about metagenomic assembly, algorithms and the tools in our dedicated tutorial: [**Assembly of metagenomic sequencing data**]({% link topics/microbiome/tutorials/metagenomics-assembly/tutorial.md %})
{: .comment}

In this workflow execution, we used **MEGAHIT**, but **metaSPAdes** is also an option as an alternative assembler in the workflow.

> <question-title></question-title>
>
> 1. How many contigs were assembled for the SRR24759598 sample?
> 2. And for the SRR24759616 sample?
> 3. What is the minimum length of the contigs?
>
> > <solution-title></solution-title>
> >
> > 1. The MEGAHIT assembly for the SRR24759598 sample produced 534,275 contigs.
> > 2. 572,074 contigs for the SRR24759616 sample.
> > 3. When we executed the workflow, we set the *"Minimum length of contigs to output"* parameter to **200 base pairs**. Upon inspecting the resulting FASTA file, we confirmed that all contigs have a length exceeding this threshold, as indicated by the `len` attribute in the sequence headers.
> >
> {: .solution}
>
{: .question}

Once the assembly process is complete, **assessing the quality of the assembled contigs** is a crucial step to ensure the reliability and accuracy of downstream analyses. Assembly quality can be systematically evaluated using **metaQUAST** ({% cite mikheenko2016 %}), the specialized metagenomics mode in the widely used **QUAST** tool ({% cite Gurevich_2013 %}).

QUAST provides a comprehensive suite of metrics to gauge assembly performance, including:
- **Contiguity statistics**, such as the number of contigs, total assembly length, and N50/L50 values, which reflect the length and distribution of assembled sequences.
- **Genomic feature analysis**, including gene and operon prediction, to assess the biological relevance of the assembly.
- **Comparison to reference genomes** (if available), enabling the identification of structural variations, misassemblies, and potential errors.

By leveraging QUAST, we can gain valuable insights into the completeness, accuracy, and overall quality of our metagenomic assemblies, ensuring robust and meaningful results for further analysis. 

QUAST generates multiple outputs but the main output is a HTML report which aggregate different metrics.

Let's inspect the HTML reports:

> <hands-on-title>Inspect QUAST HTML reports</hands-on-title>
>
> 1. Enable Window Manager
> 
>    {% snippet faqs/galaxy/features_scratchbook.md %}
> 
> 2. Open both QUAST HTML reports
> 3. Click on `Extended report`
{: .hands_on}

At the top of each report, we find a **summary table** displaying key statistics for contigs. The rows represent various metrics, while the columns correspond to the different sample assemblies.

> <comment-title></comment-title>
> Learn more about assembly statistics generated by QUAST in the [**Assembly of metagenomic sequencing data**]({% link topics/microbiome/tutorials/metagenomics-assembly/tutorial.md %})
{: .comment}

Let's now examine the table, from top to bottom, to interpret the assembly results for each sample:

- **Statistics without reference**

   > <question-title></question-title>
   >
   > 1. How many contigs are reported for SRR24759598? And for SRR24759616?
   > 2. Why are these numbers different from the number of sequences in the output of MEGAHIT? Which statistics in the QUAST report corresponds to number of sequences in the output of MEGAHIT?
   > 3. What is the length of the longest contig in SRR24759598? And in SRR24759616?
   > 4. What is the total length in SRR24759598? And in SRR24759616?
   > 5. How similar are the microbial communities between the two samples, based on the contig statistics?
   >
   > > <solution-title></solution-title>
   > >
   > > 1. Number of contigs:
   > >    - 89,312 contigs for SRR24759598
   > >    - 85,045 contigs for SRR24759616.
   > > 2. The numbers in the QUAST report are lower because QUAST **only includes contigs longer than 500 bp** by default. The statistic # contigs (\(( \geq \)) 0 bp) in the QUAST report corresponds to the total number of sequences in the MEGAHIT output.
   > > 3. Length of the largest Contig:
   > >   - 239,878 bp in SRR24759598
   > >   - 428,888 bp for SRR24759616.
   > > 4. Total Assembly Length:
   > >   - 137,726,640 bp in SRR24759598
   > >   - 117,285,055 bp for SRR24759616.
   > > 5. The two samples, SRR24759598 and SRR24759616, exhibit comparable contig statistics, suggesting a degree of similarity in their microbial communities. Both samples have a similar number of contigs (89,312 vs. 85,045) However, the largest contigs is almost 2 times bigger in SRR24759616 and the total assembly length differs by about 20 million base pairs, which may indicate variations in microbial diversity, abundance, or sequencing depth between the two samples. Further analysis, such as taxonomic or functional annotation of the contigs, would be needed to determine the extent of their biological similarity.
   > {: .solution}
   >
   {: .question}

   Beyond simply counting contigs and measuring their lengths, two key statistics (**N50** and **L50**) are widely used to assess assembly quality:

   - **N50:** This value represents the length of the shortest contig in a set where the combined lengths of all contigs **equal or exceed half of the total assembly size**. A higher N50 indicates better contiguity, meaning fewer gaps and longer contiguous sequences in the assembly.
   - **L50:** This is the **minimum number of contigs** needed to cover at least 50% of the total assembly length. A lower L50 suggests that the assembly is more contiguous, as fewer, longer contigs are required to reach half of the genome's total length.

   > <details-title>Understanding N50 and L50: Key Metrics for Assembly Quality</details-title>
   >
   > The **N50 statistic** is a widely used metric to evaluate the **contiguity** of a genome assembly. When all contigs in an assembly are sorted by length, the N50 represents the **length of the shortest contig** in the set that collectively contains at least 50% of the total assembled bases. For example, if an assembly has an N50 of 10,000 base pairs (bp), it means that 50% of the total assembled sequence is contained in contigs that are **10,000 bp or longer**.
   > 
   > **Example Calculation**:
   > Consider an assembly with 9 contigs of the following lengths: **2, 3, 4, 5, 6, 7, 8, 9, and 10 bp**.
   > - The **total length** of the assembly is **54 bp**.
   > - Half of this total is **27 bp**.
   > - The sum of the three longest contigs (**10 + 9 + 8 = 27 bp**) reaches this halfway point.
   > - Therefore, the **N50 is 8 bp**, meaning that the smallest contig in the set covering half the assembly is 8 bp long.
   > 
   > **Important Considerations for N50**:
   > - **Assembly Size Matters:** When comparing N50 values across different assemblies, it is essential that the **total assembly sizes are similar**. N50 alone is not meaningful if the assemblies vary significantly in size.
   > - **Limitations of N50:** N50 does not always reflect the true quality of an assembly. For instance, consider two assemblies with the following contig lengths:
   >   - Assembly 1: **3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 25, 25, 150, 1500**
   >   - Assembly 2: **50, 500, 530, 650**
   > 
   > Both assemblies may have the **same N50**, but Assembly 2 is clearly more contiguous, with fewer and longer contigs.
   > 
   > 
   > The **L50 statistic** complements N50 by indicating the **minimum number of contigs** required to cover at least 50% of the total assembly length. In the previous example, the L50 is **3**, as only three contigs (10, 9, and 8 bp) are needed to reach half of the total assembly length.
   > 
   > **Why N50 and L50 Matter**:
   > - **N50** provides insight into the **length distribution** of contigs, with higher values indicating better contiguity.
   > - **L50** reflects the **compactness** of the assembly, with lower values suggesting fewer gaps and longer contiguous sequences.
   > 
   > Together, N50 and L50 offer a more comprehensive view of assembly quality, helping researchers assess the continuity and reliability of their genomic reconstructions.
   {: .details}

   > <question-title></question-title>
   >
   > 1. What is the N50 for SRR24759598? And for SRR24759616?
   > 2. What is the L50 for SRR24759598? And for SRR24759616?
   > 3. How do the N50 and L50 values compare between the two samples, and what does this indicate about their assembly quality?
   >
   > > <solution-title></solution-title>
   > >
   > > 1. N50 values:
   > >   - 2,213 for SRR24759598
   > >   - 1,751 for SRR24759616.
   > > 2. L50 valueS:
   > >   - 11,641 for SRR24759598
   > >   - 12,941 for SRR24759616.
   > > 3. The N50 value is higher for SRR24759598 (2,213 bp) compared to SRR24759616 (1,751 bp), indicating that the assembly for SRR24759598 is slightly more contiguous, with longer contigs covering half of the total assembly length. However, the L50 value is lower for SRR24759598 (11,641) than for SRR24759616 (12,941), meaning fewer contigs are needed to reach 50% of the total assembly length in SRR24759598. This suggests that while both assemblies are fragmented, SRR24759598 has a modestly better contiguity overall. The differences in these metrics may reflect variations in microbial diversity, sequencing depth, or assembly performance between the two samples.
   > {: .solution}
   >
   {: .question}

- **Read mapping**: results of the mapping of the raw reads on the contigs

    > <question-title></question-title>
    >
    > 1. What is the percentage (%) of SRR24759598 reads mapped to SRR24759598 contigs? And for SRR24759616?
    > 2. What is the percentage of reads used to build the assemblies for SRR24759598? and SRR24759616?
    >
    > > <solution-title></solution-title>
    > >
    > > 1. percentage (%) of mapped reads:
    > >   - 97.1% for SRR24759598
    > >   - 98.54% for SRR24759616
    > > 2. 97.1% of reads were used to build the contigs for SRR24759598 and 98.54% for SRR24759616.
    > {: .solution}
    >
    {: .question}

## Binning: Grouping Sequences into Microbial Genomes

**Metagenomic binning** is a computational process that classifies DNA sequences from metagenomic data into discrete groups, or **bins**, based on their similarity. The primary goal is to **assign sequences to their original organisms or taxonomic groups**, enabling researchers to reconstruct individual microbial genomes and explore the diversity and functional potential of complex microbial communities.

Binning relies on a combination of **sequence composition, coverage, and similarity** to group contigs into bins. These bins ideally represent the genomes of distinct microorganisms present in the sample. However, the process is inherently challenging due to:
- **Genomic complexity**, such as repeated sequences and strain-level variation.
- **Uneven sequencing coverage** across different microbial species.
- **Contamination and chimeras**, which can obscure the boundaries between genomes.

Numerous algorithms have been developed for metagenomic binning, each with unique strengths and limitations. The choice of tool often depends on the dataset's characteristics and the research objectives. Benchmark studies (e.g., {% cite sczyrba2017 %}, {% cite meyer2021 %}, {% cite meyer2022 %}) provide practical guidance on selecting the most suitable binner for specific environments and datasets.

> <comment-title></comment-title>
> Learn more about metagenomic binning and available tools in our dedicated tutorial: [**Binning of metagenomic sequencing data**]({% link topics/microbiome/tutorials/metagenomics-binning/tutorial.md %})
{: .comment}

A widely adopted strategy is to **use multiple binning tools** to profit from their individual advantages and methods, followed by **bin refinement** to consolidate and improve the results. This approach leverages the strengths of different algorithms, producing a more robust and reliable set of bins.

In this analysis, we employed **four complementary binning tools**, each offering distinct advantages for capturing microbial diversity:
- **MetaBAT2** ({% cite Kang_2019 %}): Known for its efficiency and accuracy in binning metagenomic contigs.
- **MaxBin2** ({% cite Wu_2015 %}): Utilizes probabilistic models to improve binning accuracy.
- **SemiBin** ({% cite Pan_2022 %}): Incorporates deep learning to enhance binning performance, especially for complex datasets.
- **CONCOCT** ({% cite Alneberg_2014 %}): Uses Gaussian mixture models to cluster contigs based on coverage and composition.

Let's know look at the results of each binner for each sample.

> <hands-on-title>Inspect bins for each binner</hands-on-title>
>
> 1. Get number of bins generated by **MetaBAT2**
>    - Open the `MetaBAT2 on collection X and collection Y: Bin sequences` collection
>    - Extract the number of bin, i.e. the number of element for each sample list
> 2. Get number of bins generated by **MaxBin2**
>    - Open the `MaxBin2 on collection X and collection Y: Bins` collection
>    - Extract the number of bin, i.e. the number of element for each sample list
> 3. Get number of bins generated by **SemiBin**
>    - Open the `SemiBin on collection X and collection Y: Reconstructed bins` collection
>    - Extract the number of bin, i.e. the number of element for each sample list
> 4. Get number of bins generated by **CONCOCT**
>    - Open the `CONCOCT: Extract a fasta file on collection X and collection Y : Bins` collection
>    - Extract the number of bin, i.e. the number of element for each sample list
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many bins have been found for each sample and binner?
> 2. Which binning tools generated the most and least bins for the samples?
> 3. Is the trend in the number of bins generated consistent across both samples?
>
> > <solution-title></solution-title>
> >
> > 1. Number of bins per sample and binner:
> >
> >    Binner | SRR24759598 | SRR24759616
> >    --- | --- | ---
> >    MetaBAT2 | 34 | 25
> >    MaxBin2 | 45 | 35
> >    SemiBin | 16 | 20
> >    CONCOCT | 80 | 59
> > 
> > 2. For the provided samples, the number of bins generated by each tool varies as follows:
> >    - **CONCOCT** generated the **most bins** for both samples. This suggests that CONCOCT tends to produce **finer, potentially more fragmented groupings**, which may capture more nuanced microbial diversity but could also result in **over-splitting** of genomes.
> >    - **SemiBin** generated the **fewest bins** for both samples. This indicates a **more conservative approach**, potentially producing **larger, more complete bins** but possibly missing finer distinctions between closely related microbes.
> >    - **MaxBin2** and **MetaBAT2** produced an **intermediate number of bins**. These tools strike a **balance between granularity and completeness**, with MaxBin2 generating slightly more bins than MetaBAT2. This suggests **comparable performance** in terms of bin granularity, with MaxBin2 being slightly more permissive.
> >
> > 3. Yes, the trend in the number of bins generated is **consistent across both samples**. For each binning tool used, **SRR24759598 consistently produced more bins than SRR24759616**. While **SemiBin is the exception** with slightly more bins for SRR24759616, the **overall trend** across the majority of tools suggests that **SRR24759598 has a higher number of bins**. This indicates that the microbial community in **SRR24759598 may be more complex, diverse, or fragmented** compared to SRR24759616. The exception with SemiBin could reflect differences in the tool's sensitivity or algorithmic approach to binning.
> {: .solution}
>
{: .question}

Beyond simply comparing the total number of bins, we can also examine the **contigs per bin** for each binning tool, which provides deeper insight into the **quality and granularity** of the reconstructed microbial genomes. 

For that, we will use the `collection X, collection Y, and others (as list)` collection of collection. This structure contains two sub-collections—one for each sample. Within each sub-collection, there are four tables, each corresponding to a different binning tool (MetaBAT2, MaxBin2, SemiBin, and CONCOCT). Each table consists of two columns: the contig identifier and its assigned bin ID. 
We will group the contigs by their bin IDs and count the number of contigs in each bin. This will allow us to compute statistics and evaluate how contigs are distributed across bins for each binning tool, providing insights into the quality and granularity of the bins.

> <hands-on-title>Get statistics for the number of contigs per bin for each binner</hands-on-title>
>
> 1. {% tool [Group data by a column and perform aggregate operation on other columns](Grouping1) %} with following parameters:
>    - {% icon param-collection %} *"Select data"*: `collection X, collection Y, and others (as list)`
>    - *"Group by column"*: `Column: 2`
>    - {% icon param-repeat %} *Insert Operation*
>       - *"Type"*: `Count`
>       - *"On column"*: `Column: 1`
> 
> 2. {% tool [Summary Statistics](Summary_Statistics1) %} with following parameters:
>    - {% icon param-collection %} *"Summary statistics on"*: output of **Group data**
>    - *"Column or expression"*: `c2`
> 
> 3. {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0) %} with following parameters:
>    - {% icon param-collection %} *"Collection of files to collapse into single dataset"*: output of **Summary Statistics**
>    - *"Keep one header line"*: `Yes`
{: .hands_on}

Now, we have **two tables**—one for each sample—containing **statistics on the number of contigs per bin**. Each table is organized with **one row per binning tool**, listed in the following order:
- **CONCOCT**
- **MetaBAT2**
- **MaxBin2**
- **SemiBin**

> <question-title></question-title>
>
> 1. What are the statistics of number of contigs per bin for each sample and binner?
> 2. How do you interpret the binning tool statistics for the samples?
> 3. Is the trend in the number of contigs per bin consistent across both samples?
> 4. What are the practical implications?
>
> > <solution-title></solution-title>
> >
> > 1. Statistics for SRR24759598
> >
> >    Binner | Sum | mean | stdev | 0% | 25% | 50% | 75% | 100%
> >    --- | --- | --- | --- | --- | --- | --- | --- | ---
> >    CONCOCT | 33,700 | 421.25 | 1,071.79 | 1 | 3 | 18.5 | 130 | 6,010
> >    MetaBAT2 | 8,588 | 252.588 | 228.014 | 2 | 96.75 | 152.5 | 429.5 | 829 
> >    MaxBin2 | 26,134 | 580.756 | 635.111 | 3 | 128 | 466 | 824 | 3,177 
> >    SemiBin | 3,526 | 220.375 | 149.77 | 1 | 131.5 | 208.5 | 262.25 | 544 
> > 
> >    Statistics for SRR24759616
> >
> >    Binner | sum | mean | stdev | 0% | 25% | 50% | 75% | 100%
> >    --- | --- | --- | --- | --- | --- | --- | --- | ---
> >    CONCOCT | 29,513 | 500.22 | 1,243.39 | 1 | 3 | 15 | 190 | 7,166 
> >    MetaBAT2 | 7,658 | 306.32 | 347.609 | 3 | 67 | 153 | 348 | 1,229 
> >    MaxBin2 | 25,105 | 717.286 | 701.627 | 1 | 114.5 | 574 | 1,170 | 2,802 
> >    SemiBin | 3,373 | 168.65 | 155.593 | 1 | 48.5 | 154 | 213.25 | 493 
> >
> > 2. General Trends Across Binners
> >    - CONCOCT consistently used more contigs to generate the bins (sum) for both samples, with a wide range of contigs per bin (high standard deviation). This indicates that CONCOCT tends to create many small bins, some of which may be fragmented or incomplete.
> >    - MetaBAT2 uses fewer contigs and shows lower variability in the number of contigs per bin. This suggests a more conservative approach, likely producing more consolidated and potentially higher-quality bins.
> >    - MaxBin2 uses a moderate number of bins but with a higher mean number of contigs per bin. Its standard deviation is also relatively high, indicating variability in bin sizes.
> >    - SemiBin uses the fewest bins and shows low variability in contig counts per bin, suggesting a balanced and consistent binning approach.
> >
> >    Distribution of Contigs per Bin
> >    - CONCOCT:
> >      - SRR24759598: The median number of contigs per bin is 18.5 (50%), but the distribution is highly skewed, with some bins containing up to 6,010 contigs. This suggests potential over-splitting of contigs into many small bins.
> >      - SRR24759616: The median is 15, with a maximum of 7,166 contigs per bin, indicating a similar trend of over-splitting.
> >    - MetaBAT2:
> >      - SRR24759598: The median number of contigs per bin is 152.5, with a relatively narrow interquartile range (96.75 to 429.5). This indicates a more even distribution of contigs across bins.
> >      - SRR24759616: The median is 153, with a similar distribution pattern, suggesting consistent binning performance.
> >    - MaxBin2:
> >      - SRR24759598: The median number of contigs per bin is 466, with a broad distribution (128 to 824). This indicates variability in bin sizes, with some bins containing many contigs.
> >      - SRR24759616: The median is 574, with a similar broad distribution, reflecting a tendency to create larger bins.
> >    - SemiBin:
> >      - SRR24759598: The median number of contigs per bin is 208.5, with a tight interquartile range (131.5 to 262.25). This suggests a uniform binning process.
> >      - SRR24759616: The median is 213.25, with a similar tight distribution, indicating consistent binning.
> >
> > 3. For both samples, CONCOCT and MaxBin2 produce more bins than MetaBAT2 and SemiBin, but the distribution of contigs per bin is more variable for CONCOCT. SRR24759598 generally has slightly more bins across all tools compared to SRR24759616, which may reflect differences in microbial diversity or sequencing depth between the samples. MetaBAT2 and SemiBin show similar trends in both samples, with relatively consistent bin sizes and fewer extreme outliers.
> >
> > 4. CONCOCT may be useful for capturing fine-scale diversity but may require additional refinement to merge small or fragmented bins. MetaBAT2 and SemiBin are likely to produce more reliable and consolidated bins, making them suitable for downstream analyses like genome reconstruction and functional annotation. MaxBin2 offers a balance between the number of bins and the number of contigs per bin but may require further validation due to its broader distribution.
> >
> {: .solution}
>
{: .question}

We can also evaluate the binning with **CheckM2** ({% cite Chklovski2023CheckM2 %}). CheckM2 is a software tool used in metagenomics binning to assess the completeness and contamination of genome bins. It compares the genome bins to a set of universal single-copy marker genes that are present in nearly all bacterial and archaeal genomes. By identifying the presence or absence of these marker genes in the bins, CheckM2 can estimate the completeness of each genome bin (i.e., the percentage of the total set of universal single-copy marker genes that are present in the bin) and the degree of contamination (i.e., the percentage of marker genes that are found in more than one bin).

> <hands-on-title>Assessing the completeness and contamination of bins for each binner</hands-on-title>
>
> 1. Find and run the **Quality control of bins generated by MetaBAT2, MaxBin2, SemiBin, and CONCOCT** public workflow with parameters:
>    
>    {% snippet faqs/galaxy/workflows_run.md tab="Public workflows" name="Quality control of bins generated by MetaBAT2, MaxBin2, SemiBin, and CONCOCT" %}
>     
>    - {% icon param-collection %} *"MetaBAT2 Bins"*: `MetaBAT2 on collection X and collection Y: Bin sequences`
>    - {% icon param-collection %} *"MaxBin2 Bins"*: `MaxBin2 on collection X and collection Y: Bins`
>    - {% icon param-collection %} *"SemiBin Bins"*: `SemiBin on collection X and collection Y: Reconstructed bins`
>    - {% icon param-collection %} *"CONCOCT Bins"*: `CONCOCT: Extract a fasta file on collection X and collection Y : Bins`
> 2. Inspect the `MultiQC for CheckM on bins right after binners` file
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many bins have a predicted completeness of 100% and greater than 90% for each sample? Are these high-completeness bins distributed across all binning tools?
> 2. How many of the bins with a predicted completeness greater than 90% have a predicted contamination above 50%? What does this imply about the quality of these bins?
>
> > <solution-title></solution-title>
> >
> > 1. SRR24759598:
> >
> >    Binner | Predicted completness = 100% | Predicted completness > 90%
> >    --- | --- | --
> >    MetaBAT2 | 1 | 5
> >    MaxBin2 | 0 | 2
> >    SemiBin | 0 | 0
> >    CONCOCT | 4 | 11
> >    **Total** | 5 | 18
> >
> >    SRR24759616:
> >
> >    Binner | Predicted completness = 100% | Predicted completness > 90%
> >    --- | --- | --
> >    MetaBAT2 | 1 | 3
> >    MaxBin2 | 0 | 1
> >    SemiBin | 0 | 0
> >    CONCOCT | 2 | 9
> >    **Total** | 3 | 13
> > 
> >    CONCOCT produces the highest number of high-completeness bins (both 100% and >90%) for both samples, suggesting it may be more effective at reconstructing near-complete genomes in these datasets. However, all binning tools except SemiBin contribute to the high-completeness bins. This indicates that combining multiple binning tools can enhance the recovery of complete and near-complete microbial genomes.
> > 
> > 2. Number of bins with completeness > 90% that also have contamination > 50%
> > 
> >    Binner | SRR24759598 | SRR24759616
> >    --- | --- | --
> >    MetaBAT2 | 0 / 5 | 1 / 3
> >    MaxBin2 | 1 / 2 | 0 / 1
> >    SemiBin | 0 / 0 | 0 / 0
> >    CONCOCT | 6 / 11 | 6 / 9
> >    **Total** | 7 / 18 | 7 / 13
> >
> >    Implications:
> >    - A significant proportion of **high-completeness bins from CONCOCT** exhibit **contamination levels above 50%**. This suggests that while CONCOCT effectively captures near-complete genomes, it may also include **substantial contamination from other organisms**, potentially compromising the accuracy of downstream analyses.
> >    - MetaBAT2 and MaxBin2produce **fewer contaminated high-completeness bins**, indicating that their bins are generally more reliable and purer for further genomic studies.
> >    - These results highlight the importance of **post-binning refinement** to reduce contamination and improve the quality of high-completeness bins, especially when using tools like CONCOCT.
> >
> {: .solution}
>
{: .question}

## Bin Refinement: Enhancing the Quality and Completeness of Bins

Metagenomic binning is a powerful approa

tools_json:
["Faster Download and Extract Reads in FASTQ", "toolshed.g2.bx.psu.edu/repos/iuc/sra_tools/fasterq_dump/3.1.1+galaxy1", "Group data by a column and perform aggregate operation on other columns", "Grouping1", "Summary Statistics", "Summary_Statistics1", "Collapse Collection", "toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0", "CheckM", "toolshed.g2.bx.psu.edu/repos/iuc/checkm_lineage_wf/checkm_lineage_wf/1.2.4+galaxy2", "CheckM2", "toolshed.g2.bx.psu.edu/repos/iuc/drep_dereplicate/drep_dereplicate/3.6.2+galaxy1", "Extract dataset", "__EXTRACT_DATASET__", "Cut", "Cut1", "Replace Text", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.5+galaxy2", "Join two Datasets", "join1", "Remove beginning", "Remove beginning1", "Column join", "toolshed.g2.bx.psu.edu/repos/iuc/collection_column_join/collection_column_join/0.0.3", "Group", "Concatenate multiple datasets or collections", "cat1", "Sort", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2", "kMetaShot", "toolshed.g2.bx.psu.edu/repos/bgruening/kmetashot/kmetashot/2.0+galaxy2", "Bakta", "toolshed.g2.bx.psu.edu/repos/iuc/bakta/bakta/1.9.4+galaxy1", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy4"]


topic: microbiome
tutorial: general-tutorial
title: Analyses of metagenomics data - The global picture
difficulty: intermediate
time_estimation: 2H30M
tools_json: []
requirements_json: []
tags_json: ["microgalaxy"]
content_hash: 02a5060380bf59461fe44bb867e6d69a
last_modified: 2026-05-15T22:28:13.976607
zenodo_link: https://doi.org/10.5281/zenodo.815875
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/general-tutorial/tutorial.html

content:
In metagenomics, information about micro-organisms in an environment can be extracted with two main techniques:

- Amplicon sequencing, which sequences only the rRNA or ribosomal DNA of organisms
- Shotgun sequencing, which sequences full genomes of the micro-organisms in the environment

In this tutorial, we will introduce the two main types of analyses with their general principles and differences. For a more in-depth look at these analyses, we recommend our detailed tutorials on each analysis.

We will use two datasets (one amplicon and one shotgun) from the same [project on the Argentinean agricultural pampean soils](https://www.ebi.ac.uk/metagenomics/projects/SRP016633). In this project, three different geographic regions that are under different types of land uses and two soil types (bulk and rhizospheric) were analyzed using shotgun and amplicon sequencing. We will focus on data from the Argentina Anguil and Pampas Bulk Soil (the original study included one more geographical regions {% cite Rascovan2013 %}.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Amplicon data

Amplicon sequencing is a highly targeted approach for analyzing genetic variation in specific genomic regions.
In the metagenomics fields, amplicon sequencing refers to capture and sequence of rRNA data in a sample.
It can be 16S for bacteria or archea or 18S for eukaryotes.

> <comment-title>Background: The 16S ribosomal RNA gene</comment-title>
> ![The 16S ribosomal RNA gene](../../images/16S_gene.png "The 16S ribosomal RNA gene. Credit: https://www.slideshare.net/beiko/ccbc-tutorial-beiko")
>
> The 16S rRNA gene has several properties that make it ideally suited to our purposes
>
> 1. Present in all living organisms
> 2. Highly conserved + highly variable regions
> 3. Huge reference databases
>
> ![Variable regions](../../images/16S_variableregions.jpg "Variable regions of the 16S rRNA")
>
> The highly conserved regions make it easy to target the gene across different organisms, while the highly variable regions allow us to distinguish between different species.
>
{: .comment}

With amplicon data, we can determine the micro-organisms from which the sequences in our sample are coming from. This is called taxonomic assignation.
We try to assign sequences to taxons and then classify or extract the taxonomy in our sample.

In this analysis, we will use the [mothur tool suite](https://mothur.org), but only a small portion of its tools and possibilities.
To learn more in detail about how to use this tool, check out the full [mothur tutorial](../mothur-miseq-sop/tutorial.html).

## Importing the data

Our datasets come from a soil samples in two different Argentinian locations, for which the 16S rDNA V4 region
has been sequenced using 454 GS FLX Titanium. For the tutorial, the original fastq data has been down sampled and converted to fasta. The original data are available at EBI Metagenomics under the following run numbers:

- Pampa soil: [SRR531818](https://www.ebi.ac.uk/metagenomics/projects/SRP016633/samples/SRS353016/runs/SRR531818/results/versions/2.0)
- Anguil soil: [SRR651839](https://www.ebi.ac.uk/metagenomics/projects/SRP016633/samples/SRS386929/runs/SRR651839/results/versions/2.0)


> <hands-on-title>Data upload</hands-on-title>
>
> 1. Import the following files from [Zenodo](https://zenodo.org/record/815875), or from the shared data library (in `GTN - Material -> {{ page.topic_name }} -> {{ page.title }}`):
>    - `SRR531818_pampa.fasta`
>    - `SRR651839_anguil.fasta`
>
>    ```
>    https://zenodo.org/record/815875/files/SRR531818_pampa.fasta
>    https://zenodo.org/record/815875/files/SRR651839_anguil.fasta
>    ```
>
>    > <tip-title>Importing data via links</tip-title>
>    >
>    > * Copy the link location
>    > * Open the Galaxy Upload Manager
>    > * Select **Paste/Fetch Data**
>    > * Paste the link into the text field
>    > * Press **Start**
>    {: .tip}
>
>    > <tip-title>Importing data from a data library</tip-title>
>    >
>    > * Go into "Data" (top panel) then "Data libraries"
>    > * Click on "Training data" and then "Analyses of metagenomics data"
>    > * Select interesting file
>    > * Click on "Import selected datasets into history"
>    > * Import in a new history
>    {: .tip}
>
>    As default, Galaxy takes the link as name, so rename them.
>
{: .hands_on}

<!--

Anguil Soil: https://www.ebi.ac.uk/metagenomics/projects/SRP016633/samples/SRS386929
Pampa Soil https://www.ebi.ac.uk/metagenomics/projects/SRP016633/samples/SRS353016/runs/SRR531818/results/versions/2.0

Project's data: https://www.ebi.ac.uk/metagenomics/projects/SRP016633/samples/SRS353016/runs/SRR531818/results/versions/2.0
Project's pipeline: https://www.ebi.ac.uk/metagenomics/pipelines/2.0
Project's QC results: https://www.ebi.ac.uk/metagenomics/projects/SRP016633/samples/SRS386929/runs/SRR651839/results/versions/2.0
-->

### Preparing datasets

We will perform a multisample analysis with mothur. To do this, we will merge all reads into a single file,
and create a *group file*, indicating which reads belong to which samples.

> <hands-on-title>Prepare multisample analysis</hands-on-title>
>
> 1. **Merge.files** {% icon tool %} with the following parameters
>   - "Merge" to `fasta files`
>   - "Inputs" to the two sample fasta files
>
> 2. **Make.group** {% icon tool %} with the following parameters
>   - "Method to create group file" to `Manually specify fasta files and group names`
>   - "Additional": Add two elements to this repeat
>        - First element
>            - "fasta - Fasta to group" to `SRR531818_pampa` file
>            - "group - Group name" to `pampa`
>        - Second element (click on "Insert Additional")
>            - "fasta - Fasta to group" to `SRR651839_anguil` file
>            - "group - Group name" to `anguil`
>
{: .hands_on}

> <comment-title>Note</comment-title>
>
> Because we only have a small number of samples, we used the manual specification. If you have hundreds of samples this would quickly become bothersome. The solution? use a collection! To read more about collections in Galaxy, please see [dedicated collections tutorial]({% link topics/galaxy-interface/tutorials/collections/tutorial.md %}).
{: .comment}

Have a look at the group file. It is a very simple file, it contains two columns: the first contains the read names, the second contains the group (sample) name, in our case `pampa` or `anguil`.


### Optimization of files for computation

Because we are sequencing many of the same organisms, we anticipate that many of our sequences are
duplicates of each other. Because it's computationally wasteful to align the same thing a bazillion
times, we'll unique our sequences using the `Unique.seqs` command:

> <hands-on-title>Remove duplicate sequences</hands-on-title>
>
> 1. **Unique.seqs** {% icon tool %} with the following parameters
>   - "fasta" to the merged fasta file
>   - "output format" to `Name File`
>
>    > <question-title></question-title>
>    >
>    > How many sequences were unique? How many duplicates were removed?
>    >
>    > > <solution-title></solution-title>
>    > > 19,502 unique sequences and 498 duplicates.
>    > >
>    > > This can be determined from the number of lines in the fasta (or names) output, compared to the
>    > > number of lines in the fasta file before this step.
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

This `Unique.seqs` tool outputs two files: a fasta file containing only the unique sequences, and a *names file*.
The names file consists of two columns, the first contains the sequence names for each of the unique sequences, and the second column contains all other sequence names that are identical to the representative sequence in the first column.

```
name          representatives
read_name1    read_name2,read_name,read_name5,read_name11
read_name4    read_name6,read_name,read_name10
read_name7    read_name8
...
```

> <hands-on-title>Count sequences</hands-on-title>
>
> 1. **Count.seqs** {% icon tool %} with the following parameters
>   - "name" to the name file from `Unique.seqs`
>   - "Use a group file" to `yes`
>   - "group" to the group file from `Make.group`
{: .hands_on}

The `Count.seqs` file keeps track of the number of sequences represented by each unique representative across multiple samples. We will pass this file to many of the following tools to be used or updated as needed.

## Quality Control

The first step in any analysis should be to check and improve the quality of our data.


> <comment-title></comment-title>
>
> For more information on the topic of quality control, please see our [dedicated training materials]({% link topics/sequence-analysis/index.md %}).
{: .comment}


First, let's get a feel for our data:

> <hands-on-title>Summarize data</hands-on-title>
>
> 1. **Summary.seqs** {% icon tool %} with the following parameters
>   - "fasta" parameter to the fasta from `Unique.seqs`
>   - "count" to count table from `Count.seqs`
>   - "output logfile?" to `yes`
>
{: .hands_on}

The `summary` output files give information per read. The `logfile` outputs also contain some summary
statistics:

```
              Start    End        NBases     Ambigs   Polymer  NumSeqs
Minimum:      1        80         80         0        3        1
2.5%-tile:    1        104        104        0        3        501
25%-tile:     1        242        242        0        4        5001
Median:       1        245        245        0        4        10001
75%-tile:     1        245        245        0        4        15001
97.5%-tile:   1        247        247        0        6        19501
Maximum:      1        275        275        2        31       20000
Mean:         1        237.519    237.519    0.00495  4.24965
# of unique seqs:   19502
total # of seqs:    20000
```

This tells us that we have a total of 19,502 unique sequences, representing 20,000 total sequences that vary in length between 80 and 275 bases. Also, note that at least some of our sequences had some ambiguous base calls.
Furthermore, at least one read had a homopolymer stretch of 31 bases, this is likely an error so we would like to filter such reads out as well.

If you are thinking that 20,000 is an oddly round number, you are correct; we downsampled the original datasets to 10,000 reads per sample for this tutorial to reduce the amount of time the analysis steps will take.

We can filter our dataset on length, base quality, and maximum homopolymer length using the `Screen.seqs` tool

The following tool will remove any sequences with ambiguous bases (`maxambig` parameter), homopolymer stretches of 9 or more bases (`maxhomop` parameter) and any reads longer than 275 bp or shorter than 225 bp.

> <hands-on-title>Filter reads based on quality and length</hands-on-title>
>
> 1. **Screen.seqs** {% icon tool %} with the following parameters
>   - "fasta" to the fasta file from `Unique.seqs`
>   - "minlength" parameter to `225`
>   - "maxlength" parameter to `275`
>   - "maxambig" parameter to `0`
>   - "maxhomop" parameter to `8`
>   - "count" to the count file from `Count.seqs`
>
{: .hands_on}

> <question-title></question-title>
>
> How many reads were removed in this screening step? (Hint: run the `Summary.seqs` tool again)
>
> > <solution-title></solution-title>
> > 1,804.
> >
> > This can be determined by looking at the number of lines in bad.accnos output of screen.seqs step or by comparing the total number of seqs between of the summary.seqs log before and after this screening step
> {: .solution }
{: .question}

## Sequence Alignment

Aligning our sequences to a reference helps improve OTU assignment [[Schloss et. al.](https://www.ncbi.nlm.nih.gov/pubmed/23018771)], so we will now align our sequences to an alignment of the V4 variable region of the 16S rRNA. This alignment has been created as described in [[mothur's MiSeq SOP](https://mothur.org/wiki/MiSeq_SOP)] from the Silva reference database.

> <hands-on-title>Align sequences</hands-on-title>
>
> 1. Import the `silva.v4.fasta` file in your history
>
>    ```
>    https://zenodo.org/record/815875/files/silva.v4.fasta
>    ```
>
> 2. **Align.seqs** {% icon tool %} with the following parameters
>   - "fasta" to the `good.fasta` output from `Screen.seqs`
>   - "Select Reference Template from" to `Your history`
>   - "reference" to the `silva.v4.fasta` reference file
>   - "flip" to `Yes`
>
>    This step may take a few minutes, please be patient.
>
> 3. **Summary.seqs** {% icon tool %} with the following parameters
>   - "fasta" parameter to the aligned output from `Align.seqs`
>   - "count" parameter to count_table output from `Screen.seqs`
>
{: .hands_on}

To get an idea of the quality of the alignment, we can view the log output from the summary step:

```
        Start   End NBases  Ambigs  Polymer NumSeqs
Minimum:    2391    10674   9   0   2   1
2.5%-tile:  3080    12071   234 0   4   455
25%-tile:   3080    13424   244 0   4   4545
Median:     3080    13424   245 0   4   9090
75%-tile:   3080    13424   245 0   4   13634
97.5%-tile: 3082    13424   246 0   6   17724
Maximum:    13396   13425   267 0   7   18178
Mean:   3080.6  13380   244.212 0   4.27946
# of unique seqs:   17698
total # of seqs:    18178
```

> <question-title></question-title>
>
> 1. How many sequences have been aligned?
> 2. Between which positions most of the reads are aligned to this references?
>
> > <solution-title></solution-title>
> > 1. 17,698 are aligned
> > 2. From this we can see that most of our reads align nicely to positions `3080-13424` on this reference. This corresponds exactly to the V4 target region of the 16S gene.
> {: .solution }
{: .question}

To make sure that everything overlaps the same region we'll re-run `Screen.seqs` to get sequences that start at or before position 3,080 and end at or after position 13,424.

> <hands-on-title>Remove poorly aligned sequences</hands-on-title>
>
> 1. **Screen.seqs** {% icon tool %} with the following parameters
>   - "fasta" to the aligned fasta file
>   - "start" to `3080`
>   - "end" to `13424`
>   - "count" to the group file created by the previous run of `Screen.seqs`
>
{: .hands_on}

> <question-title></question-title>
> How many sequences were removed in this step?
> > <solution-title></solution-title>
> > 4,579 sequences were removed. This is the number of lines in the bad.accnos output.
> {: .solution }
{: .question}

Now we know our sequences overlap the same alignment coordinates, we want to make sure they *only* overlap that region. So we'll filter the sequences to remove the overhangs at both ends. In addition, there are many columns in the alignment that only contain external gap characters (*i.e.* "."), while columns containing only internal gap characters (*i.e.*, "-") are not considered. These can be pulled out without losing any information. We'll do all this with `Filter.seqs`:

> <hands-on-title>Filter sequences</hands-on-title>
>
> 1. **Filter.seqs** {% icon tool %} with the following parameters
>   - "fasta"" to `good.fasta` output from `Screen.seqs`
>   - "trump" to `.`
{: .hands_on}


## Extraction of taxonomic information

The main questions when analyzing amplicon data are: Which micro-organisms are present in an environmental samples? And in what proportion? What is the structure of the community of the micro-organisms?

The idea is to take the sequences and assign them to a taxon. To do that, we group (or cluster) sequences based on their similarity to defined Operational Taxonomic Units (OTUs): groups of similar sequences that can be treated as a single "genus" or "species" (depending on the clustering threshold)

> <comment-title>Background: Operational Taxonomic Units (OTUs)</comment-title>
>
> In 16S metagenomics approaches, OTUs are clusters of similar sequence variants of the 16S rDNA marker gene sequence. Each of these clusters is intended to represent a taxonomic unit of a bacterial species or genus depending on the sequence similarity threshold. Typically, OTU clusters are defined by a 97% identity threshold of the 16S gene sequence variants at genus level. 98% or 99% identity is suggested for species separation.
>
> ![OTU and cluster with 97% identity threshold](../../images/otu.png "OTU and cluster with 97% identity threshold")
>
> ![OTU graph](../../images/OTU_graph.png "Cladogram of operational taxonomic units (OTUs). Credit: Danzeisen et al. 2013, 10.7717/peerj.237")
>
{: .comment}

The first thing we want to do is to further de-noise our sequences from potential sequencing errors, by pre-clustering the sequences using the `Pre.cluster` command, allowing for up to 2 differences between sequences. This command will split the sequences by group and then sort them by abundance, then go from most abundant to least and identify sequences that differ by no more than 2 nucleotides from on another. If this is the case, then they get merged. We generally recommend allowing 1 difference for every 100 basepairs of sequence:

> <hands-on-title>Perform preliminary clustering of sequences and remove undesired sequences</hands-on-title>
>
> 1. **Pre.cluster** {% icon tool %} with the following parameters
>   - "fasta" to the fasta output from the last `Filter.seqs` run
>   - "name file or count table" to the count table from the last `Screen.seqs` step
>   - "diffs" to 2
>
>   > <question-title></question-title>
>   >
>   >  How many unique sequences are we left with after this clustering of highly similar sequences?
>   > > <solution-title></solution-title>
>   > > 10,398
>   > >
>   > > This is the number of lines in the fasta output
>   > {: .solution }
>   {: .question}
>
{: .hands_on}

<!-- optional additional QC: chimera.uchime -->
We would like to classify the sequences using a training set, which is again is provided on [[mothur's MiSeq SOP](https://mothur.org/wiki/MiSeq_SOP)].

> <hands-on-title>Classify the sequences into phylotypes</hands-on-title>
>
> 1. Import the `trainset16_022016.pds.fasta` and `trainset16_022016.pds.tax` in your history
>
>    ```
>    https://zenodo.org/record/815875/files/trainset16_022016.pds.fasta
>    https://zenodo.org/record/815875/files/trainset16_022016.pds.tax
>    ```
>
> 2. **Classify.seqs** {% icon tool %} with the following parameters
>   - "fasta" to the fasta output from `Pre.cluster`
>   - "Select Reference Template from" to `History`
>   - "reference" to `trainset16_022016.pds.fasta` from your history
>   - "Select Taxonomy from" to `History`
>   - "taxonomy" to `trainset16_022016.pds.tax` from your history
>   - "count" to the count table from `Pre.cluster`
>
> This step may take a couple of minutes, now may be a good time to grab a cup of tea/coffee.
>
{: .hands_on}

Have a look at the taxonomy output.

```
name    taxonomy
SRR651839.9109-HXY9DLL01BSHQO-2 Bacteria(99);Verrucomicrobia(99);Spartobacteria(99);unclassified;unclassified;unclassified;
SRR651839.11437-HXY9DLL01AMIPI-2    Bacteria(100);Verrucomicrobia(97);Spartobacteria(97);unclassified;unclassified;unclassified;
SRR651839.15884-HXY9DLL01BDG2F-2    Bacteria(99);unclassified;unclassified;unclassified;unclassified;unclassified;
SRR651839.12048-HXY9DLL01DSOWS-2    Bacteria(100);Verrucomicrobia(100);Spartobacteria(100);unclassified;unclassified;unclassified;
SRR651839.9410-HXY9DLL01BGRG2-2 Bacteria(100);Proteobacteria(100);Alphaproteobacteria(100);Rhizobiales(100);Bradyrhizobiaceae(100);unclassified;
SRR651839.9029-HXY9DLL01E4W9Z-2 Bacteria(100);Verrucomicrobia(100);Spartobacteria(100);unclassified;unclassified;unclassified;
SRR651839.6283-HXY9DLL01CHX25-2 Bacteria(100);Acidobacteria(100);Acidobacteria_Gp4(100);Gp4(100);unclassified;unclassified;
SRR651839.3134-HXY9DLL01DOM67-2 Bacteria(100);Acidobacteria(100);Acidobacteria_Gp6(100);Gp6(100);unclassified;unclassified;
SRR651839.13044-HXY9DLL01ETN7L-2    Bacteria(100);Proteobacteria(100);Alphaproteobacteria(100);Rhizobiales(100);Bradyrhizobiaceae(100);unclassified;
SRR531818.61708-G88ZSJI01AVPPR-2    Bacteria(100);Acidobacteria(99);Acidobacteria_Gp6(99);Gp6(99);unclassified;unclassified;
```

You will see that every read now has a classification.

The next step is to use this information to determine the abundances of the different found taxa. This consists of three steps:
1. first, all individual sequences are classified, and get assigned a confidence score (0-100%)
2. next, sequences are grouped at 97% identity threshold (not using taxonomy info)
3. finally, for each cluster, a consensus classification is determined based on the classification of the individual sequences and taking their confidence scores into account

> <hands-on-title>Assign sequences to OTUs</hands-on-title>
>
> 1. **Cluster.split** {% icon tool %} with the following parameters
>   - "Split by" to `Classification using fasta`
>   - "fasta" to the fasta output from `Pre.cluster`
>   - "taxonomy" to the taxonomy output from `Classify.seqs`
>   - "count" to the count table output from `Pre.cluster`
>   - "Clustering method" to `Average Neighbour`
>   - "cutoff" to `0.15`
>
{: .hands_on}

We obtain a table with the columns being the different identified OTUs, the rows the different distances and the cells the ids of the sequences identified for these OTUs for the different distances.

Next we want to know how many sequences are in each OTU from each group with a distance of 0.03 (97% similarity). We can do this using the `Make.shared` command with the 0.03 cutoff level:

> <hands-on-title>Estimate OTU abundance</hands-on-title>
>
> 2. **Make.shared** {% icon tool %} with the following parameters
>   - "Select input type" to `OTU list`
>   - "list" to list output from `Cluster.split`
>   - "name file or count table" to the count table from `Pre.cluster`
>   - "label" to `0.03`
>
{: .hands_on}

We probably also want to know the taxonomy for each of our OTUs. We can get the consensus taxonomy for each OTU using the `Classify.otu` command:

> <hands-on-title>Classify the OTUs</hands-on-title>
>
> 3. **Classify.otu** {% icon tool %} with the following parameters
>   - "list" to output from `Cluster.split`
>   - "count" to the count table from `Pre.cluster`
>   - "Select Taxonomy from" to `History`
>   - "taxonomy" to the taxonomy output from `Classify.seqs`
>   - "label" to `0.03`
{: .hands_on}

> <question-title></question-title>
>
> 1. How many OTUs with taxonomic assignation are found for the Anguil sample? And for the Pampa sample?
> 2. What is the annotation of first OTU and its size?
>
> > <solution-title></solution-title>
> > 1. 2,195 for Anguil and 2,472 for Pampa ("tax.summary")
> > 2. Otu00001 is associated to 929 sequences and to Bacteria (kingdom), Verrucomicrobia (phylum), Spartobacteria (class) in "taxonomy" file
> {: .solution }
{: .question}

## Visualization

We have now determined our OTUs and classified them, but looking at a long text file is not very informative.
Let's visualize our data using Krona:

> <hands-on-title>Krona</hands-on-title>
>
>  First we convert our mothur taxonomy file to a format compatible with Krona
>
> - **Taxonomy-to-Krona** {% icon tool %} with the following parameters
>   - "Taxonomy file" to the taxonomy output from Classify.otu (note: this is a collection input)
>
> - **Krona pie chart** {% icon tool %} with the following parameters
>   - "Type of input" to `Tabular`
>   - "Input file" to taxonomy output from Taxonomy-to-Krona (collection)
{: .hands_on}

The result is an HTML file with an interactive visualization, for instance try clicking
on one of the rings in the image or playing around with some of the settings.

<iframe id="krona" src="krona.html" frameBorder="0" width="100%" height="900px"> ![Krona](../../images/krona.png) </iframe>

This produced a single plot for both your samples, but what if you want to compare
the two samples?

> <hands-on-title>Per-sample Krona plots</hands-on-title>
>
> 1. **Classify.otu** {% icon tool %}
>
>    Hit the rerun button on the `Classify.otu` job in your history and see if you can find settings that will give you per-sample taxonomy data
>
> 2. **Krona** {% icon tool %}
>
>    Now use this new output collection to create per-sample Krona plots
>
{: .hands_on}

In this new Krona output you can switch between the combined plot and the per-sample plots via the selector in the top-left corner.

> <question-title></question-title>
> Which soil sample had a higher percentage of Acidobacteria, anguil or pampa? what were the respective percentages?
> > <solution-title></solution-title>
> > The anguil sample had a higher proportion of Acidobacteria. The exact percentages can be found by looking at the pie charts at the
> > top right-hand corner after clicking on the label Acidobacteria. For anguil the percentage is 36%, for the pampa sample it is 26%.
> >
> > ![krona plot with acidobactaria highlighted](../../images/krona-multisample.png)
> {: .solution }
{: .question}


To further explore the community structure, we can visualize it with dedicated tools such as Phinch.

> <hands-on-title>Visualization of the community structure with Phinch</hands-on-title>
>
> 1. **Make.biom** {% icon tool %} with the following parameters
>   - "shared" to `Make.shared` output
>   - "constaxonomy" to taxonomy output from the first run of `Classify.otu` (collection)
> 1. Expand the dataset and click on the "view biom at phinch" link
>
>     > <comment-title></comment-title>
>     >
>     > If this link is not present on your Galaxy, you can download the generated BIOM file and upload directly to Phinch server at [http://phinch.org](http://phinch.org).
>    {: .comment}
>
> 2. Play with the several interactive visualisations:
>
> ![Phinch website interface](../../../../shared/images/phinch_overviewpage.png "Phinch visualizations")
>
{: .hands_on}

Once we have information about the community structure (OTUs with taxonomic structure), we can do more analysis on it: estimation of the diversity of micro-organism, comparison of diversity between samples, analysis of populations, ... We will not go into detail of such analyses here but you can follow our tutorials on amplicon data analyses to learn more about them.

# Shotgun metagenomics data

In the previous section, we saw how to analyze amplicon data to extract the community structure. Such information can also be extracted from shotgun metagenomic data.

In shotgun data, full genomes of the micro-organisms in the environment are sequenced (not only the 16S or 18S). We can then have access to the rRNA (only a small part of the genomes), but also to the other genes of the micro-organisms. Using this information, we can try to answer questions such as "What are the micro-organisms doing?" in addition to the question "What micro-organisms are present?".

In this second part, we will use a metagenomic sample of the Pampas Soil ([SRR606451](https://www.ebi.ac.uk/metagenomics/projects/SRP016633/samples/SRS372043/runs/SRR606451/results/versions/2.0)).

## Data upload

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history
> 2. Import the `SRR606451_pampa` Fasta file from [Zenodo](http://zenodo.org/record/815875) or from the data library (in "Analyses of metagenomics data")
>
>    ```
>    https://zenodo.org/record/815875/files/SRR606451_pampa.fasta
>    ```
>
{: .hands_on}

## Extraction of taxonomic information

As for amplicon data, we can extract taxonomic and community structure information from shotgun data. Different approaches can be used:

- Same approach as for amplicon data with identification and classification of OTUs

    Such an approach requires a first step of sequence sorting to extract only the 16S and 18S sequences, then using the same tools as for amplicon data. However, rRNA sequences represent a low proportion (< 1%) of the shotgun sequences so such an approach is not the most statistically supported

- Assignation of taxonomy on the whole sequences using databases with marker genes

In this tutorial, we use the second approach with MetaPhlAn2. This tools is using a database of ~1M unique clade-specific marker genes (not only the rRNA genes) identified from ~17,000 reference (bacterial, archeal, viral and eukaryotic) genomes.

> <hands-on-title>Taxonomic assignation with MetaPhlAn2</hands-on-title>
>
> 1. **MetaPhlAN2** {% icon tool %} with
>    - "Input file" to the imported file
>    - "Database with clade-specific marker genes" to `locally cached`
>    - "Cached database with clade-specific marker genes" to `MetaPhlAn2 clade-specific marker genes`
>
> This step may take a couple of minutes.
{: .hands_on}

3 files are generated:

- A tabular file with the community structure

    ```
    #SampleID   Metaphlan2_Analysis
    k__Bacteria 100.0
    k__Bacteria|p__Proteobacteria   86.20712
    k__Bacteria|p__Actinobacteria   13.79288
    k__Bacteria|p__Proteobacteria|c__Gammaproteobacteria    86.20712
    k__Bacteria|p__Actinobacteria|c__Actinobacteria 13.79288
    ```

    Each line contains a taxa and its relative abundance found for our sample. The file starts with high level taxa (kingdom: `k__`) and go to more precise taxa.


- A BIOM file with the same information as the previous file but in BIOM format

    It can be used then by mothur and other tools requiring community structure information in BIOM format

- A SAM file with the results of the mapping of the sequences on the reference database

> <question-title></question-title>
>
> 1. What is the most precise level we have access to with MetaPhlAn2?
> 2. What are the two orders found in our sample?
> 3. What is the most abundant family in our sample?
>
> > <solution-title></solution-title>
> > 1. We have access to species level
> > 2. Pseudomonadales and Solirubrobacterales are found in our sample
> > 3. The most abundant family is Pseudomonadaceae with 86.21 % of the assigned sequences
> {: .solution }
{: .question}

Even if the output of MetaPhlAn2 is bit easier to parse than the BIOM file, we want to visualize and explore the community structure with KRONA

> <hands-on-title>Interactive visualization with KRONA</hands-on-title>
>
> 1. **Format MetaPhlAn2 output for Krona** {% icon tool %} with
>    - "Input file" to `Community profile` output of `MetaPhlAn2`
>
> 2. **KRONA pie chart** {% icon tool %} with
>    - "What is the type of your input data" as `MetaPhlan`
>    - "Input file" to the output of `Format MetaPhlAn2`
>
{: .hands_on}

## Extraction of functional information

We would now like to answer the question "What are the micro-organisms doing?" or "Which functions are performed by the micro-organisms in the environment?".

In the shotgun data, we have access to the gene sequences from the full genome. We use that to identify the genes, associate them with a function, build pathways, etc., to investigate the functional part of the community.

> <hands-on-title>Metabolism function identification</hands-on-title>
>
> 1. **HUMAnN2** {% icon tool %} with
>    - "Input sequence file" to the imported sequence file
>    - "Use of a custom taxonomic profile" to `Yes`
>    - "Taxonomic profile file" to `Community profile` output of `MetaPhlAn2`
>    - "Nucleotide database" to `Locally cached`
>    - "Nucleotide database" to `Full`
>    - "Protein database" to `Locally cached`
>    - "Protein database" to `Full UniRef50`
>    - "Search for uniref50 or uniref90 gene families?" to `uniref50`
>    - "Database to use for pathway computations" to `MetaCyc`
>    - "Advanced Options"
>    - "Remove stratification from output" to `Yes`
>
>    This step is long so we generated the output for you!
>
> 2. Import the 3 files whose the name is starting with "humann2"
>
>    ```
>    https://zenodo.org/record/815875/files/humann2_gene_families_abundance.tsv
>    https://zenodo.org/record/815875/files/humann2_pathways_abundance.tsv
>    https://zenodo.org/record/815875/files/humann2_pathways_coverage.tsv
>    ```
{: .hands_on}

HUMAnN2 generates 3 files

- A file with the abundance of gene families

    Gene family abundance is reported in RPK (reads per kilobase) units to normalize for gene length. It reflects the relative gene (or transcript) copy number in the community.

    The "UNMAPPED" value is the total number of reads which remain unmapped after both alignment steps (nucleotide and translated search). Since other gene features in the table are quantified in RPK units, "UNMAPPED" can be interpreted as a single unknown gene of length 1 kilobase recruiting all reads that failed to map to known sequences.

- A file with the coverage of pathways

    Pathway coverage provides an alternative description of the presence (1) and absence (0) of pathways in a community, independent of their quantitative abundance.

- A file with the abundance of pathways

> <question-title></question-title>
>
> How many gene families and pathways have been identified?
>
> > <solution-title></solution-title>
> > 44 gene families but no pathways are identified
> {: .solution }
{: .question}

The RPK for the gene families are quite difficult to interpret in term of relative abundance. We decide then to normalize the values

> <hands-on-title>Normalize the gene family abundances</hands-on-title>
>
> 1. **Renormalize a HUMAnN2 generated table** {% icon tool %} with
>    - "Gene/pathway table" to the gene family table generated with `HUMAnN2`
>    - "Normalization scheme" to `Relative abundance`
>    - "Normalization level" to `Normalization of all levels by community total`
>
>  > <question-title></question-title>
>  >
>  > 1. What percentage of sequences has not be assigned to a gene family?
>  > 2. What is the most abundant gene family?
>  >
>  > > <solution-title></solution-title>
>  > > 1. 55% of the sequences have not be assigned to a gene family
>  > > 2. The most abundant gene family with 25% of sequences is a putative secreted protein
>  > {: .solution }
>  {: .question}
{: .hands_on}

With the previous analyses, we investigate "Which micro-organims are present in my sample?" and "What function are performed by the micro-organisms in my sample?". We can go further in these analyses (for example, with a combination of functional and taxonomic results). We did not detail that in this tutorial but you can find more analyses in our tutorials on shotgun metagenomic data analyses.

# Conclusion


We can summarize the analyses with amplicon and shotgun metagenomic data:

![Scheme to sum up the analysis](../../images/general-tutorial-scheme.png)

Both analyses are quite complex! However, in this tutorial, we only showed simple cases of metagenomics data analysis with subset of real data.

Check our other tutorials to learn more in detail of how to analyze metagenomics data.


topic: microbiome
tutorial: clinical-mp-3-verification
title: Clinical Metaproteomics 3: Verification
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: ["[label-TMT11]"]
content_hash: 57dfeaab22bca56c0134b9ccb961a7f3
last_modified: 2026-05-27T09:01:49.453901
zenodo_link: https://doi.org/10.5281/zenodo.10105821
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/clinical-mp-3-verification/tutorial.html

content:
In proteomic research, the primary goal is to obtain accurate and meaningful insights into the proteome of a biological system. Verifying the presence of peptides or proteins is a critical step in achieving this goal, ensuring the quality and reliability of the data and the biological relevance of the findings. This tutorial is a sequel to the clinical metaproteomics discovery workflow. Once you have identified microbial peptides, the next step is to verify these peptides, for which we use PepQuery.

The PepQuery tool is used to validate the identified microbial peptides from SearchGUI/PeptideShaker and MaxQuant, to ensure that they are indeed of microbial origin and that human peptides were not misassigned. To do this, all confident microbial peptides from the two database search algorithms were merged and searched against the Human UniProt Reference proteome (with Isoforms) and cRAP databases.

Interestingly, the PepQuery tool does not rely on searching peptides against a reference protein sequence database as “traditional” shotgun proteomics does, which enables it to identify novel, disease-specific sequences with sensitivity and specificity in its protein validation (Figure A). Then we extract microbial protein sequences that are assigned to the PepQuery verified peptides. To this, we again add the Human UniProt Reference proteome (with Isoforms) and cRAP databases for creating a database for quantitation purposes (Figure B).

![Peptide Verification]({% link topics/proteomics/images/clinical-mp/clinical-mp-verification-1.JPG %})

![Database generation from verified peptides]({% link topics/proteomics/images/clinical-mp/clinical-mp-verification-2.JPG %})


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F10_9Aug19_Rage_Rep-19-06-08.mgf
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F11_9Aug19_Rage_Rep-19-06-08.mgf
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F13_9Aug19_Rage_Rep-19-06-08.mgf
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F15_9Aug19_Rage_Rep-19-06-08.mgf
>    https://zenodo.org/records/10105821/files/SGPS_Peptide_Report.tabular
>    https://zenodo.org/records/10105821/files/MaxQuant_Peptide_Report.tabular
>    https://zenodo.org/records/10105821/files/Distinct_Peptides_for_PepQuery.tabular
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Add to each database a tag corresponding to input files.
> 6. Users can create a database collection of the MGF files.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}


# Import Workflow

> <hands-on-title>Running the Workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/proteomics/tutorials/clinical-mp-3-verification/workflows/WF3_Verification_Workflow.ga" title="Verification Workflow" %}
>
> 2. Run **Workflow** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *" Input MGFs Dataset Collection "*: `MGF dataset collection`
>    - {% icon param-file %} *" SGPS_peptide-report"*: `SGPS_Peptide_Report.tabular`
>    - {% icon param-file %} *" Distinct Peptides for PepQuery"*: `Distinct_Peptides_for_PepQuery.tabular`
>    - {% icon param-file %} *" MaxQuant-peptide-report "*: `MaxQuant_Peptide_Report.tabular`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}


# Extraction of Microbial Peptides from SearchGUI/PeptideShaker and MaxQuant
Now that we have identified microbial peptides from SearchGUI/PeptideShaker and MaxQuant, we need to extract the microbial peptide sequences and group them to obtain a list of distinct microbial peptides. This list of distinct peptides will be used as input for PepQuery2 to verify confident microbial peptides.

First, we will use the Cut tool to select the peptide and protein columns from the SearchGUI/PeptideShaker and MaxQuant Peptide Reports. Then we use Remove header lines from SGPS and MaxQuant to prepare for concatenation with Remove beginning.

> <hands-on-title> Extracting peptides </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c6,c2`
>    - {% icon param-file %} *"From"*: `output` (Input dataset)
>
>
> 2. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1,c35`
>    - {% icon param-file %} *"From"*: `output` (Input dataset)
>
>
> 3. {% tool [Remove beginning](Remove beginning1) %} with the following parameters:
>    - {% icon param-file %} *"from"*: `out_file1` (output of **Cut** {% icon tool %})
>
>
> 4. {% tool [Remove beginning](Remove beginning1) %} with the following parameters:
>    - {% icon param-file %} *"from"*: `out_file1` (output of **Cut** {% icon tool %})
>
>
{: .hands_on}


## Concatenate peptides from MaxQuant and SGPS for PepQuery2

We will now concatenate the peptide and protein datasets from SearchGUI/PeptideShaker and MaxQuant. Later, we will generate a list of confident peptides using PepQuery2. The list of confident peptides will be searched against the concatenated peptide-protein datasets from SearchGUI/PeptideShaker and MaxQuant to generate a list of verified peptides.

> <hands-on-title> Concatenate SGPS and MaxQuant peptides </hands-on-title>
>
> 1. {% tool [Concatenate datasets](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/0.1.1) %} with the following parameters:
>    - {% icon param-files %} *"Datasets to concatenate"*: `out_file1` (output of **Remove beginning** {% icon tool %}), `out_file1` (output of **Remove beginning** {% icon tool %})
>
>
{: .hands_on}


## Creating input database for PepQuery2

We generate and merge Human UniProt (with Isoforms) and contaminants (cRAP) to make an input database for PepQuery2.

> <hands-on-title> FASTA Merge Files and Filter Unique Sequences</hands-on-title>
>
> 1. {% tool [FASTA Merge Files and Filter Unique Sequences](toolshed.g2.bx.psu.edu/repos/galaxyp/fasta_merge_files_and_filter_unique_sequences/fasta_merge_files_and_filter_unique_sequences/1.2.0) %} with the following parameters:
>    - *"Run in batch mode?"*: `Merge individual FASTAs (output collection if input is collection)`
>        - In *"Input FASTA File(s)"*:
>            - {% icon param-repeat %} *"Insert Input FASTA File(s)"*
>                - {% icon param-file %} *"FASTA File"*: `Human UniProt+Isoforms FASTA` (output of **Protein Database Downloader** {% icon tool %})
>                - - {% icon param-file %} *"FASTA File"*: `cRAP database` (output of **Protein Database Downloader** {% icon tool %})
>
>
{: .hands_on}



## Peptide verification
The PepQuery2 tool will be used to validate the identified microbial peptides from SearchGUI/PeptideShaker and MaxQuant to ensure that they are indeed of microbial origin and that human peptides were not misassigned. We will use the list of `Distinct Peptides` (from the [Discovery Module](https://github.com/subinamehta/training-material/blob/main/topics/proteomics/tutorials/clinical-mp-discovery/tutorial.md)), `Human UniProt+Isoforms+cRAP database`, and our `MGF file collection` as inputs for PepQuery2. The outputs we are interested in are the four PSM Rank (txt) files (one for each MGF file).

Interestingly, the PepQuery2 tool does not rely on searching peptides against a reference protein sequence database as “traditional” shotgun proteomics does, which enables it to identify novel, disease-specific sequences with sensitivity and specificity in its protein validation. More information about PepQuery is available, including the first {% cite Wen2019 %} and second iterations {% cite Wen2023 %}.


> <hands-on-title> Peptide verification </hands-on-title>
>
> 1. {% tool [PepQuery2](toolshed.g2.bx.psu.edu/repos/galaxyp/pepquery2/pepquery2/2.0.2+galaxy0) %} with the following parameters:
>    - *"Validation Task Type"*: `novel peptide/protein validation`
>    - In *"Input Data"*:
>        - *"Input Type"*: `peptide`
>            - *"Peptides?"*: `Peptide list from your history`
>                - {% icon param-file %} *"Peptide Sequences (.txt)"*: `output` (Input dataset)
>        - *"Protein Reference Database from"*: `history`
>            - {% icon param-file %} *"Protein Reference Database File"*: `output` (output of **FASTA Merge Files and Filter Unique Sequences** {% icon tool %})
>        - *"MS/MS dataset to search"*: ` Spectrum Datasets from history`
>            - {% icon param-collection %} *"Spectrum File"*: `output` (Input dataset collection)
>        - *"Report Spectrum Scan as"*: `spectrum title in MGF`
>    - In *"Modifications"*:
>        - *"Fixed modification(s)"*: `1: Carbamidomethylation of C [57.02146372057]` `13: TMT 11-plex of K [229.16293213472]` `14: TMT 11-plex of peptide N-term [229.16293213472]`
>        - *"Variable modification(s)"*: `2: Oxidation of M [15.99491461956]`
>        - *"Use more stringent criterion for unrestricted modification searching"*: `Yes`
>        - *"Consider amino acid substitution modifications?"*: `Yes`
>    - In *"Digestion"*:
>        - *"Enzyme"*: `Trypsin`
>        - *"Max Missed Cleavages"*: `2`
>    - In *"Mass spectrometer"*:
>        - In *"Tolerance"*:
>            - *"Precursor Tolerance"*: `10`
>            - *"Precursor Unit"*: `ppm`
>            - *"Tolerance"*: `0.6`
>        - In *"PSM"*:
>            - *"Fragmentation Method"*: `CID/HCD`
>            - *"Scoring Method"*: `HyperScore`
>            - *"Minimum Charge"*: `2`
>            - *"Maximum Charge"*: `6`
>
>
{: .hands_on}

# Collapsing all the data

Remember that PepQuery2 generates a PSM Rank file for each input MGF file, so we will have four PSM Rank files. To make the analysis more efficient, we will collapse these four PSM Rank files into one dataset.


> <hands-on-title> Collasping PSM rank files into a singular dataset using Collapse Collection </hands-on-title>
>
> 1. {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0) %} with the following parameters:
>    - {% icon param-file %} *"Collection of files to collapse into single dataset"*: `psm_rank_txt` (output of **PepQuery2** {% icon tool %})
>    - *"Keep one header line"*: `Yes`
>
>
{: .hands_on}



## Filtering out confident peptides

Now, we want to filter for confident peptides from PepQuery2 and prepare them for the Query Tabular tool.

> <hands-on-title> Filter </hands-on-title>
>
> 1. {% tool [Filter](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `output` (output of **Collapse Collection** {% icon tool %})
>    - *"With following condition"*: `c20=='Yes'`
>    - *"Number of header lines to skip"*: `1`
>
>
{: .hands_on}


> <hands-on-title> Remove header line from filtered PepQuery peptides with Remove beginning </hands-on-title>
>
> 1. {% tool [Remove beginning](Remove beginning1) %} with the following parameters:
>    - {% icon param-file %} *"from"*: `out_file1` (output of **Filter** {% icon tool %})
>
>
{: .hands_on}


> <hands-on-title> Cut (select out) peptide sequences from PepQuery output with Cut </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - {% icon param-file %} *"From"*: `out_file1` (output of **Remove beginning** {% icon tool %})
>
>
{: .hands_on}


##  **Querying verified peptides**

We will use the Query Tabular tool {% cite Johnson2019 %} to search the PepQuery-verified peptides against the concatenated dataset that contains peptides and proteins from SearchGUI/Peptide and MaxQuant. This step ensures all the PepQuery-verified peptides are assigned to their protein/protein groups.

> <hands-on-title> Querying verified peptides </hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.0) %} with the following parameters:
>    - In *"Database Table"*:
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `out_file1` (output of **Cut** {% icon tool %})
>            - In *"Table Options"*:
>                - *"Specify Name for Table"*: `pep`
>                - *"Specify Column Names (comma-separated list)"*: `mpep`
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `out_file1` (output of **Concatenate datasets** {% icon tool %})
>            - In *"Table Options"*:
>                - *"Specify Name for Table"*: `prot`
>                - *"Specify Column Names (comma-separated list)"*: `pep,prot`
>    - *"SQL Query to generate tabular output"*: `select pep.mpep, prot.prot
>    `FROM pep
>    INNER JOIN  prot on pep.mpep=prot.pep `
>    - *"include query result column headers"*: `Yes`
>    `
>
>
> > <comment-title>SQL Query information</comment-title>
> >  The query input files are the list of peptides and the peptide report we obtained from MaxQuant and SGPS. The query is matching each peptide (m.pep) from the PepQuery results to the peptide reports so that each verified peptide has its protein/protein group assigned to it.
> {: .comment}
{: .hands_on}

> <hands-on-title> Remove Header with Remove beginning </hands-on-title>
>
> 1. {% tool [Remove beginning](Remove beginning1) %} with the following parameters:
>    - {% icon param-file %} *"from"*: `output` (output of **Query Tabular** {% icon tool %})
>
>
{: .hands_on}


Using the Group tool, we can select distinct (unique) peptides and proteins from the Query Tabular tool.

> <hands-on-title> Extract distinct peptides with Group </hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `out_file1` (output of **Remove beginning** {% icon tool %})
>    - *"Group by column"*: `c1`
>    - In *"Operation"*:
>        - {% icon param-repeat %} *"Insert Operation"*
>            - *"Type"*: `Concatenate Distinct`
>            - *"On column"*: `c2`
>
>
{: .hands_on}


## Retrieve UniProt IDs for distinct peptides

Again, we will use the Query Tabular tool to retrieve UniProt IDs (accession numbers) for the distinct (grouped) peptides.

> <hands-on-title> Query Tabular </hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.0) %} with the following parameters:
>    - In *"Database Table"*:
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `out_file1` (output of **Group** {% icon tool %})
>            - In *"Filter Dataset Input"*:
>                - In *"Filter Tabular Input Lines"*:
>                    - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>                        - *"Filter By"*: `normalize list columns, replicates row for each item in list`
>                            - *"enter column numbers to normalize"*: `2`
>                            - *"List item delimiter in column"*: `;`
>                    - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>                        - *"Filter By"*: `regex replace value in column`
>                            - *"enter column number to replace"*: `2`
>                            - *"regex pattern"*: `(tr|sp)[|]`
>                    - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>                        - *"Filter By"*: `regex replace value in column`
>                            - *"enter column number to replace"*: `2`
>                            - *"regex pattern"*: `[ ]+`
>                    - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>                        - *"Filter By"*: `regex replace value in column`
>                            - *"enter column number to replace"*: `2`
>                            - *"regex pattern"*: `[|].*$`
>            - In *"Table Options"*:
>                - *"Specify Name for Table"*: `t1`
>                - *"Use first line as column names"*: `Yes`
>                - *"Specify Column Names (comma-separated list)"*: `pep,prot`
>    ` *"SQL Query to generate tabular output"*: `SELECT distinct(prot) AS Accession
>       from t1`
>        - *"include query result column headers"*: `No`
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. What is the accession number of a protein?
> 2. Can there be multiple accession numbers for one peptide or protein?
>
> > <solution-title></solution-title>
> >
> > 1. An accession number of a protein, also called a protein accession number, is a unique identifier assigned to a specific protein sequence in a protein sequence database. These accession numbers are used to reference and catalog proteins in a standardized and systematic manner
> >
> > 2. Yes, a single peptide or protein can have multiple accession numbers, particularly when dealing with different protein sequence databases, databases for specific species, or different versions of the same database. That's the reason in our workflow we merge both accession and sequences.
> {: .solution}
>
{: .question}


## Generate FASTA database from UniProt IDs
Using the UniProt IDs from Query Tabular, we will be able to generate a FASTA database for our PepQuery-verified peptides.

> <hands-on-title> UniprotXML-downloader </hands-on-title>
>
> 1. {% tool [UniProt](toolshed.g2.bx.psu.edu/repos/galaxyp/uniprotxml_downloader/uniprotxml_downloader/2.4.0) %} with the following parameters:
>    - *"Select"*: `A history dataset with a column containing Uniprot IDs`
>        - {% icon param-file %} *"Dataset (tab separated) with ID column"*: `output` (Input dataset)
>        - *"Column with ID"*: `c1`
>        - *"Field"*: `Accession`
>    - *"uniprot output format"*: `fasta`
>
>
{: .hands_on}


## Generating compact database

Lastly, we will merge the Human UniProt (with isoforms), contaminants (cRAP) and the PepQuery-verified FASTA databases into one Quantitation Database that will be used as input for the [Quantification Module](https://github.com/subinamehta/training-material/blob/main/topics/proteomics/tutorials/clinical-mp-quantitation/tutorial.md).

> <hands-on-title> Generation of Compact Verified Database with UniProt </hands-on-title>
>
> 1. {% tool [FASTA Merge Files and Filter Unique Sequences](toolshed.g2.bx.psu.edu/repos/galaxyp/fasta_merge_files_and_filter_unique_sequences/fasta_merge_files_and_filter_unique_sequences/1.2.0) %} with the following parameters:
>    - *"Run in batch mode?"*: `Merge individual FASTAs (output collection if input is collection)`
>        - In *"Input FASTA File(s)"*:
>            - {% icon param-repeat %} *"Insert Input FASTA File(s)"*
>                - {% icon param-file %} *"FASTA File"*: `proteome` (output of **UniProt** {% icon tool %})
>
>
{: .hands_on}


# Conclusion

A peptide verification workflow is a critical step in proteomic research that enhances data reliability, quantitative accuracy, and biological understanding by confirming the presence and validity of selected peptides. It is a pivotal quality control process that ensures the trustworthiness of proteomic findings and supports downstream investigations. By completing this tutorial, you have not only verified the microbial peptides but also created a database consisting of protein sequences from the PepQuery-verified peptides. The need of such a database is to ensure that when we quantify our proteins and peptides we are reducing the introduction of false positives. This database will be now used for quantitation purposes.

tools_json:
["Cut", "Cut1", "Remove beginning", "Remove beginning1", "Concatenate datasets", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/0.1.1", "FASTA Merge Files and Filter Unique Sequences", "toolshed.g2.bx.psu.edu/repos/galaxyp/fasta_merge_files_and_filter_unique_sequences/fasta_merge_files_and_filter_unique_sequences/1.2.0", "PepQuery2", "toolshed.g2.bx.psu.edu/repos/galaxyp/pepquery2/pepquery2/2.0.2+galaxy0", "Collapse Collection", "toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0", "Filter", "Filter1", "Query Tabular", "toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.0", "Group", "Grouping1", "UniProt", "toolshed.g2.bx.psu.edu/repos/galaxyp/uniprotxml_downloader/uniprotxml_downloader/2.4.0"]


