# tutorials

topic: epigenetics
tutorial: tal1-binding-site-identification
title: Identification of the binding sites of the T-cell acute lymphocytic leukemia protein 1 (TAL1)
difficulty: intermediate
time_estimation: 3h
tools_json: []
requirements_json: []
tags_json: ["ChIP-seq"]
content_hash: 2ad9f6e741ba47290b8d419a04973d64
last_modified: 2026-02-26T21:12:15.681215
zenodo_link: https://doi.org/10.5281/zenodo.197100
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/epigenetics/tutorials/tal1-binding-site-identification/tutorial.html

content:
This tutorial uses ChIP-seq datasets from a study published by {% cite Wu_2014 %}. The goal of this study was to investigate "the dynamics of occupancy and the role in gene regulation of the transcription factor TAL1, a critical regulator of hematopoiesis, at multiple stages of hematopoietic differentiation."

To this end, ChIP-seq experiments were performed in multiple mouse cell types including G1E - a GATA-null immortalized cell line derived from targeted disruption of GATA-1 in mouse embryonic stem cells - and megakaryocytes.

This dataset (GEO Accession: [GSE51338](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE51338)) consists of biological replicate TAL1 ChIP-seq and input control experiments.
Input control experiments are used to identify and remove sampling bias, for example open/accessible chromatin or GC bias.

Because of the long processing time for the large original files, we have downsampled the original raw data files to include only reads that align to chromosome 19 and a subset of interesting genomic loci identified by {% cite Wu_2014 %}.

**Table 1**: Metadata for ChIP-seq experiments in this tutorial. SE: single-end.

| Cellular state | Datatype | ChIP Ab | Replicate | SRA Accession | Library type | Read length | Stranded? | Data size (MB) |
| ---            | ---      | :-:     | :-:       | ---           | :-:          | :-:         | :-:       | ---            |
| G1E            | ChIP-seq | input   | 1         | SRR507859     | SE           | 36          | No        | 35.8           |
| G1E            | ChIP-seq | input   | 2         | SRR507860     | SE           | 55          | No        | 427.1          |
| G1E            | ChIP-seq | TAL1    | 1         | SRR492444     | SE           | 36          | No        | 32.3           |
| G1E            | ChIP-seq | TAL1    | 2         | SRR492445     | SE           | 41          | No        | 62.7           |
| Megakaryocyte  | ChIP-seq | input   | 1         | SRR492453     | SE           | 41          | No        | 57.2           |
| Megakaryocyte  | ChIP-seq | input   | 2         | SRR492454     | SE           | 55          | No        | 403.8          |
| Megakaryocyte  | ChIP-seq | TAL1    | 1         | SRR549006     | SE           | 55          | No        | 340.3          |
| Megakaryocyte  | ChIP-seq | TAL1    | 2         | SRR549007     | SE           | 48          | No        | 356.9          |

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Quality control

As for any NGS data analysis, ChIP-seq data must be quality controlled before being aligned to a reference genome. For more detailed information on NGS quality control, check [out the tutorials]({% link topics/sequence-analysis/index.md %}).

> <hands-on-title>Performing quality control</hands-on-title>
>
> 1. Create and name a new history for this tutorial.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the ChIP-seq raw data (\*.fastqsanger) from [Zenodo](https://doi.org/10.5281/zenodo.197100).
>
>    ```
>    https://zenodo.org/record/197100/files/G1E_input_R1_downsampled_SRR507859.fastqsanger
>    https://zenodo.org/record/197100/files/G1E_input_R2_downsampled_SRR507860.fastqsanger
>    https://zenodo.org/record/197100/files/G1E_Tal1_R1_downsampled_SRR492444.fastqsanger
>    https://zenodo.org/record/197100/files/G1E_Tal1_R2_downsampled_SRR492445.fastqsanger
>    https://zenodo.org/record/197100/files/Megakaryocyte_input_R1_downsampled_SRR492453.fastqsanger
>    https://zenodo.org/record/197100/files/Megakaryocyte_input_R2_downsampled_SRR492454.fastqsanger
>    https://zenodo.org/record/197100/files/Megakaryocyte_Tal1_R1_downsampled_SRR549006.fastqsanger
>    https://zenodo.org/record/197100/files/Megakaryocytes_Tal1_R2_downsampled_SRR549007.fastqsanger
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Examine the data in a FASTQ file by clicking on the {% icon galaxy-eye %} (eye) icon.
>
>    > <question-title></question-title>
>    >
>    > 1. What are four key features of a FASTQ file?
>    > 2. What is the main difference between a FASTQ and a FASTA file?
>    >
>    > > <solution-title></solution-title>
>    > > 1. A FASTQ file contains a sequence identifier and additional information, the raw sequence, information about the sequence again with optional information, and quality information about the sequence.
>    > > 2. A FASTA file contains only the description of the sequence and the sequence itself. A FASTA file does not contain any quality information.
>    > {: .solution }
>    {: .question}
>
> 4. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72+galaxy1) %}: Run FastQC on each FASTQ file to assess the quality of the raw data. An explanation of the results can be found on the [FastQC web page](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/).
>    - {% icon param-files %} *"Short read data from your current history"*: The uploaded fastqsanger files.
>
>    ![fastqbefore](../../images/tal1/fastqc_before.png "Sequence quality per base generated by FastQC before end trimming.")
>
>    > <question-title></question-title>
>    >
>    > 1. What does the y-axis represent in Figure 3?
>    > 2. Why is the quality score decreasing across the length of the reads?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The phred-score. This score gives the probability of an incorrect base *e.g.* a score of 20 means that it is likely by 1% that one base is incorrect. See [the wikipedia page on Phred](https://en.wikipedia.org/wiki/Phred_quality_score) for more information.
>    > > 2. This is an unsolved technical issue of the sequencing machines. The longer the sequences are the more likely are errors. See [this article](https://www.ecseq.com/support/ngs/why-does-the-sequence-quality-decrease-over-the-read-in-illumina) for more information.
>    > {: .solution }
>    {: .question}
{: .hands_on}

# Trimming and clipping reads

It is often necessary to trim a sequenced read to remove bases sequenced with high uncertainty (*i.e.* low-quality bases). In addition, artificial adaptor sequences used in library preparation protocols need to be removed before attempting to align the reads to a reference genome.

> <hands-on-title>Trimming and clipping reads</hands-on-title>
>
> 1. {% tool [Trimmomatic](toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.38.0) %}: Run Trimmomatic to trim low-quality reads.
>    - *"Single-end or paired-end reads?"*: `Single-end`
>    - {% icon param-files %} *"Input FASTQ file"*: Select all of the FASTQ files
>    - *"Perform initial ILLUMINACLIP?"*: `No`
>    - *"Select Trimmomatic operation to perform"*: `Sliding window trimming (SLIDINGWINDOW)`
>    - *"Number of bases to average across"*: `4`
>    - *"Average quality required"*: `20`
>
>    > <tip-title>Changing datatypes</tip-title>
>    >
>    > If the FASTQ files cannot be selected, check whether their format is FASTQ with Sanger-scaled quality values (*fastqsanger*). If not, you can edit the data type by clicking on the pencil symbol next to a file in the history, clicking the "Datatype" tab, and choosing *fastqsanger* as the "New Type".
>    {: .tip}
>
> 2. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72+galaxy1) %}: Rerun FastQC on each trimmed/clipped FASTQ file to determine whether low-quality and adaptor sequences were correctly removed.
>    - {% icon param-files %} *"Short read data from your current history"*: The output of Trimmomatic.
>
>    > <question-title></question-title>
>    >
>    > 1. How did the range of read lengths change after trimming/clipping?
>    >
>    > > <solution-title></solution-title>
>    > > 1. Before trimming, all the reads were the same length, which reflected the number of rounds of nucleotide incorporation in the sequencing experiment. After trimming, read lengths span a range of values reflecting different lengths of the actual DNA fragments captured during the ChIP experiement.
>    > {: .solution }
>    {: .question}
>
>    ![fastqafter](../../images/tal1/fastqc_after.png "Sequence quality per base generated by FastQC after end trimming.")
{: .hands_on}

# Aligning reads to a reference genome

To determine where DNA fragments originated from in the genome, the sequenced reads must be aligned to a reference genome. This is equivalent to solving a jigsaw puzzle, but unfortunately, not all pieces are unique. In principle, you could do a BLAST analysis to figure out where the sequenced pieces fit best in the known genome. Aligning millions of short sequences this way, however, can take a couple of weeks.
Nowadays, there are many read alignment programs for sequenced DNA, BWA being one of them. You can read more about the BWA algorithm and tool in {% cite Li_2009 %}.

> <hands-on-title>Aligning reads to a reference genome</hands-on-title>
>
> 1. {% tool [BWA](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa/0.7.17.4) %}: Run BWA to map the trimmed/clipped reads to the mouse genome.
>    - *"Will you select a reference genome..."*: `Use a built-in genome index`
>    - *"Using reference genome"*: `Mouse (mus musculus) mm10`
>    - *"Select input type"*: `Single fastq`
>    - {% icon param-files %} *"Select fastq dataset"*: Select all of the trimmed FASTQ files
>
> 2. Rename files to reflect the origin and contents.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. Inspect a file produced by BWA.
>
>    > <question-title></question-title>
>    >
>    > 1. What datatype is the BWA output file?
>    > 2. How many reads were mapped from each file?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The output is a BAM file.
>    > > 2. Check the number of lines for each file in your history. This gives you a rough estimate.
>    > {: .solution }
>    {: .question}
>
> 4. {% tool [Samtools idxstats](toolshed.g2.bx.psu.edu/repos/devteam/samtools_idxstats/samtools_idxstats/2.0.3) %}: Run idxstats to get statistics of the BWA alignments.
>     - {% icon param-files %} *"BAM file"*: Select all of the mapped BAM files
>
> 5. Examine the output ({% icon galaxy-eye %})
>
>    > <question-title></question-title>
>    >
>    > 1. What does each column in the output represent (**Tip**: look at the Tool Form)?
>    > 2. How many reads were mapped to chromosome 19 in each experiment?
>    > 3. If the mouse genome has 21 pairs of chromosomes, what are the other reference chromosomes (*e.g.* chr1_GL456210_random)?
>    >
>    > > <solution-title></solution-title>
>    > > 1. Column 1: Reference sequence identifier
>    > >    Column 2: Reference sequence length
>    > >    Column 3: Number of mapped reads
>    > >    Column 4: Number of placed but unmapped reads (typically unmapped partners of mapped reads)
>    > > 2. This information can be seen in column 3, *e.g.* for Megakaryocyte_Tal1_R1 2143352 reads are mapped. Your answer might be slightly different if different references or tool versions are used.
>    > > 3. Some of these other reference sequences are parts of chromosomes but it is unclear where exactly, *e.g.* chr1_GL456210_random is a part of chromosome 1. There are entries like `chrUn` that are not associated with a chromosome but it is believed that they are part of the genome.
>    > {: .solution }
>    {: .question}
{: .hands_on}

# Assessing correlation between samples

To assess the similarity between the replicates sequencing datasets, it is a common technique to calculate the correlation of read counts for the different samples.

We expect that the replicate samples will cluster more closely to each other than to other samples. We will be use tools from the package deepTools for the next few steps. More information on deepTools can be found [in deepTools' documentation](https://deeptools.readthedocs.io/en/latest/content/list_of_tools.html).

> <hands-on-title>Assessing correlation between samples</hands-on-title>
>
> multiBamSummary splits the reference genome into bins of equal size and counts the number of reads in each bin from each sample. We set a small bin size here because we are working with a subset of reads that align to only a fraction of the genome.
>
> 1. {% tool [multiBamSummary](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_multi_bam_summary/deeptools_multi_bam_summary/3.3.2.0.0) %}: Run multiBamSummary to get read coverage of the alignments.
>     - *"Sample order matters"*: `No`
>     - {% icon param-files %} *"Bam files"*: Select all of the aligned BAM files
>     - *"Bin size in bp"*: 1000
>
> 2. {% tool [plotCorrelation](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_correlation/deeptools_plot_correlation/3.3.2.0.0) %}: Run plotCorrelation to visualize the results.
>     - *"Matrix file from the multiBamSummary tool"*: Select the multiBamSummary output file
>     - *"Correlation method"*: `Pearson`
>     - *"Plotting type"*: `Heatmap`
>     - *"Plot the correlation value"*: `Yes`
>     - *"Skip zeros"*: `Yes`
>     - *"Remove regions with very large counts"*: `Yes`
>
>     Feel free to play around with these parameter settings!
>
>     > <question-title></question-title>
>     >
>     > 1. Why do we want to skip zeros in **plotCorrelation**?
>     > 2. What happens if the Spearman's correlation method is used instead of the Pearson method?
>     > 3. What does the output of making a Scatterplot instead of a Heatmap look like?
>     >
>     > > <solution-title></solution-title>
>     > > 1. Large areas of zeros would lead to a correlation of these areas. The information we would get out of this computation would be meaningless.
>     > > 2. The clusters are different, *e.g.* Megakaryocyte_input_R2 and G1E_input_R2 are clustered together. [ More information about Pearson and Spearman's correlation. ](http://support.minitab.com/en-us/minitab-express/1/help-and-how-to/modeling-statistics/regression/supporting-topics/basics/a-comparison-of-the-pearson-and-spearman-correlation-methods/)
>     > > 3. Try making a Scatterplot to see for yourself!
>     > {: .solution }
>     {: .question}
>
>     ![heatmap](../../images/tal1/plotCorrelation_heatmap_pearson_1kb.png "Heatmap of correlation matrix generated by plotCorrelation.")
{: .hands_on}

Additional information on how to interpret plotCorrelation plots can be found [in deepTools' documentation](https://deeptools.readthedocs.io/en/latest/content/tools/plotCorrelation.html#background).

# Assessing IP strength

We will now evaluate the quality of the immunoprecipitation step in the ChIP-seq protocol.

> <hands-on-title>Assessing IP strength</hands-on-title>
>
> 1. {% tool [plotFingerprint](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_fingerprint/deeptools_plot_fingerprint/3.3.2.0.0) %}: Run plotFingerprint to assess ChIP signal strength.
>    - {% icon param-files %} *"Bam files"*: Select all of the aligned BAM files for the G1E cell type
>    - *"Show advanced options"*: `yes`
>    - *"Bin size in bases"*: `100`
>    - *"Skip zeros"*: `Yes`
>
> 2. Rerun plotFingerprint for the Megakaryocyte cell type.
>
> 3. View the output images.
>
>    ![fingerprintimage](../../images/tal1/plotFingerprint_graph_v2.png "Strength of immunoprecipitation of TAL1.")
>
>    > <question-title></question-title>
>    >
>    > 1. What does this graph in Figure 10 represent?
>    > 2. How do (or should) input datasets differ from IP datasets?
>    > 3. What do you think about the quality of the IP for this experiment?
>    > 4. How does the quality of the IP for megakaryocytes compare to G1E cells?
>    >
>    > > <solution-title></solution-title>
>    > > 1. It shows us how good the ChIP Signal compared to the control signal is. An ideal control (input) with perfect uniform distribution of reads along the genome (*i.e.* without enrichment in open chromatin etc.) and infinite sequencing coverage should generate a straight diagonal line. A very specific and strong ChIP enrichment will be indicated by a prominent and steep rise of the cumulative sum towards the highest rank.
>    > > 2. We expect that the control (input) signal is more or less uniform distributed over the genome (*e.g.* like the green line in the image above). The IP dataset should look more like the red line but it would be better if the values for IP start to increase at around 0.8 on the x-axis.
>    > > 3. The enrichment did not work as it should. Compare the blue line with the red one! For your future experiments: You can never have enough replicates!
>    > > 4. The quality of megakaryocytes is better then G1E.
>    > {: .solution }
>    {: .question}
{: .hands_on}

Additional information on how to interpret plotFingerprint plots can be found [in deepTools' documentation](https://deeptools.readthedocs.io/en/latest/content/tools/plotFingerprint.html#background).

# Determining TAL1 binding sites

Now that BWA has aligned the reads to the genome, we will use the tool MACS2 to identify regions of TAL1 occupancy, which are called "peaks". Peaks are determined from pileups of sequenced reads across the genome that correspond to where TAL1 binds.

MACS2 will perform two tasks:
1. Identify regions of TAL1 occupancy (peaks).
2. Generate bedGraph files for visual inspection of the data on a genome browser.

More information about MACS2 can be found in {% cite Zhang_2008 %}.

> <hands-on-title>Determining TAL1 binding sites</hands-on-title>
>
> 1. {% tool [MACS2 callpeak](toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.1.1.20160309.6) %}: Run MACS2 callpeak with the aligned read files from the previous step as Treatment (TAL1) and Control (input).
>    - *"Are you pooling Treatment Files?"*: `Yes`
>    - {% icon param-files %} *"ChIP-Seq Treatment File"*: Select all of the replicate ChIP-Seq treatment aligned BAM files for one cell type
>    - *"Do you have a Control File?"*: `Yes`
>    - *"Are you pooling Control Files?"*: `Yes`
>    - {% icon param-files %} *"ChIP-Seq Control File"*: Select replicate ChIP-Seq control aligned BAM files for the same cell type
>    - *"Format of Input Files"*: `Single-end BAM`
>    - *"Effective genome size"*: `M. musculus`
>    - *"Additional Outputs"*: Select `Peaks as tabular file (compatible wih MultiQC)`, `Peak summits`, `Scores in bedGraph files (--bdg)`
>
> 2. Rename files to reflect the origin and contents.
>
> 3. Repeat for the other cell type.
{: .hands_on}

# Inspection of peaks and aligned data

It is critical to visualize NGS data on a genome browser after alignment to evaluate the "goodness" of the analysis. Evaluation criteria will differ for various NGS experiment types, but for ChIP-seq data we want to ensure reads from a Treatment/IP sample are enriched at peaks and do not localize non-specifically (like the control/input condition).

MACS2 generates bedGraph and BED files that we will use to visualize read abundance and peaks, respectively, at regions MACS2 determines to be TAL1 peaks using Galaxy's in-house genome browser, Trackster.

## Inspection of peaks and aligned data with Trackster

We will import a gene annotation file so we can visualize aligned reads and TAL1 peaks relative to gene features and positions.

> <hands-on-title>Inspecting peaks and aligned data with Trackster</hands-on-title>
>
> 1. Import gene annotations file from Zenodo [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.197100.svg)](https://doi.org/10.5281/zenodo.197100)
>
> 2. Click "Visualize" on the page header and select "Create Visualization"
>
>    ![vizbutton](../../images/tal1/create-visualization.png "Trackster can be accessed from the Visualize button at the top of the screen.")
>
> 3. Set up Trackster
>    - Select Trackster
>    - *"Select a dataset to visualize"*: Select the imported gene annotation file (**Tip**: if this file doesn't appear as an option, go back to the history and edit the attribute Database/Build to be `mm10`)
>    - Click "Create Visualization"
>
> 4. Configure the visualization
>    - Select "View in new visualization"
>    - *"Browser name"*: Enter a name for your visualization
>    - *"Reference genome build (dbkey):"*: `mm10`
>    - Click "Create"
>
>    ![tracksterdb](../../images/tal1/configure-visualization.png "Assign session name and reference genome.")
>
> 5. Add tracks to Trackster
>    - Click the "Add tracks" (plus sign) button at the top right
>    - Using the search bar, search for and add the following tracks from MACS2 callpeak output to your view
>       - G1E Treatment bedGraph
>       - G1E Control bedGraph
>       - G1E narrow peaks
>       - Megakaryocytes Treatment bedGraph
>       - Megakaryocytes Control bedGraph
>       - Megakaryocytes narrow peaks
>    - Rename the tracks, if desired
>    - Play around with the track configurations, for example the color or the display mode
>
>    ![tracksterselect](../../images/tal1/add-tracks.png "Select data from the history to view in Trackster.")
>
> 6. Navigate to the *Runx1* locus (`chr16:92501466-92926074`) to inspect the aligned reads and TAL1 peaks.
>
>    > <question-title></question-title>
>    >
>    > 1. What do you see at the *Runx1* locus in Trackster?
>    >
>    > > <solution-title></solution-title>
>    > > 1. Directly upstream of the shorter *Runx1* gene models is a cluster of 3 TAL1 peaks that only appear in the G1E cell type, but not in Megakaryocytes. Further upstream, there are some shared TAL1 peaks in both cell types.
>    > {: .solution }
>    {: .question}
>
>    ![runx1](../../images/tal1/Trackster_Runx1_locus.png "The Runx1 locus with TAL1 peaks identified.")
{: .hands_on}

## Inspection of peaks and aligned data with IGV
We show here an alternative to Trackster, [IGV](http://software.broadinstitute.org/software/igv/).

> <hands-on-title>Inspecting peaks with IGV</hands-on-title>
>
> 1. Open IGV on your local computer.
> 2. Click on each narrow peaks result file from the MACS2 computations on "display with IGV" --> "local Mouse mm10"
> 3. For more information about IGV see [the IGV Tutorial]({% link topics/introduction/tutorials/igv-introduction/tutorial.md %})
{: .hands_on}

# Identifying unique and common TAL1 peaks between stages

We have processed ChIP-seq data from two stages of hematopoiesis and have lists of TAL1-occupied sites (peaks) in both cellular states. The next analysis step is to identify TAL1 peaks that are *shared* between the two cellular states and peaks that are *specific* to either cellular state.

> <hands-on-title>Identifying unique and common TAL1 peaks between states</hands-on-title>
>
> 1. {% tool [bedtools Intersect intervals](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.29.0) %}: Run bedtools Intersect intervals to find peaks that exist both in G1E and megakaryocytes.
>    - {% icon param-file %} *"File A to intersect with B"*: Select the TAL1 G1E narrow peaks BED file
>    - {% icon param-file %} *"File B to intersect with A"*: Select the TAL1 Megakaryocytes narrow peaks BED file
>    - Running this tool with the default settings will return overlapping peaks of both files.
>
> 2. {% tool [bedtools Intersect intervals](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.29.0) %}: Run bedtools Intersect intervals to find peaks that exist only in G1E.
>    - {% icon param-file %} *"File A to intersect with B"*: Select the TAL1 G1E narrow peaks BED file
>    - {% icon param-file %} *"File B to intersect with A"*: Select the TAL1 Megakaryocytes narrow peaks BED file
>    - *"Report only those alignments that \*\*do not\*\* overlap the BED file"*: `Yes`
>
> 3. {% tool [bedtools Intersect intervals](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.29.0) %}: Run bedtools Intersect intervals to find peaks that exist only in megakaryocytes.
>    - {% icon param-file %} *"File A to intersect with B"*: Select the TAL1 Megakaryocytes narrow peaks BED file
>    - {% icon param-file %} *"File B to intersect with A"*: Select the TAL1 G1E narrow peaks BED file
>    - *"Report only those alignments that \*\*do not\*\* overlap the BED file"*: `Yes`
>
> 4. Rename files to reflect the origin and contents.
>
>    > <question-title></question-title>
>    >
>    > 1. How many TAL1 peaks are common to both G1E cells and megakaryocytes?
>    > 2. How many are unique to G1E cells?
>    > 3. How many are unique to megakaryocytes?
>    >
>    > > <solution-title></solution-title>
>    > > 1. 1 peak (answer may vary depending on references and tool versions used)
>    > > 2. 407 peaks (answer may vary depending on references and tool versions used)
>    > > 3. 139 peaks (answer may vary depending on references and tool versions used)
>    > {: .solution }
>    {: .question}
{: .hands_on}

# Generating Input normalized coverage files

We will generate Input normalized coverage (bigWig) files for the ChIP samples, using the bamCompare tool from deepTools2. bamCompare provides multiple options to compare the two files (*e.g.* log2ratio, subtraction). We will use log2 ratio of the ChIP samples over Input.

> <hands-on-title>Generating input-normalized bigwigs</hands-on-title>
>
> 1. {% tool [bamCompare](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_compare/deeptools_bam_compare/3.3.2.0.0) %}: Run bamCompare to get the log2 read ratios between treatment and control samples.
>    - *"First BAM/CRAM file (*e.g.* treated sample)"*: Select the Megakaryocyte TAL1 aligned BAM file for replicate 1 (R1)
>    - *"Second BAM/CRAM file (*e.g.* control sample)"*: Select the Megakaryocyte input aligned BAM file for replicate 1 (R1)
>    - *"How to compare the two files"*: `Compute log2 of the number of reads`
>
> 2. Repeat this step for all treatment and control samples:
>     - Megakaryocyte TAL1 aligned BAM R2 and Megakaryocyte input aligned BAM R2
>     - G1E TAL1 aligned BAM R1 and G1E input aligned BAM R1
>     - G1E TAL1 aligned BAM R2 and G1E input aligned BAM R2
>
> 3. Rename files to reflect the origin and contents.
{: .hands_on}

# Plot the signal on the peaks between samples

Plotting your region of interest will involve using two tools from the deepTools suite:
+ computeMatrix: Computes the signal on given regions, using the bigwig coverage files from different samples.
+ plotHeatmap: Plots heatMap of the signals using the computeMatrix output.

Optionally, you can use plotProfile to create a profile plot using to computeMatrix output.

> <hands-on-title>Calculating signal matrix on MACS2 output</hands-on-title>
>
> 1. {% tool [computeMatrix](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.3.2.0.0) %}: Run computeMatrix to prepare data for plotting a heatmap of TAL1 peaks.
>    - {% icon param-file %} *Select Regions* > *"Regions to plot"*: Select the MACS2 narrow peaks files for G1E cells (TAL1 over Input)
>    - {% icon param-file %} *"Score file"*: Select the bigWig files for the G1E cells (log2 ratios from bamCompare)
>    - *"computeMatrix has two main output options"*: `reference-point`
>    - *"The Reference point for plotting"*: `center of region`
>    - *"Distance upstream of the start site of the regions defined in the region file"*: `5000`
>    - *"Distance downstream of the end site of the given regions"*: `5000`
>    - *"Show advanced options"*: `Yes`
>    - *"Convert missing values ot zero"*: `Yes`
>    - *"Skip zeros"*: `Yes`
>
> 2. Repeat for Megakaryoctes.
{: .hands_on}

> <hands-on-title>Plotting a heatmap of TAL1 peaks</hands-on-title>
>
> 1. {% tool [plotHeatmap](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_heatmap/deeptools_plot_heatmap/3.3.2.0.1) %}: Run plotHeatmap to create a heatmap for score distributions across TAL1 peak genomic regions in each cell type.
>    - *"Matrix file from the computeMatrix tool"*: Select the computeMatrix output for G1E cells
>    - *"Show advanced options"*: `Yes`
>    - *"Labels for the samples (each bigwig) plotted"*: Enter sample labels in the order you added them in computeMatrix, separated by spaces.
>
> 2. Repeat for Megakaryocytes.
>
> The outputs should look similar to this:
>
> ![hm](../../images/tal1/TAL1-plotheatmap.png)
{: .hands_on}

# Additional optional analyses

## Assessing GC bias

A common problem of PCR-based protocols is the observation that GC-rich regions tend to be amplified more readily than GC-poor regions.

We will now check whether the samples have more reads from regions of the genome with high GC.

> <hands-on-title>Assessing GC bias</hands-on-title>
>
> 1. {% tool [computeGCbias](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_gc_bias/deeptools_compute_gc_bias/3.3.2.0.0) %}: Run computeGCbias to determine the GC bias of the sequenced reads.
>    - {% icon param-file %} *"Bam file"*: Select an aligned BAM file
>    - *"Reference genome"*: `locally cached`
>    - *"Using reference genome"*: `mm10`
>    - *"Effective genome size"*: `user specified`
>    - *"Effective genome size"*: `10000000`
>    - *"Fragment length used for the sequencing"*: `50`
>
>    > <question-title></question-title>
>    >
>    > 1. Why would we worry more about checking for GC bias in an input file?
>    > 2. Does this dataset have a GC bias?
>    >
>    > > <solution-title></solution-title>
>    > > 1. In an input ChIP-seq file, the expectation is that DNA fragments are uniformly sampled from the genome. This is in contrast to an IP ChIP-seq file where it is expected that certain genomic regions contain more reads (*i.e.* regions that are bound by the protein that is immunopurified). Therefore, non-uniformity of reads in the input sample could be a result of GC-bias, whereby more GC-rich fragments are preferentially amplified during PCR.
>    > > 2. To answer this question, run the computeGCbias tool as described above and check out the results. What do YOU think? For more examples and information on how to interpret the results, check out the tool usage documentation [in deepTools' documentation](https://deeptools.readthedocs.io/en/latest/content/tools/computeGCBias.html#background).
>    > {: .solution }
>    {: .question}
>
> 2. {% tool [correctGCbias](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_correct_gc_bias/deeptools_correct_gc_bias/3.3.2.0.0) %}: Run correctGCbias to generate GC-corrected BAM/CRAM files.
>
>    > <question-title></question-title>
>    >
>    > 1. What does the tool correctGCbias do?
>    > 2. What is the output of this tool?
>    > 3. What are some caveats to be aware of if using the output of this tool in downstream analyses?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The correctGCbias tool removes reads from regions with higher coverage than expected (typically corresponding to GC-rich regions) and adds reads to regions with lower coverage than expected (typically corresponding to AT-rich regions).
>    > > 2. The output of this tool is a GC-corrected file in BAM, bigWig, or bedGraph format.
>    > > 3. The GC-corrected output file likely contains duplicated reads in low-coverage regions where reads were added to match the expected read density. Therefore, it is necessary to avoid filtering or removing duplicate reads in any downstream analyses.
>    > {: .solution }
>    {: .question}
{: .hands_on}

Additional information on how to interpret computeGCbias plots can be found [in deepTools' documentation](https://deeptools.readthedocs.io/en/latest/content/tools/computeGCBias.html).


# Conclusion


In this exercise you imported raw Illumina sequencing data, evaluated the quality before and after you trimmed reads with low confidence scores, aligned the trimmed reads, identified TAL1 peaks relative to the negative control (background), and visualized the aligned reads and TAL1 peaks relative to gene structures and positions. Additional, you assessed the "goodness" of the experiments by looking at metrics such as GC bias and IP enrichment.


topic: epigenetics
tutorial: formation_of_super-structures_on_xi
title: Formation of the Super-Structures on the Inactive X
difficulty: intermediate
time_estimation: 3h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 327dd3632d27026d0637ff29a5132b7e
last_modified: 2026-02-26T21:12:15.677268
zenodo_link: https://zenodo.org/record/1324070
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/epigenetics/tutorials/formation_of_super-structures_on_xi/tutorial.html

content:
Within a cell nucleus, the DNA is tightly-packed and the chromatin is spatially distributed with different levels and scales of organizations.

At the smallest scale, DNA is packaged into units called nucleosomes, made of eight histone proteins.

On a larger scale than nucleosomes, DNA is forming loops. DNA elements that would be otherwise separated by large distances can interact. The corresponding self-interacting (or self-associating) domains are found in many organisms: they are called Topologically Associating Domains (TADs) in mammalian cells. Mammalian chromosomes are also partitioned into two spatial compartments, labeled "A" and "B", where regions in compartment A tend to interact preferentially with A compartment-associated regions than B compartment-associated ones. Similarly, regions in compartment B tend to associate with other B compartment-associated regions.

> <comment-title>Tip: Learn more about chromosome conformation</comment-title>
> To learn more about chromosome conformation and TADs, you can follow our [HiC tutorial]({{site.baseurl}}/topics/epigenetics/tutorials/hicexplorer/tutorial.html)
{: .comment}

In mammals, the X chromosome inactivation (XCI) balances the dosage of X-linked genes between females and males. The genes on the inactive X (Xi) chromosome are not expressed.

Binding certain molecules to histone proteins can modify the chromatin structure and may result in changes in transcription level. For example, the H3K4me3 results from adding 3 methyl-group to the 4th lysine in the histone 3 protein. This modification is known to activate the transcription on nearby genes by opening the chromatin. The H3K27me3 on the other hand is inactivating the transcription of the nearby genes:

![Fadloun et al, 2013](../../images/formation_of_super-structures_on_xi/histone_modifications.jpg "Source: Fadloun et al, 2013")

More H3K27me3 and less H3K4me3 on the Xi could explain a lower expression of the genes there.

It has been also observed that the Xi reconfigures uniquely into a specific chromosomal conformation. cohesins, condensins and CCCTC-binding factor (CTCF) play key roles in chromosomal architectures and TAD formation which are other potential cause of the repression of the expression of the genes on Xi.

The structural-maintenance-of-chromosomes hinge domain containing 1 (SMCHD1) has been found enriched on the Xi. It may be the potential actor in the shape of Xi and the change in gene expression there.

{% cite Wang2018 %} investigates the mechanism by which the SMCHD1 gene shapes the Xi and represses the expression of the genes on Xi in mouse.

Their idea was to identify the differences which could be observed between the Xi and activated X chromosome, on both wild-type and SMCHD1 gene knockdown samples to study the SMCHD1 effect.
In different experiments, they targeted histones with H3K27me3 or H3K4me3 and CTCF using ChIP-seq experiments.

![ChIP-seq workflow](../../images/formation_of_super-structures_on_xi/Chromatin_immunoprecipitation_sequencing.jpg "Source: http://e.biohackers.net/ChIP-seq")

They obtained sequences corresponding to a portion of DNA linked to histones with H3K27me3, H3K4me3 or CTCF. Using this information, they could identify if there are differences in the H3K27me3, H3K4me3 and CTCF between the X (active or inactive) chromosomes and the potentially influenced genes.

In the upcoming tutorial, we will only use the wild type data from {% cite Wang2018 %} and analyze the ChIP-seq data step by step:

- CTCF with 2 replicates: `wt_CTCF_rep1` and `wt_CTCF_rep2`
- H3K4me3 with 2 replicates: `wt_H3K4me3_rep1` and `wt_H3K4me3_rep2`
- H3K27me3 with 2 replicates: `wt_H3K27me3_rep1` and `wt_H3K27me3_rep2`
- 'input' with 2 replicates: `wt_input_rep1` and `wt_input_rep2`

    In 'input' samples, the same treatment as the ChIP-seq samples was done except for the immunoprecipitation step.
    They are also used along with the 'ChIP-seq' samples to identify the potential sequencing bias and help for differential analysis.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Step 1: Quality control and treatment of the sequences

The first step of any ChIP-Seq data analysis is quality control of the raw sequencing data.

To save time, we will do it only on the data of one sample `wt_H3K4me3_rep1` which has been already down-sampled. keep in mind that with real data this should be done on each and every sample.

> <hands-on-title>Import the data</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import `wt_H3K4me3_read1.fastq.gz` and `wt_H3K4me3_read2.fastq.gz` from [Zenodo](https://zenodo.org/record/1324070) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/1324070/files/wt_H3K4me3_read1.fastq.gz
>    https://zenodo.org/record/1324070/files/wt_H3K4me3_read2.fastq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Add a tag called `#wt_H3K4me3_read1` to the read1 file and a tag called `#wt_H3K4me3_read2` to the read2 file.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
> 4. Inspect the first file by clicking on the {% icon galaxy-eye %} (eye) icon (**View data**)
>
> 5. Create a paired collection named `Paired Reads`
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
{: .hands_on}

{% include topics/sequence-analysis/tutorials/quality-control/fastq_question.md %}

During sequencing, errors are introduced, such as incorrect nucleotides being called. These are due to the technical limitations of each sequencing platform. Sequencing errors might bias the analysis and can lead to a misinterpretation of the data.

Sequence quality control is therefore an essential first step in your analysis. We use here similar tools as described in ["Quality control" tutorial]({{site.baseurl}}/topics/sequence-analysis): [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) and [Trim Galore](https://www.bioinformatics.babraham.ac.uk/projects/trim_galore/).

> <hands-on-title>Quality control</hands-on-title>
> 
> 1. Run {% tool [Flatten collection](__FLATTEN__) %} with the following parameters:
>    - *"Input collection"*: `Paired Reads`
>   
> 2. Rename the flatten collection: `Flat Collection`
>   
> 3. Run {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy1) %} with the following parameters:
>    - {% icon param-collection %} *"Raw read data from your current history"*: `Flat Collection` (Flattened paired end read dataset collection)
>
> 4. Inspect the generated HTML files
>
>    > <question-title></question-title>
>    >
>    > 1. How is the quality of the reads in `wt_H3K4me3_read1`?
>    > 2. And in `wt_H3K4me3_read2`?
>    > 3. What should we do if the quality of the reads is not good?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The reads in `wt_H3K4me3_read1` are of good quality:
>    > >     - There is 50,000 sequences, all of 51 bp
>    > >     - The "Per base sequence quality" is not decreasing too much at the end of the sequences
>    > >
>    > >        ![Per base sequence quality for read1](../../images/formation_of_super-structures_on_xi/read1_per_base_sequence_quality.png "Per base sequence quality")
>    > >
>    > >     - The mean quality score over the reads is quite high
>    > >
>    > >        ![Per sequence quality scores for read1](../../images/formation_of_super-structures_on_xi/read1_per_sequence_quality_scores.png "Per sequence quality scores")
>    > >
>    > >     - Homogeneous percentage of the bases
>    > >
>    > >        ![Per base sequence content for read1](../../images/formation_of_super-structures_on_xi/read1_per_base_sequence_content.png "Per base sequence content")
>    > >
>    > >     - No N in the reads
>    > >
>    > >        ![Per base N content for read1](../../images/formation_of_super-structures_on_xi/read1_per_base_n_content.png "Per base N content")
>    > >
>    > >     - No duplicated sequences
>    > >
>    > >        ![Sequence Duplication Levels for read1](../../images/formation_of_super-structures_on_xi/read1_sequence_duplication_levels.png "Sequence Duplication Levels")
>    > >
>    > >     - No more known adapters
>    > >
>    > >        ![Adapter Content for read1](../../images/formation_of_super-structures_on_xi/read1_adapter_content.png "Adapter Content")
>    > >
>    > > 2. The reads in `wt_H3K4me3_reverse` are a bit worse:
>    > >
>    > >     - The "Per base sequence quality" is decreasing more at the end of the sequences, but it stays correct
>    > >
>    > >        ![Per base sequence quality for read2](../../images/formation_of_super-structures_on_xi/read1_per_base_sequence_quality.png "Per base sequence quality")
>    > >
>    > >     - The sequence `NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN` is over represented
>    > >
>    > >     Despite these small things, the overall quality of the reads is really good.
>    > >
>    > > 3. If the quality of the reads is not good, we should:
>    > >    1. Check what is wrong and think about it: it may come from the type of sequencing or what we sequenced (high quantity of overrepresented sequences in transcriptomics data, biaised percentage of bases in HiC data)
>    > >    2. Ask the sequencing facility about it
>    > >    3. Perform some quality treatment (in a reasonable way to not loose too much information) with some trimming or removal of bad reads
>    > >
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

It is often necessary to trim sequenced read, for example, to get rid of bases that were sequenced with high uncertainty (= low quality bases) at the read ends.

{% include topics/sequence-analysis/tutorials/quality-control/paired_end_question.md forward="wt_H3K4me3_read1" reverse="wt_H3K4me3_read2" %}

> <hands-on-title>Trimming low quality bases</hands-on-title>
>
> 1. Run {% tool [Trim Galore!](toolshed.g2.bx.psu.edu/repos/bgruening/trim_galore/trim_galore/0.6.7+galaxy0) %} with the following parameters:
>    - *"Is this library paired- or single-end?"*: `Paired Collection`
>       - {% icon param-collection %} *"Select a paired collection"*: `Paired Reads` (Input collection)
>
>       > <tip-title>Not selectable files?</tip-title>
>       >
>       > If your FASTQ files cannot be selected, you might check whether their format is FASTQ with Sanger-scaled quality values (`fastqsanger`). You can edit the data type by clicking on the `pencil` symbol.
>       {: .tip}
>
>    - *"Trim Galore! advanced settings"*: `Full parameter list`
>       - *"Trim low-quality ends from reads in addition to adapter removal"*: `15`
>       - *"Overlap with adapter sequence required to trim a sequence"*: `3`
>       - *"Generate a report file"*: `Yes`
>
> 2. Rename the Trim Galore! output `reads` collection: `Trimmed Reads`
>
> 3. Inspect the generated txt file (`report file`)
>
>    > <question-title></question-title>
>    >
>    > 1. How many basepairs has been removed from the forwards reads because of bad quality?
>    > 2. And from the reverse reads?
>    > 3. How many sequence pairs have been removed because at least one read was shorter than the length cutoff?
>    >
>    > > <solution-title></solution-title>
>    > > 1. 32,198 bp (1.3%) (first `Quality-trimmed:`)
>    > > 2. 116,414 bp (4.6%) (second `Quality-trimmed:`). It is not a surprise: we saw that the quality was dropping more at the end of the sequences for the reverse reads thant for the forward reads.
>    > > 3. 1569 (3.14%) sequences (last line of the file)
>    > {: .solution }
>    {: .question}
{: .hands_on}

# Step 2: Mapping of the reads

With ChiP sequencing, we obtain sequences corresponding to a portion of DNA linked to the histone mark of interest, H3K4me3 in this case. As H3K4me3 opens the chromatime, nearby genes are going to be more transcribed. It would be interesting to know if there is a difference in the quantity of DNA impacted by H3K4me3 and the impacted genes between active and inactive X chromosome.

{% include topics/sequence-analysis/tutorials/mapping/mapping_explanation.md
    to_identify="binding sites"
    mapper="Bowtie2"
    mapper_link="http://bowtie-bio.sourceforge.net/bowtie2/index.shtml"
    answer_3="Wang et al. (2018) did ChIP-seq on mouse. So we should use the mouse reference genome. We will use mm10 (the latest build)"
%}

## Running Bowtie2

> <hands-on-title>Mapping</hands-on-title>
>
> 1. {% tool [Bowtie2](toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.4+galaxy0) %} with the following parameters:
>    - *"Is this single or paired library"*: `Paired-end`
>        - {% icon param-collection %} *"FASTQ Paired Dataset"*: `Trimmed Reads`
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>        - *"Select reference genome"*: `Mouse (Mus musculus): mm10`
>    - *"Save the bowtie2 mapping statistics to the history"*: `Yes`
>
> 2. Inspect the mapping stats
>
>    > <question-title></question-title>
>    >
>    > How many reads where mapped? Uniquely or several times?
>    >
>    > > <solution-title></solution-title>
>    > > The overall alignment rate is 98.57%. This score is quite high. If you have less than 70-80%, you should investigate the cause: contamination, etc.
>    > >
>    > > 41514 (85.72%) reads have been aligned concordantly exactly 1 time and 5190 (10.72%) aligned concordantly >1 times. The latter ones correspond to multiple mapped reads. Allowing for multiple  mapped reads increases the number of usable reads and the sensitivity of peak detection;
>    > > however, the number of false positives may also increase.
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

The output of Bowtie2 is a BAM file.

## Inspection of a BAM file

{% include topics/sequence-analysis/tutorials/mapping/bam_explanation.md mapper="Bowtie2" %}

## Visualization using a Genome Browser

{% include topics/sequence-analysis/tutorials/mapping/igv.md tool="Bowtie2" region_to_zoom="chr2:91,053,413-91,055,345" %}

### Visualization using JBrowse2

Besides running IGV locally on our computer, we can use Jbrowse2 to look at the genomes and visualize different tracks such as Chip-Seq data using the Galaxy. To do so:

> <hands-on-title>Genome Visualization via Jbrowse2</hands-on-title>
>
> 1. {% tool [BAM BED GFF coverage bigWigs](toolshed.g2.bx.psu.edu/repos/iuc/bbgbigwig/bbgtobigwig/0.1) %} with the following parameters:
>    - *"bam/bed/gff to convert"*: `Output of the bowtie2`
>    - *"Source Genome Build"*: `Mouse (Mus Musculus): mm10 Full`
> 2. {% tool [JBrowse2](toolshed.g2.bx.psu.edu/repos/fubar/jbrowse2/jbrowse2/2.17.0+galaxy0) %} with the following parameters:
>    - *"Select a built in reference genome or custom genome"*: `Mouse (Mus Musculus): mm10 Full`
>    - *"Insert Track Group"*
>        - *"Insert Annotation Track"*
>        - *"Track Type"*: `BigWig Track`
>        - *"BigWig Track Data"*: `The output from step 1 (The BigWig File made from the BAM file)`
>
{: .hands_on}

This takes some time to finish. Again, you can explore the genome. For example, you can look at the following coordinate in the genome: `chr2:91,053,413-91,055,345`. For a comprehensive tutorial on Jbrowse please look at this tutorial: https://training.galaxyproject.org/training-material/topics/visualisation/tutorials/jbrowse/tutorial.html.

# Step 3: ChIP-seq Quality Control

We already checked the quality of the raw sequencing reads in the first step.
Now we would like to test the quality of the ChIP-seq preparation, to know if our ChIP-seq samples are more enriched than the control (input) samples.

## Correlation between samples

To assess the similarity between the replicates of the ChIP and the input, respectively, it is a common technique to calculate the correlation of read counts on different regions for all different samples.
We expect that the replicates of the ChIP-seq experiments should be clustered more closely to each other than the replicates of the input sample. That is, because the input samples should not have enriched regions included - remember the immuno-precipitation step was skiped during the sample preparation.

To compute the correlation between the samples we are going to to use the QC modules of [deepTools](http://deeptools.readthedocs.io/), a software package for the QC, processing and analysis of NGS data. Before computing the correlation a time consuming step is required, which is to compute the read coverage (number of unique reads mapped at a given nucleotide) over a large number of regions from each of the inputed BAM files. For this we will use the tool **multiBamSummary** {% icon tool %}. Then, we use **plotCorrelation** {% icon tool %} from deepTools to compute and visualize the sample correlation. This is a fast process that allows to quickly try different color combinations and outputs.

Since in this tutorial we are interested in assessing H3K4me3, H3K27me3 and CTCF ChIP samples, the previous steps (quality control and mapping) needs to be run on all the replicates of ChIP samples as well as the input samples. To save time, we have already done that and you can now work directly on the BAM files of the provided 8 samples. For simplicity, the files include only the ChrX.

> <hands-on-title>Correlation between samples</hands-on-title>
>
> 1. Create a new history
> 2. Import the 8 BAM files from [Zenodo](https://zenodo.org/record/1324070) or from the data library into the history
>
>    ```
>    https://zenodo.org/record/1324070/files/wt_CTCF_rep1.bam
>    https://zenodo.org/record/1324070/files/wt_CTCF_rep2.bam
>    https://zenodo.org/record/1324070/files/wt_H3K4me3_rep1.bam
>    https://zenodo.org/record/1324070/files/wt_H3K4me3_rep2.bam
>    https://zenodo.org/record/1324070/files/wt_H3K27me3_rep1.bam
>    https://zenodo.org/record/1324070/files/wt_H3K27me3_rep2.bam
>    https://zenodo.org/record/1324070/files/wt_input_rep1.bam
>    https://zenodo.org/record/1324070/files/wt_input_rep2.bam
>    ```
>
> 3. Rename the files
> 3. {% tool [multiBamSummary](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_multi_bam_summary/deeptools_multi_bam_summary/3.5.1.0.0) %} with the following parameters:
>    - *"Sample order matters"*: `No`
>       - {% icon param-files %} *"BAM/CRAM file"*: the 8 imported BAM files
>    - *"Choose computation mode"*: `Bins`
>       - *"Bin size in bp"*: `1000`
>
>           This corresponds to the length of the fragments that were sequenced; it is not the read length!
>
>       - *"Distance between bins"*: `500`
>
>           It reduces the computation time for the tutorial
>
>    - *"Region of the genome to limit the operation to"*: `chrX`
>
>    Using these parameters, the tool will take bins of 1000 bp separated by 500 bp on the chromosome X. For each bin the overlapping reads in each sample will be computed and stored into a matrix.
>
> 4. {% tool [plotCorrelation](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_correlation/deeptools_plot_correlation/3.5.1.0.0) %} with the following parameters:
>    - {% icon param-files %} *"Matrix file from the multiBamSummary tool"*: `correlation matrix`(output of **multiBamSummary** {% icon tool %})
>    - *"Correlation method"*: `Pearson`
>
>    Feel free to try different parameters for the configuration of the plot (colors, title, etc)
>
{: .hands_on}

> <question-title></question-title>
>
> ![Output for plotCorrelation with the correlation scores between the 8 samples](../../images/formation_of_super-structures_on_xi/plotCorrelation.png "Correlation scores between the 8 samples")
>
> How are your samples clustered? Does that correspond to your expectations?
>
> > <solution-title></solution-title>
> > As one could expect, the input replicates cluster together and the ChIP replicates cluster together. It confirms that the immuno-precipitation step worked on our ChIP replicates.
> >
> > Moreover, for each sample, there is a high correlation between the two replicates which confirms the validity of the experiments.
> >
> {: .solution }
{: .question}

## IP strength estimation

To evaluate the quality of the immuno-precipitation(IP) step, we can compute the IP strength. It determines how well the signal in the ChIP-seq sample can be differentiated from the background distribution of reads in the control sample ('input'). After all, around 90% of all DNA fragments in a ChIP experiment will represent the genomic background.

To do that we take the data from the `rep1` of the `wt_H3K4me3` ChIP-seq sample and compare it with its corresponding input sample, using **plotFingerprint** {% icon tool %} of deepTools.

Similar to **multiBamSummary** {% icon tool %}, **plotFingerprint** {% icon tool %} randomly samples genome regions of a specified length (bins) and sums the per-base coverage in the indexed BAM files that overlap with those regions. These coverage values are then sorted according to their rank and the cumulative sum of read counts is plotted.

> <hands-on-title>IP strength estimation</hands-on-title>
>
> 1. {% tool [plotFingerprint](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_fingerprint/deeptools_plot_fingerprint/3.5.1.0.0) %} with the following parameters:
>    - *"Sample order matters"*: `No`
>       - {% icon param-files %} *"BAM/CRAM file"*: `wt_input_rep1` and `wt_H3K4me3_rep1`
>    - *"Region of the genome to limit the operation to"*: `chrX`
>    - *"Show advanced options"*: `Yes`
>       - *"Number of samples"*: `10000`
{: .hands_on}

The plotFingerprint tool generates a fingerprint plot. You need to intepret it to know the IP strength.

An ideal 'input' with perfect uniform distribution of reads along the genome (i.e. without enrichments in open chromatin) and infinite sequencing coverage should generate a straight diagonal line. A very specific and strong ChIP enrichment will be indicated by a prominent and steep rise of the cumulative sum towards the highest rank. This means that a big chunk of reads from the ChIP sample is located in few bins which corresponds to high, narrow enrichments typically seen for transcription factors.

![A guide to interpret a fingerprint plot](../../images/estrogen-receptor-binding-site-identification/QC_fingerprint.png "How to interpret a fingerprint plot? Image extracted from the deepTools documentation")

> <question-title></question-title>
>
> ![Output for plotFingerprint with the fingerprint plot to estimate the IP strength](../../images/formation_of_super-structures_on_xi/plotFingerPrint.png "Fingerprint plot for the first replicates to estimate the IP strength")
>
> What do you think about the quality of the IP for this experiment?
>
> > <solution-title></solution-title>
> > There is clear distinction between H3K4me3 and the input.
> >
> > A small percentage of the genome contain a very large fraction of the reads (>70%, point of change in the blue curve)
> >
> > The curves start to rise around 0.25. It means that almost 25% of the chromosome X is not sequenced at all.
> {: .solution }
{: .question}

> <hands-on-title>(Optional) IP strength estimation (other samples)</hands-on-title>
>
> Run the same analysis on the other ChIP-seq data along with their corresponding input and compare the output
{: .hands_on}

# Step 4: Normalization

One of the goals in ChIP-seq data analysis is finding regions on the genome which are enriched for the ChIP data of interest (regions with significantly higher read coverage for the ChIP data comparing to its corresponding input). In the following exercise we would like to know where the H3K4me3 sites are. For this we need to extract which parts of the genome have been enriched (i.e. more reads mapped) within the samples that underwent immunoprecipitation. However, to reach a reliable comparison the data needs to be normalized to remove any technical bias. For the normalization we have two steps:

1. Normalization by sequencing depth
2. Normalization by input file

To learn how to do the normalization, we will take the `wt_H3K4me3_rep1` sample as ChIP data and `wt_input_rep1` as input.

## Generation of coverage files normalized by sequencing depth

> <hands-on-title>Estimation of the sequencing depth</hands-on-title>
>
> 1. {% tool [Samtools idxstats](toolshed.g2.bx.psu.edu/repos/devteam/samtools_idxstats/samtools_idxstats/2.0.4) %} with the following parameters:
>    - {% icon param-files %} *"BAM file"*: `wt_H3K4me3_rep1.bam` and `wt_input_rep1.bam`
>
> > <question-title></question-title>
> >
> > 1. What is the output of **Samtools idxstats** {% icon tool %}?
> > 2. How many reads has been mapped on chrX for the input and for the ChIP-seq samples?
> > 3. Why are the number of reads different? And what could be the impact of this difference?
> >
> > > <solution-title></solution-title>
> > > 1. This tool generates a table with 4 columns:  reference sequence identifier, reference sequence length, number of mapped reads and number of placed but unmapped reads. Here it estimates how many reads mapped to which chromosome. Furthermore, it tells the chromosome lengths and naming convention (with or without 'chr' in the beginning)
> > > 2. 1,204,821 for ChIP-seq samples and 1,893,595 for the input
> > > 3. The number of reads can be different because of  different sequencing depth. It can bias the interpretation of the number of reads mapped to a specific genome region and the identification of the H3K4me3 sites. Specially here, as the number of reads for the input is higher than the ChIP data less regions could be identified having a significantly higher read coverage for the ChIP data comparing to the corresponding input.
> > {: .solution }
> {: .question}
{: .hands_on}

The different samples have usually a different sequencing depth, i.e. a different number of reads. These differences can bias the interpretation of the number of reads mapped to a specific genome region. We first need to make the samples comparable by normalizing the coverage by the sequencing depth.

We are using **bamCoverage** {% icon tool %}. Given a BAM file, this tool generates coverages by first calculating all the number of reads (either extended to match the fragment length or not) that overlap each bin in the genome and then normalizing with various options. It produces a coverage file where for each bin the number of overlapping reads (possibly normalized) is noted.

> <hands-on-title>Coverage file normalization</hands-on-title>
>
> 1. {% tool [bamCoverage](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_coverage/deeptools_bam_coverage/3.5.1.0.0) %} with the following parameters:
>    - {% icon param-files %} *"BAM file"*: `wt_H3K4me3_rep1.bam` and `wt_input_rep1.bam`
>    - *"Bin size in bases"*: `25`
>    - *"Scaling/Normalization method"*: `Normalize coverage to 1x`
>       - *"Effective genome size"*: `GRCm38/mm10 (2308125349)`
>    - *"Coverage file format"*: `bedgraph`
>    - *"Region of the genome to limit the operation to"*: `chrX`
>
>    > <question-title></question-title>
>    >
>    > 1. What is a `bedgraph` file?
>    > 2. Which regions have the highest coverage in ChIP data and in the input?
>    >
>    > > <solution-title></solution-title>
>    > > 1. It is a tabular file with 4 columns: chrom, chromStart, chromEnd and a data value (coverage)
>    > > 2. We can run **Sort** {% icon tool %} on the 4th column in descending order to get the regions with the highest (normalized) coverage. For `wt_H3K4me3_rep1`, the regions between 152,233,600	and 152,233,625	 are the most covered. For `wt_input_rep1`, between 143,483,000 and 143,483,100.
>    > {: .solution }
>    {: .question}
>
> 2. {% tool [bamCoverage](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_coverage/deeptools_bam_coverage/3.5.1.0.0) %} with the same parameters but
>    - *"Coverage file format"*: `bigWig`
>
>    > <question-title></question-title>
>    >
>    > What is a `bigWig` file?
>    >
>    > > <solution-title></solution-title>
>    > > A `bigWig` file is a compressed `bedgraph` file. Similar in relation as BAM to SAM, but this time just for coverage data. This means bigWig and bedgraph files are much smaller than BAM or SAM files (no sequence or quality information).
>    > {: .solution }
>    {: .question}
>
> 3. Use **IGV** {% icon tool %} to inspect both signal coverages (input and ChIP samples). Alternatively, you can use **JBrowse2** as mentioned above to load multiple tracks and look at genome intervals.
>
{: .hands_on}

> <question-title></question-title>
>
> If you zoom to `chrX:151,385,260-152,426,526`, what do you observe?
>
> ![Output of bamCoverage](../../images/formation_of_super-structures_on_xi/bamcoverage_igv.png "bamCoverage for wt_H3K4me3_rep1 and wt_input_rep1 on chrX:151,385,260-152,426,526")
>
> > <solution-title></solution-title>
> > The track with the coverage for the input (`wt_input_rep1`) seems quite homogeneous. On the other hand, for `wt_H3K4me3_rep1`, we can observe some clear peaks at the beginning of the genes.
> {: .solution }
{: .question}

## Generation of input-normalized coverage files

To extract only the information induced by the immunoprecipitation, we normalize the coverage file for the sample that underwent immunoprecipitation by the coverage file for the input sample. Here we use the tool **bamCompare** {% icon tool %} which compare 2 BAM files while caring for sequencing depth normalization.

> <hands-on-title>Generation of input-normalized coverage files</hands-on-title>
>
> 1. {% tool [bamCompare](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_compare/deeptools_bam_compare/3.5.1.0.0) %} with the following parameters:
>    - {% icon param-file %} *"First BAM file (e.g. treated sample)"*: `wt_H3K4me3_rep1.bam`
>    - {% icon param-file %} *"Second BAM file (e.g. control sample)"*: `wt_input_rep1.bam`
>    - *"Bin size in bases"*: `50`
>    - *"How to compare the two files"*: `Compute log2 of the number of reads ratio`
>    - *"Coverage file format"*: `bedgraph`
>    - *"Region of the genome to limit the operation to"*: `chrX`
>
>    > <question-title></question-title>
>    >
>    > 1. What does a positive value or a negative value mean in the 4th column?
>    > 2. Which regions have the highest coverage in the ChIP data? and the lowest?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The 4th column contains the log2 of the number of reads ratio between the ChIP-seq sample and the input sample. A positive value means that the coverage on this portion of genome is higher in the ChIP-seq sample than in the input sample
>    > > 2. The highest: 152,233,800-152,233,850 (consistent with the most covered regions in `wt_H3K4me3_rep1` given by **bamCoverage** {% icon tool %}). The lowest: 169,916,600-169,916,650
>    > {: .solution }
>    {: .question}
>
> 2. {% tool [bamCompare](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_compare/deeptools_bam_compare/3.5.1.0.0) %} with the same parameters but:
>    - *"Coverage file format"*: `bigWig`
>
> 3. Use **IGV** {% icon tool %} to inspect the log2 ratio. Or, as an alternative, you can use **JBrowse2** from the tools on Galaxy.
>
{: .hands_on}

> <question-title></question-title>
>
> How could you interpret the new track if you zoom to `chrX:151,385,260-152,426,526`?
>
> ![Output of bamCoverage](../../images/formation_of_super-structures_on_xi/bamcompare_igv.png "bamCoverage for wt_H3K4me3_rep1 and wt_input_rep1 on chrX:151,385,260-152,426,526")
>
> > <solution-title></solution-title>
> > The new track is the difference between the first track (`wt_H3K4me3_rep1`) and the second track (`wt_input_rep1`)
> {: .solution }
{: .question}

# Step 5: Detecting enriched regions (peak calling)

We could see in the ChIP data some enriched regions (peaks). We now would like to call these regions to obtain their coordinates, using **MACS2 callpeak** {% icon tool %}

> <hands-on-title>Peak calling</hands-on-title>
>
> 1. {% tool [MACS2 callpeak](toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.2.7.1+galaxy0) %} with the following parameters:
>    - *"Are you pooling Treatment Files?"*: `No`
>       - {% icon param-file %} *"ChIP-Seq Treatment File"*: `wt_H3K4me3_rep1.bam`
>    - *"Do you have a Control File?"*: `Yes`
>       - *"Are you pooling Treatment Files?"*: `No`
>           - {% icon param-file %} *"ChIP-Seq Treatment File"*: `wt_input_rep1.bam`
>    - *"Format of Input Files"*: `Paired-end BAM`
>    - *"Effective genome size"*: `M.musculus (1.87e9)`
>    - *"Outputs"*: `Summary page (html)`
>
>    > <comment-title></comment-title>
>    > The advanced options may be adjusted, depending of the samples.
>    > If your ChIP-seq experiment targets regions of broad enrichment, *e.g.* non-punctuate histone modifications, select calling of broad regions.
>    > If your sample has a low duplication rate (*e.g.* below 10%), you might keep all duplicate reads (tags). Otherwise, you might use the 'auto' option to estimate the maximal allowed number of duplicated reads per genomic location.
>    {: .comment}
>
> 2. Inspect the {% icon param-file %} `(narrow Peaks)` file (output of **MACS2 callpeak** {% icon tool %})
>
>    > <question-title></question-title>
>    >
>    > Which type of files were generated? What do they include?
>    >
>    > > <solution-title></solution-title>
>    > > **MACS2 callpeak** {% icon tool %} has generated a bed file with the coordinates of the identified peaks: chromosome, start, stop, name, integer score, strand, fold-change, -log10pvalue, -log10qvalue and relative summit position to peak start, as well as a html report which contains links to additional bed and xls files.
>    > {: .solution }
>    {: .question}
>
> 3. **IGV** {% icon tool %} (or **JBrowse2**) to inspect with the signal coverage and log2 ratio tracks
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many peaks have been identified in `chrX:151,385,260-152,426,526` based on IGV?
>
>    ![Output of MACS2](../../images/formation_of_super-structures_on_xi/macs2_igv.png "Peaks for wt_H3K4me3_rep1 and wt_input_rep1 on chrX:151,385,260-152,426,526")
>
> 2. What are the fold change of the peaks identified in `chrX:151,385,260-152,426,526`? Hint: using the BED file
> 3. How many peaks have been identified on the full chromosome X? How many peaks have a fold change > 50?
>
> > <solution-title></solution-title>
> > 1. We can see 11 peaks (track below the genes).
> > 2. Using {% tool [Filter](Filter1) %} with `c2>151385260 and c3<152426526`, we found that the 11 peaks with fold changes between 3.81927 and 162.06572
> > 4. On the 656 peaks on the full chromosome (number of lines of the original BED file) there are 252 peaks with FC>50 (using **Filter** {% icon tool %} with `c7>50`)
> {: .solution }
{: .question}

The called peak regions can be filtered by, *e.g.* fold change, FDR and region length for further downstream analysis.

# Step 6: Plot the signal between samples

So far, we have normalized the data and identified peaks. Now, we would like to visualize scores associated with certain genomic regions, for example ChIP enrichment values around the TSS of genes. Moreover, we would like to compare the enrichment of several ChIP samples (e.g. CTCF and H3K4me3) on the regions of interest.

Since we already generated the required files for the H3K4me3 sample, let's make them only for the CTCF sample:

> <hands-on-title>Prepare the peaks and data for CTCF</hands-on-title>
>
> 1. {% tool [bamCompare](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_compare/deeptools_bam_compare/3.5.1.0.0) %} with the following parameters:
>    - {% icon param-file %} *"First BAM file (e.g. treated sample)"*: `wt_CTCF_rep1.bam`
>    - {% icon param-file %} *"Second BAM file (e.g. control sample)"*: `wt_input_rep1.bam`
>    - *"Bin size in bases"*: `50`
>    - *"How to compare the two files"*: `Compute log2 of the number of reads ratio`
>    - *"Coverage file format"*: `bigwig`
>    - *"Region of the genome to limit the operation to"*: `chrX`
> 2. Rename the output of **bamCompare** {% icon tool %} with the name of the sample
> 3. {% tool [MACS2 callpeak](toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.2.7.1+galaxy0) %} with the following parameters
>    - *"Are you pooling Treatment Files?"*: `No`
>       - {% icon param-file %} *"ChIP-Seq Treatment File"*: `wt_CTCF_rep1.bam`
>    - *"Do you have a Control File?"*: `Yes`
>       - *"Are you pooling Treatment Files?"*: `No`
>           - {% icon param-file %} *"ChIP-Seq Treatment File"*: `wt_input_rep1.bam`
>    - *"Format of Input Files"*: `Paired-end BAM`
>    - *"Effective genome size"*: `M.musculus (1.87e9)`
>
{: .hands_on}

We can now concatenate the MACS2 outputs with the location of the peaks (concatenate the files and merge the overlapping regions) to obtain one BED file corresponding to the coordinates of the interesting regions to plot.

> <hands-on-title>Prepare the peak coordinates</hands-on-title>
>
> 1. {% tool [Concatenate datasets](cat1) %} with the following parameters:
>    - {% icon param-file %} *"Concatenate Dataset"*: output of **MACS2 callpeak** {% icon tool %} for `wt_CTCF_rep1`
>    - Click on *"Insert Dataset"*:
>       - For *"Dataset"*: select the output of **MACS2 callpeak** {% icon tool %} for `wt_H3K4me3_rep1`
> 2. {% tool [bedtools SortBED](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_sortbed/2.30.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Sort the following BED/bedGraph/GFF/VCF/EncodePeak file"*: output of **Concatenate** {% icon tool %}
> 3. {% tool [bedtools MergeBED](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_mergebed/2.30.0) %} with the following parameters:
>    - {% icon param-file %} *"Sort the following BAM/BED/bedGraph/GFF/VCF/EncodePeak file"*: output of **SortBED** {% icon tool %}
>
{: .hands_on}

To plot the the peaks score on the region generated above (MergeBED output) two tools from the [deepTools](http://deeptools.readthedocs.io/) package are used:
- **computeMatrix** {% icon tool %}: it computes the signal on given regions, using the `bigwig` coverage files from different samples.
- **plotHeatmap** {% icon tool %}: it plots heatMap of the signals using the **computeMatrix** {% icon tool %} output.

Optionally, we can also use **plotProfile** {% icon tool %} to create a profile plot from **computeMatrix** {% icon tool %} output.

> <hands-on-title>Plot the heatmap</hands-on-title>
>
> 1. {% tool [computeMatrix](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.5.1.0.0)%} with the following parameters:
>    - *"Select regions"*:
>       - {% icon param-file %} *"Regions to plot"*: output of **MergeBED** {% icon tool %}
>    - *"Sample order matters"*: `No`
>       - {% icon param-files %} *"Score file"*: the 2 `bigwig` files generated by **bamCompare** {% icon tool %} and renamed
>    - *"computeMatrix has two main output options"*: `reference-point`
>       - *"The reference point for the plotting"*: `center of region`
>       - *"Distance upstream of the start site of the regions defined in the region file"*: `3000`
>       - *"Distance downstream of the end site of the given regions"*: `3000`
> 2. {% tool [plotHeatmap](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_heatmap/deeptools_plot_heatmap/3.5.1.0.1) %} with the following parameters
>    - {% icon param-file %} *"Matrix file from the computeMatrix tool"*: `Matrix` (output of **computeMatrix** {% icon tool %})
>    - *"Show advanced options"*: `yes`
>       - *"Reference point label"*: select the right label
>       - *"Did you compute the matrix with more than one groups of regions?"*: `No, I used only one group`
>           - *"Clustering algorithm"*: `Kmeans clustering`
>           - *"Number of clusters to compute"*: `2`
{: .hands_on}

It should generate an heatmap similar to:

![Output of plotHeatmap for 2 samples](../../images/formation_of_super-structures_on_xi/peak_heatmap_1.png "Scores around the peaks for wt_H3K4me3_rep1 and wt_input_rep1")

When we look at this graph, it seems that less but larger peaks are found for `H3K4me3_rep1` and that only few peaks are shared.

> <question-title></question-title>
>
> 1. How many peaks have been found for `CTCF_rep1` and for `H3K4me3_rep1`?
> 2. What are the mean width of the peaks for `CTCF_rep1` and for `H3K4me3_rep1`?
> 3. How many peaks are specific to `CTCF_rep1` or `H3K4me3_rep1`?
>
> > <solution-title></solution-title>
> > 1. 656 peaks for `H3K4me3_rep1` and 2,688 for `CTCF_rep1` (number of lines in the **MACS2 callpeak** {% icon tool %} BED file)
> > 2. 1630.77 bp for `H3K4me3_rep1` and 404.55 for `CTCF_rep1` (**Compute** {% icon tool %} with `c3-c2` and then **Datamash** {% icon tool %} with `Mean` on `Column:11`)
> > 3. 443 peaks (over 656) are specific to `H3K4me3_rep1` and 2,464 (over 2,688) to `CTCF_rep1` (**Intersect intervals** {% icon tool %}). Around 220 peaks are then overlapping.
> {: .solution }
{: .question}

So far, we have only analyzed 2 samples, but we can do the same for all the 6 samples:

> <hands-on-title>(Optional) Plot the heatmap for all the samples</hands-on-title>
>
> 1. {% tool [bamCompare](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_compare/deeptools_bam_compare/3.5.1.0.0) %} for each combination input - ChIP data:
>     1. `wt_CTCF_rep1` - `wt_input_rep1` (already done)
>     2. `wt_H3K4me3_rep1` - `wt_input_rep1` (already done)
>     3. `wt_H3K27me3_rep1` - `wt_input_rep1`
>     4. `wt_CTCF_rep2` - `wt_input_rep2`
>     5. `wt_H3K4me3_rep2` - `wt_input_rep2`
>     6. `wt_H3K27me3_rep2` - `wt_input_rep2`
> 2. Rename the outputs of **bamCompare** {% icon tool %} with the name of the ChIP data
> 3. {% tool [MACS2 callpeak](toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.2.7.1+galaxy0) %} for each combination input - ChIP data
> 4. {% tool [Concatenate datasets](cat1) %} with the following parameters:
>    - {% icon param-file %} *"Concatenate Dataset"*: one output of **MACS2 callpeak** {% icon tool %}
>    - Click on *"Insert Dataset"*:
>       - In *"Select"*: one other output of **MACS2 callpeak** {% icon tool %}
>    - Redo for the 6 outputs of **MACS2 callpeak** {% icon tool %}
> 5. {% tool [bedtools SortBED](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_sortbed/2.30.0+galaxy2) %} with the following parameters
>    - {% icon param-file %} *"Sort the following bed,bedgraph,gff,vcf file"*: output of **Concatenate** {% icon tool %}
> 6. {% tool [bedtools MergeBED](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_mergebed/2.30.0) %} with the following parameters
>    - {% icon param-file %} *"Sort the following bed,bedgraph,gff,vcf file"*: output of **SortBED** {% icon tool %}
> 7. {% tool [computeMatrix](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.5.1.0.0)%} with the same parameters but:
>    - *"Select regions"*:
>       - {% icon param-file %} *"Regions to plot"*: output of **MergeBED** {% icon tool %}
>    - *"Sample order matters"*: `No`
>       - {% icon param-files %} *"Score file"*: the 6 `bigwig` files generated by **bamCompare** {% icon tool %} and renamed
>    - *"computeMatrix has two main output options"*: `reference-point`
>       - *"The reference point for the plotting"*: `center of region`
>       - *"Distance upstream of the start site of the regions defined in the region file"*: `3000`
>       - *"Distance downstream of the end site of the given regions"*: `3000`
> 8. {% tool [plotHeatmap](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_heatmap/deeptools_plot_heatmap/3.5.1.0.1) %} with the following parameters
>    - {% icon param-file %} *"Matrix file from the computeMatrix tool"*: `Matrix` (output of **computeMatrix** {% icon tool %})
>    - *"Show advanced options"*: `yes`
>       - *"Reference point label"*: select the right label
>       - *"Did you compute the matrix with more than one groups of regions?"*: `No, I used only one group`
>           - *"Clustering algorithm"*: `Kmeans clustering`
>           - *"Number of clusters to compute"*: `6`
{: .hands_on}

> <question-title></question-title>
>
> 1. How many peaks are found for the different samples?
> 2. How are the peaks?
>
>    ![Output of plotHeatmap for all samples](../../images/formation_of_super-structures_on_xi/peak_heatmap_all.png "Scores around the peaks for all samples")
>
> 3. How could be interpreted the peaks and read coverage in the `chrX:151,385,260-152,426,526` region?
>
>    ![Coverage and peaks for the replicate 1](../../images/formation_of_super-structures_on_xi/rep1_igv.png "Coverage and peaks for the replicate 1")
>
> > <solution-title></solution-title>
> > 1. Found peaks (number of lines in **MACS2 callpeak** {% icon tool %} outputs):
> >
> >     Target | Rep 1 | Rep 2
> >     --- | --- | ---
> >     CTCF | 2,672 | 2,061
> >     H3K4me3 | 657 | 718
> >     H3K27me3 | 220 | 73
> >
> >    The tendencies are similar for both replicates: more peaks for CTCF, less for H3K4me3 and only few for H3K27me3.
> >
> > 2. As observed with the 2 samples, the peaks for H3K4me3 are wider than for CTCF. We also observe that the peaks found with one replicate are found with the other replicate.
> > 3. The H3K4me3 sample has clear and large regions in which the read coverage are enriched. H3K4me3 is one of the least abundant histone modifications. It is highly enriched at active promoters near transcription start sites (TSS) and positively correlated with transcription.
> >
> >    For H3K27me3, the coverage is more homogeneous. A gene is a broad domain of H3K27me3 enrichment across its body of genes corresponds to a gene with a transcription inhibited by H3K27me3. We can also identified some "bivalent" genes: gene with a peak around the TSS for H3K27me3 (e.g. region_208 for gene Gpr173) but also H3K4me3. We also observe some H3K27me3-depleted regions sharply demarcated, with boundaries coinciding with gene borders (e.g. Kdm5c). This is a chromatin signature reminiscent of genes that escape XCI.
> >
> >    To reproduce, run **bamCoverage** {% icon tool %}, **IGV** {% icon tool %} and **MACS2 callpeak** {% icon tool %} outputs.
> {: .solution }
{: .question}

# Conclusion


Along this tutorial, we learn how to extract peaks and coverage information from raw data of ChIP experiments:

![Workflow to extract peaks and coverage information from raw data of ChIP experiments](../../images/formation_of_super-structures_on_xi/tutorial-scheme.png "Workflow to extract peaks and coverage information from raw data of ChIP experiments")

This information can be then related to the biology to answer the original question. We tried then to relate the observed differences of peak and read coverage between H3K27me3 and H3K4me3 to the known biology. We could go even further in the analysis to reproduce the results of the original paper (e.g. by looking at the bivalent genes, identifying the differences between Xa and Xi).


topic: epigenetics
tutorial: cut_and_run
title: CUT&RUN data analysis
difficulty: intermediate
time_estimation: 3h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 802607e67d97dc26ea8536870a837a89
last_modified: 2026-02-26T21:12:15.672379
zenodo_link: https://zenodo.org/record/6823059
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/epigenetics/tutorials/cut_and_run/tutorial.html

content:
In many organism, {TF} play an important tole in the regulation of the gene expression. In human, we have up to 2,800 proteins and more than 1,600 are
TF ([list of transcription factors](https://en.wikipedia.org/wiki/List_of_human_transcription_factors)), although the number might change over timer.
Investigating the role of TFs, such as [GATA1](https://en.wikipedia.org/wiki/GATA1), is a very important task to understand the regulatory mechanisms in the cell and thus ascertain
the source of a disease, such as [myelofibrosis](https://en.wikipedia.org/wiki/Primary_myelofibrosis) a type of blood cancer.

![CUT_and_RUN](../../images/cut_and_run/cut_and_run.png "Skene and Henikoff 2017 eLIFE")

{CUT&RUN} {% cite Skene2017 %} became the new and advanced method to analyse DNA-associated proteins. CUT&RUN uses an antibody just as [ChIP-Seq](https://en.wikipedia.org/wiki/ChIP_sequencing) to select the {POI}.
The big difference, CUT&RUN couples the usage of an antibody with the usage of a pA-MNase, which you can see in **Figure 1**. In the revised version of the protocol {% cite Meers2019 %},
the pA-MNase has been replaced by a {pA(G)-MNase} to expand this technology to mouse primary antibodies. The [enzyme](https://en.wikipedia.org/wiki/Micrococcal_nuclease) fused to the protein A/G is an
endo-exonuclease that cleaves and shortens DNA. The fusion will ensure that the MNase comes in close proximity with the POI and thus cleaves DNA *in-situ*, close to POI binding sites. CUT&RUN allows to
fragment the DNA in unfixed cells and thus allows to study protein-DNA interactions in a more natural state. The added pA(G)-MNase thus creates shorted fragments that lead to a higher resolution for the mapping
in comparison to your standard ChIP-Seq protocol. CUT&RUN follows four fundamental steps: (1) permeabilize the nuclei or cells coated on beads, (2) incubate with the selective antibody of the POI, (3)
add the pA(G)-MNase and activate the enzyme for a small period of time, (4) release CUT&RUN fragments and collect the DNA from the supernatant. Afterward, the DNA can be purified, PCR amplified and
prepared for sequencing. You can find the CUT&RUN protocol at [protocols.io](https://www.protocols.io/view/cut-amp-run-targeted-in-situ-genome-wide-profiling-14egnr4ql5dy/v3).

In this tutorial, we will use data from the study of {% cite Zhu2019 %}. The article introduces a CUT&RUN pipeline that we will **not** completely follow. It is important to note at this point that
a CUT&RUN data analysis is more similar to an ATAC-Seq experiment than a standard ChIP-Seq. We will analyze the two biological replicates from a CUT&RUN experiment for the aforementioned TF GATA1 in humans.
We downsampled the data to speed up the run times in this tutorial. Our results will be compared to identified binding sites of GATA1 of a ChIP-Seq experiment {% cite Canver2017 %}.

## When working with real data
The workflow for this training material can be found [at the European Galaxy instance](https://usegalaxy.eu/u/heylf/w/copy-of-cutandrunlong). When you use your data we suggest using [this workflow](https://usegalaxy.eu/u/heylf/w/cutandrunlong)
which includes additional steps for your data analysis. Both workflows do not support peak calling with controls as CUT&RUN has a low background.
It is often recommended to use a positive or negative control as a comparison. Spike-in controls can be done for CUT&RUN but need then additional steps in the provided workflows to consider them.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet faqs/galaxy/analysis_results_may_vary.md %}

# Step 1: Preprocessing

## Get Data

We first need to download the sequenced reads (FASTQs) as well as other annotation files. Then, to increase the number of reads that will map to the reference genome (here human genome version 38, GRCh38/hg38), we need to preprocess the reads.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo](https://doi.org/10.5281/zenodo.6823059) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/record/6823059/files/chip_seq_peaks.bed
>    https://zenodo.org/record/6823059/files/Rep1_R1.fastq
>    https://zenodo.org/record/6823059/files/Rep1_R2.fastq
>    https://zenodo.org/record/6823059/files/Rep2_R1.fastq
>    https://zenodo.org/record/6823059/files/Rep2_R2.fastq
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 4. Add a tag called `#rep1` to the Rep1 files and a tag called `#rep2` to the Rep2 files.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
> 5. Check that the datatype of the 4 FASTQ files is `fastqsanger` and the peak file (chip_seq_peaks.bed) is `bed`. If they are not then change the datatype as described below.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 6. Create a paired collection named `2 PE fastqs`, and rename your pairs with the sample name followed by the attributes: `Rep1` and `Rep2`.
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
> 7. Rename the bed file `GATA1 ChIP-Seq peaks`.
>
{: .hands_on}

> <comment-title>FASTQ format</comment-title>
> If you are not familiar with FASTQ format, see the [Quality Control tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %})
{: .comment}
>
> <comment-title>BED / encode narrowPeak format</comment-title>
> If you are not familiar with BED format or encode narrowPeak format, see the [BED Format](https://genome.ucsc.edu/FAQ/FAQformat.html)
{: .comment}

## Quality Control

We first have to check if our data contains adapter sequences that we have to remove. A typical CUT&RUN experiment has a read length of 30-80 nt. We can check the raw data quality with **Falco**.

> <hands-on-title>Quality Control</hands-on-title>
>
> 1. {% tool [Flatten collection](__FLATTEN__) %} with the following parameters convert the list of pairs into a simple list:
>     - *"Input Collection"*: `2 PE fastqs`
>
> 2. {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} with the following parameters:
>       - {% icon param-collection %} *"Raw read data from your current history"*: Choose the output of **Flatten collection** {% icon tool %} selected as **Dataset collection**.
> 3. Inspect the web page output of **Falco** {% icon tool %} for the `Rep1_forward` sample. Check what adapters are found at the end of the reads.
>
>    > <question-title></question-title>
>    >
>    > 1. How many reads are in the FASTQ?
>    > 2. Which sections have a warning or an error?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are 300,000 reads.
>    > > 2. The 4 steps below have warnings:
>    > >
>    > >    1. **Per base sequence content**
>    > >
>    > >       CUT&RUN has sometimes base biases like ChIP-Seq.
>    > >
>    > >    2. **Sequence Duplication Levels**
>    > >
>    > >       The read library quite often has PCR duplicates that are introduced
>    > >       simply by the PCR itself. We will remove these duplicates later on.
>    > >
>    > >    3. **Overrepresented sequences**
>    > >
>    > >       Our data contains TruSeq adapter, Illumina PCR Primer, and a read from the mitochondrial chromosome.
>    > >
>    > >    4. **Adapter Content**
>    > >
>    > >       Our data contains an adapter (Illumina Universal Adapter) that we still have to remove.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
>    As it is tedious to inspect all these reports individually we will combine them with {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3) %}.
>
> 4. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3) %} to aggregate the FastQC reports with the following parameters:
>    - In *"Results"*:
>        - *"Results"*
>            - *"Which tool was used generate logs?"*: `FastQC`
>                - In *"FastQC output"*:
>                    - {% icon param-repeat %} *"Insert FastQC output"*
>                        - {% icon param-collection %} *"FastQC output"*: `Falco on collection N: Raw data` (output of **Falco** {% icon tool %})
>
{: .hands_on}

> <comment-title>Falco Results</comment-title>
> This is what you should expect from the **Adapter Content** section of multiQC:
> ![MultiQC screenshot of the Adapter Content section](../../images/cut_and_run/fastqc_adapter_content_plot_pre.png "MultiQC screenshot on the Adapter Content section")
{: .comment}

The MultiQC report (of FastQC) pointed out that we have in our data some standard Illumina adapter sequences, which we will remove with Trim Galore!.

## Trimming Reads

**Trim Galore!** is a handy tool that can automatically detect and trim standard Illumina adapters.

> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [Trim Galore!](toolshed.g2.bx.psu.edu/repos/bgruening/trim_galore/trim_galore/0.6.7+galaxy1) %} with the following parameters:
>    - *"Is this library paired- or single-end?"*: `Paired Collection`
>        - *"Select a paired collection"*: `2 PE fastqs`
>    - In *"Adapter sequence to be trimmed"*: `Illumina universal`
>    - *Avanced settings*: `Full parameter list`
>    - In *"Trim low-quality ends from reads in addition to adapter removal (Enter phred quality score threshold)"*: `30`
>    - In *"Discard reads that became shorter than length N"*: `15`
>    - In *"Generate a report file"*:   `Yes`
>
> 2. Click on the {% icon galaxy-eye %} (eye) icon of the report for `Rep1` and look for the info.
{: .hands_on}

> <question-title></question-title>
>
> 1. What percentage of reads contain adapters?
> 2. What percentage of reads are still longer than 15bp after the trimming?
>
> > <solution-title></solution-title>
> >
> > 1. 163,656 (54.6%) for Rep 1 and 169,776 (56.6%) for Rep 2
> > 2. The last line indicates that 10477 (3.49%) of pairs have been removed from Rep1 and 15775 (5.26%) pairs from Rep2.
> >
> {: .solution}
>
{: .question}

# Step 2: Mapping

## Mapping Reads to Reference Genome

Next, we map the trimmed reads to the human reference genome. Here we will use **Bowtie2**. We will extend the maximum fragment length (distance between read pairs) from 500 to 1000 because
we know some valid read pairs are from this fragment length. We will use the `--very-sensitive` parameter to have more chances to get the best match even if it takes a bit longer to run.
We will run the **end-to-end** mode because we trimmed the adapters, so we expect the whole read to map, no clipping of ends is needed. Regarding the genome to choose.
The hg38 version of the human genome contains [alternate loci](https://www.ncbi.nlm.nih.gov/grc/help/definitions/#ALTERNATE). This means that some regions of the genome
are present both in the canonical chromosome and on its alternate loci. The reads that map to these regions would map twice. To be able to filter reads falling into
repetitive regions but keep reads falling into regions present in alternate loci, we will map on the Canonical version of hg38 (only the chromosome with numbers, chrX, chrY, and chrM).

> <hands-on-title>Mapping reads to reference genome</hands-on-title>
>
> 1. {% tool [Bowtie2](toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.3+galaxy1) %} with the following parameters:
>    - *"Is this single or paired library"*: `Paired-end Dataset Collection`
>        - *"FASTQ Paired Dataset*: select the output of **Trim Galore!** {% icon tool %} *"paired reads"*
>        - *"Do you want to set paired-end options?"*: `Yes`
>            - *"Set the maximum fragment length for valid paired-end alignments"*: `1000`
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>        - *"Select reference genome"*: `Human (Homo sapiens): hg38 Canonical`
>    - *"Set read groups information?"*: `Do not set`
>    - *"Select analysis mode"*: `1: Default setting only`
>        - *"Do you want to use presets?"*: `Very sensitive end-to-end (--very-sensitive)`
>    - *"Do you want to tweak SAM/BAM Options?"*: `No`
>    - *"Save the bowtie2 mapping statistics to the history"*: `Yes`
>
> 2. Click on the {% icon galaxy-eye %} (eye) icon of the mapping stats `Rep1`.
{: .hands_on}

> <comment-title>Bowtie2 Results</comment-title>
> You should get similar results to this from Bowtie2:
> ![Mapping statistics of bowtie2](../../images/cut_and_run/mapping_stats.png "Mapping statistics of bowtie2")
{: .comment}

> <question-title></question-title>
>
> What percentage of read pairs from Rep1 mapped concordantly?
>
> > <solution-title></solution-title>
> >
> > 36.47+62.39=98.86%
> >
> {: .solution}
>
{: .question}

# Step 3: Filtering Mapped Reads and second level Quality Control

## Plot Enrichment signal

Before we apply any filters on our mapped data, let us check the cumulative enrichment of signal using **plotFingerprint** {% icon tool %} from the **deepTools** suite.

> <hands-on-title>Assess Signal CUT&RUN Enrichment using plotFingerprint</hands-on-title>
>
> 1. {% tool [plotFingerprint](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_fingerprint/deeptools_plot_fingerprint/3.5.4+galaxy0) %} with the following parameters:
>    - *"Sample order matters"*: `No`
>    - {% icon param-collection %} *"BAM/CRAM file"*: Select the output of  **Bowtie2** {% icon tool %} *"alignments"*
>
>
> 2. Check the plot.
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Do you see a high or low enrichment?
> 2. Do you expect such enrichment signal?
>
> > <solution-title></solution-title>
> >
> > ![plotFingerprint](../../images/cut_and_run/fingerprint.png "Signal Enrichment")
> >
> > 1. Both the replicates have almost identical enrichment profile indicating reproducibility. The sharp peak at the end indicates that only a few genomic regions (putative binding sites) were covered with most of the reads which is a very strong enrichment.
> > 2. There are mainly two reasons to expect such a high enrichment:
> >     1. Unlike ChIP-seq, CUT&RUN has low background noise.
> >     2. Our protein of interest (GATA1) in this data set is a transcription factor (TF). TFs usually show high binding specificity.
> >
> {: .solution}
>
{: .question}

## Filter Uninformative Reads

We apply some filters to the reads after the mapping. CUT&RUN datasets can have many reads that map to the mitochondrial genome because it is nucleosome-free and thus very accessible
to the pA(G)-MNase. CUT&RUN inserts adapter more easily in open chromatin regions due to the pA(G)-MNase activity. The mitochondrial genome is uninteresting for CUT&RUN,
so we remove these reads. We also remove reads with low mapping quality and reads that are not properly paired.

> <hands-on-title>Filtering of uninformative reads</hands-on-title>
>
> 1. {% tool [Filter BAM datasets on a variety of attributes](toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.5.2+galaxy2) %} with the following parameters:
>    - {% icon param-collection %} *"BAM dataset(s) to filter"*: Select the output of  **Bowtie2** {% icon tool %} *"alignments"*
>    - In *"Condition"*:
>        - {% icon param-repeat %} *"Condition"*
>            - In *"Filter"*:
>                - *"Filter"*
>                    - *"Select BAM property to filter on"*: `Mapping quality`
>                        - *"Filter on read mapping quality (phred scale)"*: `>=30`
>                - {% icon param-repeat %} *"Insert Filter"*
>                    - *"Select BAM property to filter on"*: `Proper Pair`
>                        - *"Select properly paired reads"*: `Yes`
>                - {% icon param-repeat %} *"Insert Filter"*
>                    - *"Select BAM property to filter on"*: `Reference name of the read`
>                        - *"Filter on the reference name for the read"*: `!chrM`
>    - *"Would you like to set rules?"*: `False`
>
>
> 2. Click on the input and the output BAM files of the filtering step. Check the size of the files.
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Based on the file size, what proportion of alignments was removed (approximately)?
> 2. Which parameter should be modified if you are interested in repetitive regions?
>
> > <solution-title></solution-title>
> >
> > 1. The original BAM files are 14-15 MB, the filtered ones are 5.8 and 7.1 MB. Approximately half of the alignments were removed.
> >
> > 2. You should modify the mapQuality criteria and decrease the threshold.
> >
> {: .solution}
>
{: .question}

## Filter Duplicate Reads

Because of the PCR amplification, there might be read duplicates (different reads mapping to the same genomic region) from over amplification of some regions. We will remove them with **Picard MarkDuplicates**.

> <hands-on-title>Remove duplicates</hands-on-title>
>
> 1. {% tool [MarkDuplicates](toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_MarkDuplicates/3.1.1.0) %} with the following parameters:
>    - {% icon param-collection %} *"Select SAM/BAM dataset or dataset collection"*: Select the output of  **Filter** {% icon tool %} *"BAM"*
>    - *"If true do not write duplicates to the output file instead of writing them with appropriate flags set"*: `Yes`
>
>    > <comment-title>Default of  **MarkDuplicates** {% icon tool %}</comment-title>
>    >
>    > By default, the tool will only "Mark" the duplicates. This means that it will change the Flag of the duplicated reads to enable them to be filtered afterwards. We use the parameter *"If true do not write duplicates to the output file instead of writing them with appropriate flags set"* to directly remove the duplicates.
>    {: .comment}
>
> 2. Click on the {% icon galaxy-eye %} (eye) icon of the MarkDuplicate metrics.
{: .hands_on}

> <comment-title>MarkDuplicates Results</comment-title>
> You should get a similar output to this from MarkDuplicates:
> ![Metrics of MarkDuplicates](../../images/cut_and_run/mark_duplicates_screen.png "Metrics of MarkDuplicates")
{: .comment}

> <tip-title>Formatting the MarkDuplicate metrics for readability</tip-title>
>
> 1. {% tool [Select lines that match an expression](Grep1) %} with the following parameters:
>    - {% icon param-collection %} *"Select lines from"*: Select The *tabular* output of  **MarkDuplicates** {% icon tool %}
>    - *"that*: `Matching`
>    - *"the pattern*: `(Library|LIBRARY)`
> 2. Check that the datatype is tabular. If not, change it.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 3. {% tool  [Transpose rows/columns in a tabular file](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.1.0) %}:
>    - {% icon param-collection %} *"Select lines from"*: Select the output of **Select** {% icon tool %}
>
> ![Metrics of MarkDuplicates](../../images/cut_and_run/mark_duplicates_transpose.png "Metrics of MarkDuplicates")
>
{: .tip}

> <question-title></question-title>
>
> 1. How many pairs were in the input?
> 2. How many pairs are duplicates?
>
> > <solution-title></solution-title>
> >
> > 1. 81460 for Rep1 and 100507 for Rep2
> > 2. 982 for Rep1 and 1042 for Rep2
> >
> {: .solution}
>
{: .question}

Once again, if you have a high number of duplicates it does not mean that your data are not good, it just means that you sequenced
too much compared to the diversity of the library you generated. Consequently, libraries with a high portion of duplicates should not be re-sequenced as this would not increase the amount of data.

## Check Deduplication and Adapter Removal

> <hands-on-title>Check Adapter Removal with Falco</hands-on-title>
>
> 1. {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} with the following parameters:
>       - {% icon param-collection %} *"Raw read data from your current history"*: select the output of **MarkDuplicates** BAM.
>
> 2. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3) %} to aggregate the FastQC reports with the following parameters:
>    - In *"Results"*:
>        - *"Results"*
>            - *"Which tool was used generate logs?"*: `FastQC`
>                - In *"FastQC output"*:
>                    - {% icon param-repeat %} *"Insert FastQC output"*
>                        - {% icon param-collection %} *"FastQC output"*: `Falco on collection N: Raw data` (output of **Falco** {% icon tool %})
>
{: .hands_on}

> <comment-title>FastQC Results</comment-title>
> Now, you should see under **Overrepresented sequences** that there is no more overrepresented sequences and under **Adapter Content** that the Illumina adapters are no longer present.
> ![MultiQC screenshot on the adapter content section after Trim Galore!](../../images/cut_and_run/fastqc_adapter_content_plot_post.png "MultiQC screenshot on the adapter content section after Trim Galore!")
> However, you may have noticed that you have a new section with warning: **Sequence Length Distribution**. This is expected as you trimmed part of the reads.
> Furthermore, you will notice under **Sequence Duplication Levels** that we removed nearly all PCR duplicates.
> ![MultiQC screenshot on the duplication level after MarkDuplicates](../../images/cut_and_run/fastqc_sequence_duplication_levels_plot.png "MultiQC screenshot on the duplication level after MarkDuplicates")
{: .comment}

## Check Insert Sizes

We will check the insert sizes with **Paired-end histogram** of insert size frequency. The insert size is the distance between the R1 and R2 read pairs. This tells us the size of the DNA fragment the read pairs came from. The fragment length distribution of a sample gives an excellent indication of the quality of the CUT and RUN experiment.

> <hands-on-title>Plot the distribution of fragment sizes</hands-on-title>
>
> 1. {% tool [Paired-end histogram](toolshed.g2.bx.psu.edu/repos/iuc/pe_histogram/pe_histogram/1.0.2) %} with the following parameters:
>    - {% icon param-collection %} *"BAM file"*: Select the output of  **MarkDuplicates** {% icon tool %} *"BAM output"*
>    - *"Lower bp limit (optional)"*: `0`
>    - *"Upper bp limit (optional)"*: `1000`
>
> 2. Click on the {% icon galaxy-eye %} (eye) icon of the lower one of the 2 outputs (the png file).
{: .hands_on}

> <comment-title>Paired-end histogram Results</comment-title>
> This is what you get from Paired-end histogram:
> ![Fragment size distribution](../../images/cut_and_run/insert_size.png "Fragment size distribution")
{: .comment}

> <question-title></question-title>
>
> Could you guess what the peaks at approximately 30bp correspond to?
>
> > <solution-title></solution-title>
> >
> > The first peak (30bp) corresponds to where the cleavage around our POI. Nevertheless, the cutting of our pA(G)-MNase-antibody-protein complex is not perfect. The arms can actually cut further away from the protein. Little bumps in the distribution are an indication of another protein(complex), such as a nucleosome that is located between the two cleavage sites.
> >
> {: .solution}
>
{: .question}

This fragment size distribution is a good indication if your experiment worked or not. The CUT&RUN yield can gradually increase over the digestion time {% cite Skene2017 %}, which of course depends on the enzyme used for the cleavage of the DNA.

![CUT and RUN yield](../../images/cut_and_run/cut_and_run_yield.png "Fragment length distributions of a CUT&RUN experiment with uniform digestion over time. No anti-FLAG primary antibody (No Ab). The track with No_Ab is the control experiment. Data is from two biological replicates. The studied protein showed fragments of size ≤120 bp corresponding to the binding sites, as seen on the left, whereas reads with ≥150 bp fragment size are randomly distributed in the local vicinity. The yield in the example increases from 5s to 10 minutes, which can be seen in the mapped profiles in the right. (Skene and Henikoff 2017 eLife)")

# Step 4: Peak calling

## Call Peaks

We have now finished the data preprocessing. Next, to find regions corresponding to potential binding sites, we want to identify regions where reads have piled up (peaks) greater than the background read coverage. [MACS2](https://github.com/taoliu/MACS) is widely used for CUT&RUN.

At this step, two approaches exist: Like in ATAC-Seq, we have to take a few things into account. The pA(G)-MNase does not need to bind the protein it can create fragments unspecifically a
few bases downstream and upstream away from the true binding site. This is to the *"long"* arms of the proteins. A big jump can happen if a nucleosome is between the protein binding site and the cutting site.

It is critical to re-center the signal since the pA(G)-MNase cuts in different distances. Since some read pairs are not symmetric (i.e., the first mate might be further away from the
binding site than the second mate) we will turn off the shifting mode of MACS2 and apply a direct shift. If we only assess the coverage of the 5' extremity of the reads, the data would be too sparse,
and it would be impossible to call peaks. Thus, we will extend the start sites of the reads by 200bp (100bp in each direction) to assess coverage.

### Using MACS2

We convert the BAM file to BED format because when we set the extension size in MACS2, it will only consider one read of the pair while here we would like to use the information from both.

> <hands-on-title>Convert the BAM to BED</hands-on-title>
>
> 1. {% tool [bedtools BAM to BED converter](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_bamtobed/2.31.1+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Convert the following BAM file to BED"*: Select the output of **MarkDuplicates** {% icon tool %} BAM output
>
{: .hands_on}

We call peaks with MACS2. To get the coverage centered on the 5' extended 100bp each side we will use `--shift -100` and `--extend 200`:
![MACS2 options to get 100bp each side](../../images/atac-seq/macs2Options.jpg "MACS2 options to get 100bp each side")

> <hands-on-title>Call peaks with MACS2</hands-on-title>
>
> 1. {% tool [MACS2 callpeak](toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.2.9.1+galaxy0) %} with the following parameters:
>    - *"Are you pooling Treatment Files?"*: `No`
>        - {% icon param-collection %} Select the output of **bedtools BAM to BED** converter {% icon tool %}
>    - *"Do you have a Control File?"*: `No`
>    - *"Format of Input Files"*: `Single-end BED`
>    - *"Effective genome size"*: `H. sapiens (2.7e9)`
>    - *"Build Model"*: `Do not build the shifting model (--nomodel)`
>        - *"Set extension size"*: `200`
>        - *"Set shift size"*: `-100`. It needs to be - half the extension size to be centered on the 5'.
>    - *"Additional Outputs"*:
>        - Check `Peaks as tabular file (compatible with MultiQC)`
>        - Check `Peak summits`
>        - Check `Scores in bedGraph files`
>    - In *"Advanced Options"*:
>        - *"Composite broad regions"*: `No broad regions`
>            - *"Use a more sophisticated signal processing approach to find subpeak summits in each enriched peak region"*: `Yes`
>        - *"How many duplicate tags at the exact same location are allowed?"*: `all`
>
>    > <comment-title>Why keeping all duplicates is important</comment-title>
>    >
>    > We previously removed duplicates using **MarkDuplicates** {% icon tool %} using paired-end information. If two pairs had identical R1 but different R2, we knew it was not a PCR duplicate. Because we converted the BAM to BED we lost the pair information. If we keep the default (removing duplicates) one of the 2 identical R1 would be filtered out as duplicate.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many peaks have been identified in each replicate?
>
> > <solution-title></solution-title>
> >
> > 1. 6314 for Rep1 and 7678 for Rep2
> >
> {: .solution}
>
{: .question}

# Step 5: Identifying Binding Motifs

## Prepare the Datasets


### Extracting individual replicates from your dataset collection

In the following step, we want to apply a tool to identify robust peaks between our two replicates. However, we still have one dataset collection, holding both replicates, which we have to split.

> <hands-on-title>Extract replicates from a dataset collection:</hands-on-title>
>
> 1. {% tool [Extract Dataset](__EXTRACT_DATASET__) %} with:
>    - {% icon param-collection %} *"Input List"*: `MACS2 callpeak on collection N (narrow Peaks)`
>    - *"How should a dataset be selected?"*: `Select by index`
>    - *"Element index:"*: `0`
>
>    You can use the arrow to rerun the same tool just changing the last parameter:
>
> 2. {% tool [Extract Dataset](__EXTRACT_DATASET__) %} with:
>    - {% icon param-collection %} *"Input List"*: `MACS2 callpeak on collection N (narrow Peaks)`
>    - *"How should a dataset be selected?"*: `Select by index`
>    - *"Element index:"*: `1`
{: .hands_on}

### Extract Robust Peaks
MACS helped us to identify potential binding sites for GATA1. Yet, our peak set will contain false positives because of technical and biological noise.
We can first try to reduce some noise using the results for our two biological replicates. Since we applied MACS we have created two peak files.
Yet, both replicates have peaks that are only present in one replicate and thus might be a potential false positive due to noise.
We can remove such peaks if we simply overlap the two peak files and consequently search for peaks that appear in both replicates.

> <hands-on-title>Select robust GATA1 peaks:</hands-on-title>
>
> 1. {% tool [bedtools Intersect intervals find overlapping intervals in various ways](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.31.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"File A to intersect with B"*: Select `Rep1`
>    - *"Combined or separate output files"*: `One output file per 'input B' file`
>        - {% icon param-file %} *"File B to intersect with A"*:  Select `Rep2`
>    - *"Calculation based on strandedness?"*: `Overlaps occurring on either strand`
>    - *"Required overlap"*: `Specify minimum overlap(s)`
>        - *"Minimum overlap required as a fraction of A"*: `0.5`
>        - *"Require that the fraction of overlap be reciprocal for A and B"*: `Yes`
>    - *"Write the original A entry _once_ if _any_ overlaps found in B.": `Yes`
>
> 2. Rename the datasets `Robust GATA1 CUT and RUN peaks`.
>
> > <question-title></question-title>
> >
> > 1. How many potential true positives do we obtain?
> > 2. How many potential false positives have we removed?
> > 3. Why have we set specified an overlap fraction of A with `0.5`?
> > 4. Why have we set **"Require that the fraction of overlap be reciprocal for A and B"**?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. 2,865 peaks
> > > 2. We started with 6,314 and 7,678 peaks. Thus, we could remove 3,449 and 4,813 peaks.
> > > 3. Finding robust peaks is a tricky process and not as simple as you think. Our approach defines a robust peak as an entity that is covered by two similar peaks in either one of the peak files of our two replicates. Our measurement for similarity is simply the amount of bases the peaks overlap. Yet, ask yourself, how much overlap do you need to state that they are similar? Is 1 base enough or maybe 10? The answer is not true or false and like so often needs more investigation. We define the overlap by a 50% fraction of each of the peak files. That means, if our peak in A is 100 bases then A has to overlap with 50 bases with the peak in file B.
> > > 4. **"Require that the fraction of overlap be reciprocal for A and B"** means that 50% of the peak in file A overlaps with the peak in file B and also 50% of the peak in file B overlaps with the peak in file A. We set this to make our filtering very conservative and filter for peaks that we can be sure are true positives. For example, A has a 10 bp peak that overlaps a peak in B with 100 bp. **Viewpoint of file A:** An overlap-fraction of 0.5 for A means that we accept peak A if it overlaps with 5 bp with the peak in B. This is very likely because the peak in B is 100 bp long. **Viewpoint of file B**: An overlap-fraction of 0.5 for B means that we reject A no matter what, even if A completely overlaps B because 10 bp are not enough to cover 50% of B. That is to say the option **"Require that the fraction of overlap be reciprocal for A and B"** takes care that we trust peaks that have in file A **and** in file B a fair overlap.
> > {: .solution}
> >
> {: .question}
{: .hands_on}

Note, that this is one way to define robust peaks. Yet, there are more sophisticated algorithms in Galaxy, such as
{% tool [IDR compare ranked list of identifications](toolshed.g2.bx.psu.edu/repos/iuc/idr/idr/2.0.3) %}, and other methods that can help to define a robust peak.
The tool **IDR** has not been tested so far for CUT and RUN.

### Extract Positive Peaks

We can further remove some noise with a positive control, that is why we have downloaded a peak set from a ChIP experiment of GATA1.

> <hands-on-title>Select GATA1 peaks from ChIP-Seq data:</hands-on-title>
>
> 1. {% tool [bedtools Intersect intervals find overlapping intervals in various ways](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.31.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"File A to intersect with B"*: Select **Robust GATA1 CUT and RUN peaks**
>    - *"Combined or separate output files"*: `One output file per 'input B' file`
>        - {% icon param-file %} *"File B to intersect with A"*:  Select the dataset `GATA1 ChIP-Seq peaks`
>    - *"Calculation based on strandedness?"*: `Overlaps occurring on either strand`
>    - *"Required overlap"*: `Specify minimum overlap(s)`
>        - *"Minimum overlap required as a fraction of A"*: `0.5`
>        - *"Require that the fraction of overlap be reciprocal for A and B"*: `No`
>    - *"Write the original A entry _once_ if _any_ overlaps found in B.": `Yes`
>
> 3. Rename the datasets `True GATA1 CUT and RUN peaks`.
>
> > <question-title></question-title>
> >
> > 1. How many potential true positives have we found (common between CUT&RUN and ChIP-seq) ?
> > 2. How many potential false positives have we removed?
> > 3. What is the precision of our analysis at this point? (Precision = True Positive / True Positive + False Positive)
> >
> > > <solution-title></solution-title>
> > >
> > > 1. 2,809 peaks
> > > 2. 56 peaks
> > > 3. Our precision is ~ 98%. A high precision is an indicator that we can predict true binding regions with high confidence.
> > {: .solution}
> >
> {: .question}
{: .hands_on}

### Extract DNA sequence

> <hands-on-title>Obtain DNA sequences from a BED file</hands-on-title>
>
> 1. {% tool [Extract Genomic DNA using coordinates from assembled/unassembled genomes](toolshed.g2.bx.psu.edu/repos/iuc/extract_genomic_dna/Extract genomic DNA 1/3.0.3+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Fetch sequences for intervals in"*: `True GATA1 CUT and RUN peaks`
>    - *"Interpret features when possible"*: `No`
>    - *"Choose the source for the reference genome": `locally cached`
>    - *"Using reference genome"*: `hg38`
>    - *"Select output format"*: `fasta`
>    - *"Select fasta header format"*: `bedtools getfasta default`
>
{: .hands_on}

## Motif detection with MEME-ChIP

Let's find out the sequence motifs of the TF GATA1. Studies have revealed that GATA1 has the DNA binding motif (T/A)GATA(A/G) {% cite Hasegawa2017 %}. In this last task we search for this motif in our data.

> <hands-on-title>Motif detection</hands-on-title>
>
> 1.  {% tool [MEME-ChIP](toolshed.g2.bx.psu.edu/repos/iuc/meme_chip/meme_chip/4.11.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Primary sequences"*: Select the output of **Extract Genomic DNA** {% icon tool %}
>    - *"Sequence alphabet"*: `DNA`
>    - *"Options Configuration"*: `Advanced`
>        - *"Should subsampling be random?"*: `Yes`
>            - *"Seed for the randomized selection of sequences"*: `123`
>        - *"E-value threshold for including motifs"*: `0.05`
>        - *"What is the expected motif site distribution?"*: `Zero or one occurrence per sequence`
>        - *"Minimum motif width"*: `5`
>        - *"Maximum motif width"*: `20`
>        - *"Maximum number of motifs to find"*: `5`
>        - *"Stop DREME searching after reaching this E-value threshold"*: `0.05`
>        - *"Stop DREME searching after finding this many motifs"*: `5`
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
> 2. View the **MEME-ChIP** HTML.
>
> > <question-title></question-title>
> >
> > 1. What is the E-value of the main motif?
> > 2. How many peaks support the main motif GATA?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. 1.1e-411
> > > 2. You need to click on the DREME link for the GATA motif. Then when you are in the DREME html, click on the arrow pointing down for the first motif (HGATAA) in the More column. Here, it gives you some numbers including how many peaks were positive for the motif. Here 1939.
> > {: .solution}
> >
> {: .question}
{: .hands_on}

![MEME-ChiP motifs for GATA1](../../images/cut_and_run/GATA_motif.png "Plot of the first five sequence motifs of MEME-ChIP.")

Let us first investigate the result of MEME-ChIP. **First column:** The x-axis of the sequence plots represents the nucleotide position of the motif.
The y-axis stands for the total information (uncertainty) and thus stands for the probability that the nucleotide at a certain position is the specific letter (for DNA: T,C,G,A).
Bigger letters stand for a higher probability. For more information please read [the corresponding Wikipedia page](https://en.wikipedia.org/wiki/Sequence_logo). **Second column:** Defines the algorithm that found the sequence motif.
MEME-ChIP uses two different approaches called MEME and DREME, which can find more complex or simpler motifs. More information about these tools can be found [in the Meme documentation](http://meme-suite.org/).
**Third column:** Defines the E-value, which represents the expected number of times we would find our sequence motif in a database (peak file), just by chance.
This means, that a small E-value corresponds to a very significant motif because the expected number we would find that motif in our peak file just by chance is very low.
Other sequences like repeats will have a high E-value on the other hand. **Fourth column:** This graph is the result of CentriMo, a tool that shows the distribution of
the best matches for that particular motif. The vertical line is the center of the motif. Therefore, if you have a long sequence motif CentriMo helps you to pinpoint a smaller sub-motif.
You can find more information on CentriMo [in the Meme documentation](https://meme-suite.org/meme/doc/centrimo.html).

The results of MEME-ChIP endorse the findings about the DNA binding motif (T/A)GATA(A/G) {% cite Hasegawa2017 %}. We also found other motifs, that might be secondary sequence motifs.
This, we could test with a deeper analysis.

# Step 6: Visualisation of Coverage

In the previous section we generated the `True GATA1 CUT and RUN peaks` file by eliminating the false positives. In this step, we will visualize the the coverage across the peaks using the `bedgraph` file generated by **MACS2** {% icon tool %}.

## Convert bedgraph from **MACS2** to bigwig
The BedGraph format is human-readable but can be quite large, making it slow to visualize specific regions. For better efficiency, we will convert it to the BigWig format, a binary format that allows for fast visualization of any genomic region.

> <hands-on-title>Convert BedGraphs to BigWig.</hands-on-title>
>
> 1. {% tool [Wig/BedGraph-to-bigWig](wig_to_bigWig) %} with the following parameters:
>    - {% icon param-file %} *"Convert"*: Select the output of **MACS2** {% icon tool %} (Bedgraph Treatment).
>    - *"Converter settings to use"*: `Default`
>
> 2. Rename the datasets `MACS2 bigwig`.
{: .hands_on}


## Create heatmap of coverage at TSS with deepTools

You might be interested in checking the coverage on binding GATA1 binding sites. For this, you can compute a heatmap. We will use the **deepTools plotHeatmap**. Here we make a heatmap centered on the transcription start sites (TSS).

### Generate computeMatrix

The input of **plotHeatmap** is a matrix in a hdf5 format. To generate it we use the tool **computeMatrix** that will evaluate the coverage at each GATA1 peak.

> <hands-on-title>Generate the matrix</hands-on-title>
>
> 1. {% tool [computeMatrix](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.5.4+galaxy0) %} with the following parameters:
>    - In *"Select regions"*:
>        - {% icon param-repeat %} *"Insert Select regions"*
>            - {% icon param-file %} *"Regions to plot"*: `True GATA1 CUT and RUN peaks`
>    - *"Sample order matters"*: `No`
>        - {% icon param-file %} *"Score file"*: `MACS2 bigwig`, output of **Wig/BedGraph-to-bigWig** {% icon tool %}.
>    - *"computeMatrix has two main output options"*: `reference-point`
>    - *"The reference point for the plotting"*: `beginning of region (e.g. TSS)`
>    - *"Show advanced output settings"*: `no`
>    - *"Show advanced options"*: `yes`
>        - *"Convert missing values to 0?"*: `Yes`
>
{: .hands_on}


### Plot with **plotHeatmap**

Now we will generate a heatmap. Each line will be a peak. The coverage will be summarized with a color code from red (no coverage) to blue (maximum coverage). All TSS will be aligned in the middle of the figure and only the 1 kb around the TSS will be displayed. Another plot, on top of the heatmap, will show the mean signal at the TSS. There will be one heatmap per replicate.

> <hands-on-title>Generate the heatmap</hands-on-title>
>
> 1. {% tool [plotHeatmap](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_heatmap/deeptools_plot_heatmap/3.3.2.0.1) %} with the following parameters:
>    - {% icon param-file %} *"Matrix file from the computeMatrix tool"*: output of **computeMatrix** {% icon tool %}.
>    - *"Show advanced output settings"*: `no`
>    - *"Show advanced options"*: `Yes`
>    - *"The x-axis label”*: `distance from peak center (bp)`
>    - *"The y-axis label for the top panel"* : `GATA1 peaks`
>    - *"Reference point label”*: `peak center`
>    - *"Labels for the regions plotted in the heatmap"*: `GATA1_peaks`
{: .hands_on}

> <comment-title>plotHeatmap Results</comment-title>
> This is what you get from plotHeatmap:
> ![plotHeatmap output](../../images/cut_and_run/peak_coverage_heatmap.png "plotHeatmap output")
{: .comment}

> <question-title></question-title>
>
> 1. Is the coverage similar in replicates?
> 2. What does the coverage tell us about the binding sites?
>
> > <solution-title></solution-title>
> >
> > 1. Except a slightly more average coverae in replicate 2, both of them have similar sharp pattern.
> > 2. Such sharp coverage indicates that the binding sites are highly specific, which is expected for transcription factor binding.
> >
> {: .solution}
>
{: .question}


# Conclusion

In this training, you have learned the general principles of CUT&RUN data analysis. CUT&RUN is a method to analyse DNA-associated proteins. In contrast to ChIP-seq, CUT&RUN couples the
antibody with a pA(G)-MNase to reduce the size of the obtained sequences to increase the accuracy for the detection of the protein interaction. Based on this training material,
you gained some insights into the quality control of the data. You should know some pitfalls of CUT&RUN and look for adapter contamination and PCR duplicates, which disrupt the later enrichment analysis.
You learned about **Bowtie2** to do the alignment, and how to filter for properly paired, good-quality and reads that do not map to the mitochondrial genome. Furthermore, you learned that
an indicator for a good CUT&RUN experiment is the fragment size plot and that you optimize your protocol based on this parameter. With **MACS2** we identified enriched regions, but
we had to remove some noise. We performed some robust peak analysis with **bedtools intersect** and performed a check with a ChIP-Seq experiment to remove some false positives.
Finally, we used **MEME-ChIP** to identify our binding motifs.

![CUT&RUN workflow](../../images/cut_and_run/cut_and_run_mindmap.png)


topic: epigenetics
tutorial: ewas-suite
title: Infinium Human Methylation BeadChip
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 414269423520ff659eb984483268a833
last_modified: 2026-02-26T21:12:15.675215
zenodo_link: https://zenodo.org/record/1251211
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/epigenetics/tutorials/ewas-suite/tutorial.html

content:
This tutorial is based on Hugo W, Shi H, Sun L, Piva M et al.: Non-genomic and Immune Evolution of Melanoma Acquiring MAPKi Resistance {% cite Hugo2015 %}.

> <agenda-title></agenda-title>
> In this tutorial we will do:
> 1. TOC
> {:toc}
> We will use a small subset of the original data. If we run the tutorial on the orginal dataset, analysis will be time consuming and not reproducible Infinium Human Methylation BeadChip computation on the orginal data can be found at [case study](https://github.com/kpbioteam/case_study).
>
>
{: .agenda}

# Introduction


The field of cancer genomics has demonstrated the power of massively parallel sequencing techniques to inform on genes and specific alterations that drive tumor onset and progression. Although large comprehensive sequence data sets continue to be made increasingly available, data analysis remains an ongoing challenge, particularly for laboratories lacking dedicated resources and bioinformatics expertise. To address this, we have provided training based on Galaxy Infinium Human Methylation BeadChip tool that represents many popular algorithms for detecting somatic genetic alterations from genome and exome data.

![epimechanism](../../images/Epigenetic_mechanisms.jpg "How epigenetics mechanism can effect health (adapted from https://commonfund.nih.gov/epigenomics/figure)")

This exercise uses datasets from the Cell publication by {% cite Hugo2015 %}. with the goal being the identification of differentially methylated regions and positions associated with treatment resistant melanomas.
Datasets include the Infinium Human Methylation BeadChip array performed in melanoma tumors in a sample of patients pre and post MAPKi and BRAFi treatment with different outcomes (sensitive and resistant). For each sample there is raw green (methylated) and red (unmethylated) colour arrays containing the summarised bead information generated by the Infinium Human Methylation BeadChip scanner.

The Infinium Human Methylation BeadChip uses two different bead types to detect changes in DNA methylation levels. In the figure we can see M - methylated and U - unmethylated bead types. In our study unmethylated and methylated bead signals are reported as green and red colors respectively.

![methassay](../../images/methylationassay.png "Infinium Methylation Assay Overview (adapted from {% cite Illumina %})")


Accession  | Sensitivity | Treatment
--- | ---  | ---
GSM1588704 | baseline    | pre-treatment
GSM1588705 | baseline    | pre-treatment
GSM1588706 | resistant   | BRAFi
GSM1588707 | resistant   | BRAFi

The workflow combines 5 main steps, starting with raw intensity data loading (.idat) and then optional preprocessing and normalisation of the data. The next quality control step performs an additional sample check to remove low-quality data, which normalisation cannot detect. The workflow gives the user the opportunity to perform any of these preparation and data cleaning steps, including a highly recommended genetic variation annotation step resulting in single nucleotide polymorphism identification and removal. Finally, the dataset generated through all of these steps can be used to hunt (find) differentially-methylated positions (DMP) and regions (DMR) with respect to a phenotype covariate.

# Raw intensity data loading

The first step of the Infinium Human Methylation BeadChip array analysis is raw methylation data loading (intensity information files for each two colour micro array)

> <hands-on-title>Data Loading</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the following IDAT files from [Zenodo](https://zenodo.org/record/1251211#.WwREQ1Mvz-Y) or from the data library (ask your instructor)
>    - `GSM1588704_8795207135_R01C02_Red.idat`
>    - `GSM1588705_8795207119_R05C02_Red.idat`
>    - `GSM1588706_8795207135_R02C02_Red.idat`
>    - `GSM1588707_8795207119_R06C02_Red.idat`
>    - `GSM1588704_8795207135_R01C02_Grn.idat`
>    - `GSM1588705_8795207119_R05C02_Grn.idat`
>    - `GSM1588706_8795207135_R02C02_Grn.idat`
>    - `GSM1588707_8795207119_R06C02_Grn.idat`
>    - `phenotypeTable.txt`
>
>    > <details-title>List of Zenodo URLs</details-title>
>    > ```
>    > https://zenodo.org/record/1251211/files/GSM1588704_8795207135_R01C02_Red.idat
>    > https://zenodo.org/record/1251211/files/GSM1588706_8795207135_R02C02_Red.idat
>    > https://zenodo.org/record/1251211/files/GSM1588705_8795207119_R05C02_Red.idat
>    > https://zenodo.org/record/1251211/files/GSM1588707_8795207119_R06C02_Red.idat
>    > https://zenodo.org/record/1251211/files/GSM1588704_8795207135_R01C02_Grn.idat
>    > https://zenodo.org/record/1251211/files/GSM1588706_8795207135_R02C02_Grn.idat
>    > https://zenodo.org/record/1251211/files/GSM1588705_8795207119_R05C02_Grn.idat
>    > https://zenodo.org/record/1251211/files/GSM1588707_8795207119_R06C02_Grn.idat
>    > https://zenodo.org/record/1251211/files/phenotypeTable.txt
>    > ```
>    {: .details}
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>    > <comment-title>Phenotype table</comment-title>
>    > Phenotype table can be in different sizes with different arguments, however the second column is required to contain phenotype covariate information for each sample.
>    {: .comment}
>
> 3. Run **UCSC Main** {% icon tool %} to obtain the reference genome. The tool will take you to the **UCSC table browser**. Use the following parameters to extract the reference genome
>   - *"clade"*: `Mammal`
>   - *"genome"*: `Human`
>   - *"assembly"*: `Feb. 2009 (GRCh37/hg19)`
>   - *"group"*: `Regulation`
>   - *"track"*: `HAIB Methyl450`
>   - *"table"*: `GM12878 (wgEncodeHaibMethyl450Gm12878SitesRep1)`
>   - *"region"*: `genome`
>   - *"output format"*: `GTF - gene transfer (limited)`
>   - *"Send output to"*: `Galaxy` (only)
>   - Click on the **get output** button at the bottom of the screen
>   - On the next page, click on the **Send Query to Galaxy** button
>   - Wait for the upload to finish
>
>
> After exporting the reference genome from UCSC, we need to make sure that it is in the right dataset build.
>
> Click on the **Differentially_Methylated_Positions.bed** output in your history to expand it. \
> Set the database build of your dataset to `Human Feb. 2009 (GRCh37/hg19) (hg19)`(if it is not set automatically)
>
> {% snippet faqs/galaxy/datasets_change_dbkey.md dbkey="hg19" %}
>
> Click on `display at UCSC` towards the bottom of the history item.
> This will launch UCSC Genome Browser with your Custom Track
> ![Display at UCSC](../../images/ucsc.png "UCSC genome track showing differentialy methylated regions located on chromosome 6")
{: .hands_on}


# .idat preprocessing

Preprocessing and data quality assurance is an important step in Infinium Methylation Assay analysis. Idat dataset represents two colour data with a green and a red channel and can be converted into methylated and unmethylated signals or into Beta values. The Infinium Human Methylation BeadChip tool extracts and plots the quality control data frame with two columns mMed and uMed which are the medians of methylation signals (Meth and Unmeth). Comparing them against one another allows users to detect and remove low-quality samples.

![quality_control](../../images/quality_control.png)

> <comment-title>Normalisation of the data</comment-title>
> If your files require normalisation, you might prefer to use one of the other preprocessing tools provided in Infinium Human Methylation BeadChip tool i.e. Preprocess Funnorm or Preprocess Quantile look for recommendation at {% cite Hansen %}.
>
{: .comment}

# Differentially methylated regions and positions analysis

The main goal of the **Infinium Human Methylation BeadChip** analysis is to simplify the way differentially methylated loci sites are detected. The **Infinium Human Methylation BeadChip** pipeline contains differentially methylated positions (DMPs) detection with respect to a phenotype covariate, and more complex solutions for finding differentially methylated regions (DMRs). Genomic regions that are differentially methylated between two conditions can be tracked using a bumphunting algorithm. The algorithm first implements a t-statistic at each methylated loci location, with optional smoothing, then groups probes into clusters with a maximum location gap and a cutoff size to refer the lowest possible value of genomic profile hunted by our tool.

> <hands-on-title>detecting methylated loci sites</hands-on-title>
> 1. Run {% tool [Infinium Human Methylation BeadChip](toolshed.g2.bx.psu.edu/repos/kpbioteam/ewastools/minfi_analysis/2.1.0) %} with the following parameters to map the imported datasets against phenotype covariate and reference genome obtained from UCSC.
>    - {% icon param-files %} *"red channel files"*: all files ending in `_Red`
>    - {% icon param-files %} *"green channel files"*: all files ending in `Grn`
>
>
> Ilumina methylation array data can be mapped to the genome with or without additional preprocessing methods. Incomplete annotation of genetic variations such as single nucleotide polymorphism (SNP) may affect DNA measurements and disrupt downstream analysis of results. {% cite Hansen %} It is highly recommended to remove the probes that contain either an SNP at the methylated loci interrogation or at the single nucleotide extension. In this tutorial we will remove probes affected by genetic variation by selecting **(Optional) Preprocessing Method** {% icon tool %}.
>
>    - *"(Optional) Preprocessing Method"*: `Remove SNPS`
>
>
>    - *"Phenotype Table"*:`The phenotypeTable.txt file uploaded from Zenodo`
>    - *"maxGap Size"*:`250`
>    We will use the default gap of 250 base pairs (bps), i.e. any two points more than 250 bps away are put in a new cluster.
>    - *"Cutoff Size"*:`0.1`
>    In order to find segments that are positive, near zero, and negative. We need a cutoff which is one number in which case “near zero” default 0.1
>    - *"Number of Resamples"*:`0`
> Default value 0 for permutation method apply selection of randomized cases with replacement from the original data while using 'bootstrap' method.
>    - *"nullMethod"*:`permutation`
> Method used to generate null candidate regions, must be one of ‘bootstrap’ or
> ‘permutation’ (defaults to ‘permutation’).
>    - *"Phenotype Type"*:`categorical`
> Identify regions where methylation is associated with a continuous or categorical phenotype.
>    - *"qCutoff Size"*:`0.5`
> Diffrentialy methylated positions with an FDR q-value greater than this value will not be returned.
>    - *"Variance Shrinkage"*:` TRUE`
> Default TRUE as it is recommended when sample sizes are small <10
>    - *"Genome Table"*: `wgEncodeHaibMethyl450 ... `
>
{: .hands_on}

> <question-title></question-title>
> How do we define phenotype covariate?
> > <solution-title></solution-title>
> > Phenotype covariate is the set of observable characteristics of an individual resulting from the gene-environment interactions
> {: .solution}
{: .question}
# Annotation and visualization
In addition to downstream analysis users can annotate the differentially methylated loci at the promoter regions of genes with gene function descriptions, and relationships between these concepts.
> <hands-on-title> Annotate Differentially Methylated Position</hands-on-title>
> 1. Run {% tool [ChIPpeakAnno annoPeaks](toolshed.g2.bx.psu.edu/repos/kpbioteam/chipeakanno_annopeaks/chipeakanno_annopeaks/0.1.0) %} on the output of {% tool [Infinium Human Methylation BeadChip](toolshed.g2.bx.psu.edu/repos/kpbioteam/ewastools/minfi_analysis/2.1.0) %} with the following parameters
>   - {% icon param-file %} *"Differentialy methylated Positions"*: output of {% tool [Infinium Human Methylation BeadChip](toolshed.g2.bx.psu.edu/repos/kpbioteam/ewastools/minfi_analysis/2.1.0) %}
>   - *"bindingType"*: `StartSite`
>   - *"bindingRegionStart"*:`-5000`
>   - *"bindingRegionEnd"*:`3000`
>   - *"Additional Column of Score"*:`5`
>
>        Position of column of score optional value if it is required
>
> 2. Run {% tool [Cut](Cut1) %} on the previous output adjusting the following parameters to cut "gene_name" column from table of annotated peaks and then get a list of genes
>   - *"Cut columns"*: `c16`
>   - *"Delimited by"*: `Tab`
>   - {% icon param-file %} *"From"*: output of {% tool [ChIPpeakAnno annoPeaks](toolshed.g2.bx.psu.edu/repos/kpbioteam/chipeakanno_annopeaks/chipeakanno_annopeaks/0.1.0) %}
>
> 3. Use {% tool [Remove beginning](Remove beginning1) %} on `Gene List` with the following parameters
>   - *"Remove first"*: `1`
>   - {% icon param-file %} *"from"*: output of {% tool [Cut](Cut1) %}
>
> 4. Run {% tool [Cluster Profiler Bitr](toolshed.g2.bx.psu.edu/repos/kpbioteam/clusterprofiler_bitr/clusterprofiler_bitr/0.1.0) %} on the previous output adjusting the following parameters to convert the list of genes to list of entrez ID
>   - *"Input Type Gene ID"*: `SYMBOL`
>   - *"Output Type Gene ID"*: `ENTREZID`
>
> 5. Run a GO Enrichment Analysis using {% tool  [clusterProfiler go](toolshed.g2.bx.psu.edu/repos/kpbioteam/clusterprofiler_go/clusterprofiler_go/0.1.0) %} on the output of the {% tool  [Cluster Profiler Bitr](toolshed.g2.bx.psu.edu/repos/kpbioteam/clusterprofiler_bitr/clusterprofiler_bitr/0.1.0) %}
{: .hands_on}

![Functional annotations](../../images/funcann.jpg "Results of GO enrichments analysis for DMPs")

ID  | Description | pvalue | qvalue | geneID | Count
--- | ---  | --- | --- | --- | ---
GO:0048732 | gland development  | 1.38E-58 | 4.23E-55 | PTGS2 / KCNC1 / FZD1 /SLC22A18 /SLC22A3 (...) | 372
GO:1901652 | response to peptide | 3.99E-57 | 8.13E-54 | SULF1/ LAMA5/ MED1 /CFLAR/ MSX2 (...) | 359
GO:0048545 | response to steroid hormone | 1.38EE-54 | 2.11E-51 | HDAC9/ RAB10/ CFLAR/ WDTC1 (...) | 394

# Conclusion


Epigenetic aberrations which involve DNA modifications give researchers an interest in identifying novel non-genetic factors responsible for complex human phenotypes such as height, weight, and disease. To identify methylation changes researchers need to perform complicated and time consuming computational analysis. Here, the EWAS suite becomes a solution for this inconvenience and provides a simplified downstream analysis available as a ready to run pipline in supplementary materials. For more details, see the associated publication {% cite Murat2020 %}


topic: genome-annotation
tutorial: repeatmasker
title: Masking repeats with RepeatMasker
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["eukaryote", "biodiversity"]
content_hash: c53b148d7ae6ec35d43497c24950ee5c
last_modified: 2026-02-26T21:12:16.600421
zenodo_link: https://zenodo.org/record/7085837
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/repeatmasker/tutorial.html

content:
When you assemble a new genome, you get its full sequence in FASTA format, in the form of contigs, scaffolds, or even whole chromosomes if you are lucky. However genomes, in particular for eukaryote organisms, contain a varying but significant proportion of repeated elements all along the sequence. These elements belong to different classes, including:

- Tandem repeats: small sequences (<60 base pairs) repeated next to each other, found in many places in the genome, in particular centromeres and telomeres
- Interspersed repeats: sequences repeated in distant positions, including transposons, {SINEs} or {LINEs}

These repeats are interesting on their own: they can originate from transposons or viral insertions, and they can have direct effects on the expression of genes. But they are also the source of a lot of trouble when you work on genomics data. First when sequencing a genome, assembly tools often have problems reconstructing the genome sequence in regions containing repeats (in particular when repeats are longer than the read size). Then, when you have a good assembly, you want to annotate it to find the location of genes. Unfortunately many annotation tools have trouble identifying gene locations in regions rich in repeats.

The aim of repeat masking is to identify the location of all repeated elements along a genome sequence. Other tools (like annotation pipelines) can then take this information into account when producing their results.

The output of repeat masking tools is most often composed of a fasta file (with sometimes a BED or GFF file containing the position of each repeat). There is two types of masking, producing slightly different fasta output:

- Soft-masking: repeat elements are written in lower case
- Hard-masking: repeat elements are replaced by stretches of the letter N

Normal (non-repeated) sequences are always kept in uppercase. Doing hard-masking is destructive because you lose large parts of the sequence which are replaced by stretches of N. If you want to perform an annotation, it is best to choose soft-masking.

We call this operation "masking" because, by making repeats lowercase, or replacing them with Ns, you make them "invisible" by annotation tools (they are written to mostly ignore the regions marked like this).

Multiple tools exist to perform the masking: [RepeatMasker](https://www.repeatmasker.org/), [RepeatModeler](https://www.repeatmasker.org/RepeatModeler/), [REPET](https://urgi.versailles.inra.fr/Tools/REPET), ... Each one have specificities: some can be trained on specific genomes, some rely on existing databases of repeated elements signatures ([Dfam](https://www.dfam.org/), [RepBase](https://www.girinst.org/repbase/)).

In this tutorial you will learn how to soft-mask the genome sequence of a small eukaryote: Mucor mucedo (a fungal plant pathogen). You can learn how this genome sequence was assembled by following the [Flye assembly tutorial]({% link topics/assembly/tutorials/flye-assembly/tutorial.md %}). We will use two different tools, Red and RepeatMasker, which are probably two of the simplest solutions giving an acceptable result before annotating the genome in the [Funannotate annotation tutorial]({% link topics/genome-annotation/tutorials/funannotate/tutorial.md %}).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/record/7085837/files/genome_raw.fasta
>    https://zenodo.org/record/7085837/files/Muco_library_RM2.fasta
>    https://zenodo.org/record/7085837/files/Muco_library_EDTA.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

# Soft-masking using Red

First let's try Red, a tool than can mask repeats *de novo*. For that, select the input assembly in fasta format.

> <comment-title>*Ab initio* tool</comment-title>
>
> Red is an *ab initio* tool, it means that it will try to predict repeat elements using only the genomic sequence. It's perfect when you know nothing about the organism that you are working on.
>
{: .comment}

> <hands-on-title></hands-on-title>
>
> 1. {% tool [Red](toolshed.g2.bx.psu.edu/repos/iuc/red/red/2018.09.10+galaxy1) %} with the following parameters:
>   - {% icon param-file %} *"Genome sequence to mask"*: `genome_raw.fasta` (Input dataset)
>
{: .hands_on}

Red produces 2 output files :

- A *fasta* file: this is the soft-masked genome that you can use for future analysis. If you display it, you will notice that some portions of the sequence are in lowercase: these are the regions that were identified as repeats
- A *bed* file: this one contains the coordinates on the genome of each repeated loci

> <question-title></question-title>
>
> What proportion of the whole genome sequence is masked?
>
> > <solution-title></solution-title>
> >
> > You need to click on the {% icon galaxy-info %} on one of the output. You should find it at the end of the extended `Tool Standard Output` in **Job Information**. It should be ~ 30.62%
> >
> {: .solution}
>
{: .question}

> <question-title></question-title>
>
> How to hard-mask a genome with **Red** ?
>
> > <solution-title></solution-title>
> >
> > As you can see **Red** has no option to hard-mask your genome. However, one of the output is a `bed` file, so you can use {% tool bedtools MaskFastaBed %} to replace repeated regions with stretches of N:
> >
> > > <hands-on-title></hands-on-title>
> > >
> > > 1. {% tool [bedtools MaskFastaBed](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_maskfastabed/2.30.0) %} with the following parameters:
> > >   - {% icon param-file %} *"BED/bedGraph/GFF/VCF/EncodePeak file"*: `Red on data` (`bed` file produced by red)
> > >   - {% icon param-file %} *"FASTA file"*: `genome_raw.fasta`
> > >
> > {: .hands_on}
> >
> {: .solution}
>
{: .question}

Red uses only the sequence of the genome to detect repeated regions, and does not provide a detailed classification of the detected repeats. Let's use another tool that works differently: RepeatMasker.

# Soft-masking using RepeatMasker

Let's run RepeatMasker, by selected the input assembly in fasta format. We select the soft-masking option, and we choose to use the Dfam database.

> <comment-title>Choosing the right species</comment-title>
>
> We select the `Human (Homo sapiens)` species here, even though we are masking a fungi genome. It means RepeatMasker will identify very common repeats found in many organisms. For more precise results, you can consider selecting a species closer to the one you analyse in the drop down list, or using other more advanced tools like RepeatModeler.
>
{: .comment}

> <hands-on-title></hands-on-title>
>
> 1. {% tool [RepeatMasker](toolshed.g2.bx.psu.edu/repos/bgruening/repeat_masker/repeatmasker_wrapper/4.1.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Genomic DNA"*: `genome_raw.fasta` (Input dataset)
>    - *"Repeat library source"*: `DFam (curated only, bundled with RepeatMasker)`
>    - *"Select species name from a list?"*: `Yes`
>        - *"Species"*: `Human (Homo sapiens)`
>    - *"Output annotation of repeats in GFF format"*: `Yes`
>    - *"Perform soft-masking instead of hard-masking"*: `Yes`
>
{: .hands_on}

RepeatMasker produces 5 output files:

- `masked sequence`: this is the fasta file that you will use for future analysis. If you display it, you will notice that some portions of the sequence are in lowercase: these are the regions that were identified as repeats.
- `repeat statistics`: this one contains some statistics on the number of repeats found in each category, and the total number of base pairs masked.
- `output log`: this is a tabular file listing all repeats.
- `repeat catalogue`: this one contains the list of all repeat sequences that were identified, with their position, and their similarity with known repeats from the Dfam database.
- `repeat annotation` : this one contains the coordinate of each repeat element in GFF2 format.

> <question-title></question-title>
>
> What proportion of the whole genome sequence is masked?
>
> > <solution-title></solution-title>
> >
> > You should find it in the `repeat statistics` output. It should be ~2.41%.
> {: .solution}
>
{: .question}

As we have used a generic species (Human), we only identified the most common and simple repeats, not very specific to this species. If you compare with Red results, your are missing at least ~28% of repeated content in the genome. However, RepeatMasker gives interesting information about repeat classification which could be interesting for future analysis.

To boost RepeatMasker performance, we need a tailored repeat library for *Mucor mucedo*. This step can take from a few hours to a few days and a large number of tools could be used. We pre-computed two librairies:

- `Muco_library_RM2.fasta` using RepeatModeler ({% cite Flynn_2020 %})
- `Muco_library_EDTA.fasta` using EDTA ({% cite Ou_2019 %})

> <details-title>Preparing a library</details-title>
>
> You can prepare your own repeat library, specific to the genome you're using, by running RepeatModeler in Galaxy (it will take several hours to run):
>
> > <hands-on-title>Building a repeat library with RepeatModeler</hands-on-title>
> >
> > 1. {% tool [RepeatModeler](toolshed.g2.bx.psu.edu/repos/csbl/repeatmodeler/repeatmodeler/2.0.4+galaxy1) %} with the following parameters:
> >    - {% icon param-file %} *"Input genome fasta"*: `genome_raw.fasta` (Input dataset)
> {: .hands_on}
>
> The library is in the "consensus sequences" output dataset.
{: .details}

> <hands-on-title></hands-on-title>
>
> 1. {% tool [RepeatMasker](toolshed.g2.bx.psu.edu/repos/bgruening/repeat_masker/repeatmasker_wrapper/4.1.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Genomic DNA"*: `genome_raw.fasta` (Input dataset)
>    - *"Repeat library source"*: `Custom library of repeats`
>    - *"Custom library of repeats"*
>        - *"One of the two pre-computed libraires"*: `Muco_library_RM2.fasta` or `Muco_library_EDTA.fasta`
>    - *"Output annotation of repeats in GFF format"*: `Yes`
>    - *"Perform soft-masking instead of hard-masking"*: `Yes`
{: .hands_on}

> <question-title></question-title>
>
> Compare the different `repeat statistics` files produced, what is the highest library for RepeatMasker?
>
> > <solution-title></solution-title>
> >
> > The RepeatModeler library seems to have the highest percentage of repeats found with ~ 34.89%. It could be explained as RepeatModeler is specifically made to work with RepeatMasker.
> {: .solution}
>
{: .question}

Other tools might mask a greater proportion of the genome, at the cost of a more complex workflow with training steps. But masking more isn't always a positive results! In fact, large family of genes could be considered as a repeat by some tools or certain library. Only a manual curation can correct those mistakes, but this result is sufficient to perform an annotation by following the [Funannotate annotation tutorial]({% link topics/genome-annotation/tutorials/funannotate/tutorial.md %}).

# Conclusion



By following this tutorial you have learn how to mask an eukaryotic genome using Red and RepeatMasker, after assembling ([Flye assembly tutorial]({% link topics/assembly/tutorials/flye-assembly/tutorial.md %})) and before annotating it ([Funannotate annotation tutorial]({% link topics/genome-annotation/tutorials/funannotate/tutorial.md %})).

Often times, annotation tools prefer to use soft-masked genomes, as they primarily search for genes in non repeated regions, but tolerate that some genes overlap partially with these regions.


topic: genome-annotation
tutorial: bacterial-comparative-genomics-dataset-construction
title: Dataset construction for bacterial comparative genomics
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["microgalaxy"]
content_hash: a96e511f6b3a0a33777e7b9d8c1221f5
last_modified: 2026-02-26T21:12:16.192543
zenodo_link: https://zenodo.org/records/1
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/bacterial-comparative-genomics-dataset-construction/tutorial.html

content:
Introduction


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get genomes from GTDB

> <hands-on-title> Task description </hands-on-title>
>
> 1. 
>
{: .hands_on}



> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 3. {% tool [Import](upload1) %} the contig file from [Zenodo]({{ page.zenodo_link }}) or from Galaxy shared data libraries:
>
>    ```
>    {{ page.zenodo_link }}/files/DRR187559_contigs.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
{: .hands_on}



# Extract IDs for NCBI

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - *"Delimited by"*: `Comma`
>    - {% icon param-file %} *"From"*: `output` (Input dataset)
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}


> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c6`
>    - {% icon param-file %} *"From"*: `output` (Input dataset)
>
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}


> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Concatenate datasets](cat1) %} with the following parameters:
>    - {% icon param-file %} *"Concatenate Dataset"*: `out_file1` (output of **Cut** {% icon tool %})
>    - In *"Dataset"*:
>        - {% icon param-repeat %} *"Insert Dataset"*
>            - {% icon param-file %} *"Select"*: `out_file1` (output of **Cut** {% icon tool %})
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Sort](sort1) %} with the following parameters:
>    - {% icon param-file %} *"Sort Dataset"*: `out_file1` (output of **Concatenate datasets** {% icon tool %})
>    - *"on column"*: `c1`
> 1. {% tool [Unique](toolshed.g2.bx.psu.edu/repos/bgruening/unique/bg_uniq/0.3) %} with the following parameters:
>    - {% icon param-file %} *"from query"*: `out_file1` (output of **Sort** {% icon tool %})
>    - *"Advanced Options"*: `Hide Advanced Options`
>
>    ***TODO***: *Check parameter descriptions*
>
>    ***TODO***: *Consider adding a comment or tip box*
>
>    > <comment-title> short description </comment-title>
>    >
>    > A comment about the tool or something else. This box can also be in the main text
>    {: .comment}
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Remove beginning](Remove beginning1) %} with the following parameters:
>    - *"Remove first"*: `2`
>    - {% icon param-file %} *"from"*: `outfile` (output of **Unique** {% icon tool %})
>
>    ***TODO***: *Check parameter descriptions*
>
>    ***TODO***: *Consider adding a comment or tip box*
>
>    > <comment-title> short description </comment-title>
>    >
>    > A comment about the tool or something else. This box can also be in the main text
>    {: .comment}
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}

# Download genomes from NCBI

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [NCBI Datasets Genomes](toolshed.g2.bx.psu.edu/repos/iuc/ncbi_datasets/datasets_download_genome/16.42.0+galaxy0) %} with the following parameters:
>    - In *"Query"*:
>        - *"Choose how to find genomes to download"*: `By NCBI assembly or BioProject accession`
>            - *"Enter accession or read from file ?"*: `Read a list of NCBI Assembly accessions from a dataset`
>                - {% icon param-file %} *"Select dataset with list of NCBI Assembly accessions"*: `out_file1` (output of **Remove beginning** {% icon tool %})
>
>    ***TODO***: *Check parameter descriptions*
>
>    ***TODO***: *Consider adding a comment or tip box*
>
>    > <comment-title> short description </comment-title>
>    >
>    > A comment about the tool or something else. This box can also be in the main text
>    {: .comment}
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}


# Conclusion

Sum up the tutorial and the key takeaways here. We encourage adding an overview image of the
pipeline used.


topic: genome-annotation
tutorial: funannotate
title: Genome annotation with Funannotate
difficulty: intermediate
time_estimation: 8h
tools_json: []
requirements_json: []
tags_json: ["gmod", "eukaryote", "jbrowse1", "biodiversity"]
content_hash: c66586eb60be50d062218c1b9e729fb2
last_modified: 2026-02-26T21:12:16.593337
zenodo_link: https://zenodo.org/record/7867921
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/funannotate/tutorial.html

content:
Genome annotation of eukaryotes is a little more complicated than for prokaryotes: eukaryotic genomes are usually larger than prokaryotes, with more genes. The sequences determining the beginning and the end of a gene are generally less conserved than the prokaryotic ones. Many genes also contain introns, and the limits of these introns (acceptor and donor sites) are not highly conserved.

In this tutorial we will use a software tool called Funannotate ({% cite jonathan_m_palmer_2020_4054262 %}) to annotate the genome sequence of a small eukaryote: [*Mucor mucedo*](https://en.wikipedia.org/wiki/Mucor_mucedo) (a fungal plant pathogen).

As explained on [Funannotate's website](https://funannotate.readthedocs.io/), "it was originally written to annotate fungal genomes (small eukaryotes ~ 30 Mb genomes), but has evolved over time to accomodate larger genomes". As other annotation tools like [Maker]({% link topics/genome-annotation/tutorials/annotation-with-maker/tutorial.md %}) ({% cite Campbell2014 %}) or Braker ({% cite braker %}), it works by aligning as many evidences as possible along the genome sequence, and then reconciliating all these signals to determine probable gene structures.

The evidences can be transcript or protein sequences from the same (or closely related) organism. These sequences can come from public databases (like NR or GenBank) or from your own experimental data (transcriptome assembly from an RNASeq experiment for example). Funannotate is also able to take into account repeated elements.

Funannotate uses ab-initio predictors ([Augustus](http://bioinf.uni-greifswald.de/augustus/), [SNAP](https://github.com/KorfLab/SNAP), [glimmerHMM](https://bio.tools/glimmer-hmm), [CodingQuarry](https://bmcgenomics.biomedcentral.com/articles/10.1186/s12864-015-1344-4) and [GeneMark-ES/ET](http://exon.gatech.edu/GeneMark/) (optional due to licensing)) to improve its predictions: these software tools are able to make gene structure predictions by analysing only the genome sequence with a statistical model.

While for [Maker]({% link topics/genome-annotation/tutorials/annotation-with-maker/tutorial.md %}) you need to perform training steps for the ab-initio predictors, Funannotate is able to take care of that for you, which makes it much easier to use.

In this tutorial, you will learn how to perform a structural genome annotation, and how to evaluate its quality. Then, you will learn how to run functional annotation, using EggNOG-mapper and InterProScan to automatically assign names and functions to the annotated genes. And you will also learn how Funannotate can prepare files ready for submission of your annotation to the NCBI. Finally, you will learn how to use the [JBrowse](http://jbrowse.org/) genome browser to visualise your new annotation.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data upload

To annotate our genome using Funannotate, we will use the following files:

- The **genome sequence** in fasta format. For best results, the sequence should be soft-masked beforehand. You can learn how to do it by following the [RepeatMasker tutorial]({% link topics/genome-annotation/tutorials/repeatmasker/tutorial.md %}). For this tutorial, we will try to annotate the genome assembled in the [Flye assembly tutorial]({% link topics/assembly/tutorials/flye-assembly/tutorial.md %}) and already masked for you using RepeatMasker.
- Some RNASeq data in fastq format. We will align them on the genome, and Funannotate will use it as evidence to annotate genes.
- A set of **protein sequences**, like UniProt/SwissProt. It is important to have good quality, curated sequences here, that's why, by default, Funannotate will use the UniProt/SwissProt databank. In this tutorial, we have prepared a subset of this databank to speed up computing, but you should use UniProt/SwissProt for real life analysis.

 Funannotate will take into account the position of mapped RNASeq reads, and the alignment of protein sequences on the genome sequence to determine gene positions.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/record/7867921/files/genome_masked.fasta
>    https://zenodo.org/record/7867921/files/rnaseq_R1.fq.gz
>    https://zenodo.org/record/7867921/files/rnaseq_R2.fq.gz
>    https://zenodo.org/record/7867921/files/SwissProt_subset.fasta
>    https://zenodo.org/record/7867921/files/alternate_annotation.gbk
>    https://zenodo.org/record/7867921/files/alternate_annotation.gff3
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

# Preparing RNASeq data

When you sequence a new genome, you usually sequence a few libraries of RNASeq data, from different tissues and in different conditions, because this data will help you in annotating the genome. Here, we are using data from one RNASeq dataset that is available on [Sequence Read Archive (SRA)](https://www.ncbi.nlm.nih.gov/sra): [SRR8534859](https://www.ncbi.nlm.nih.gov/sra/?term=SRR8534859).

You would normally get the Fastq files directly from SRA and use them in the following step. To speed up the tutorial (without impairing too much the quality of the results), we have reduced the size of the dataset into a single pair of (smaller) fastq files, available from Zenodo (or the GTN Data Libraries).

To make use of this RNASeq data, we need to map it on the genome. We will use **RNA star** {% icon tool %} and get a result in the form of a BAM file, that we will use in the rest of the tutorial.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [RNA STAR](toolshed.g2.bx.psu.edu/repos/iuc/rgrnastar/rna_star/2.7.8a+galaxy0) %} with the following parameters:
>    - *"Single-end or paired-end reads"*: `Paired-end (as individual datasets)`
>        - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file, forward reads"*: `rnaseq_R1.fq.gz` (Input dataset)
>        - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file, reverse reads"*: `rnaseq_R2.fq.gz` (Input dataset)
>    - *"Custom or built-in reference genome"*: `Use reference genome from history and create temporary index`
>        - {% icon param-file %} *"Select a reference genome"*: `genome_masked.fasta` (Input dataset)
>        - *"Length of the SA pre-indexing string"*: `11`
>
{: .hands_on}

We select `11` for the *"Length of the SA pre-indexing string"* parameter as it is the recommended value for a small genome of this size. If you select 14 (the default value), STAR will advise you to use `11` instead in its logs.

> <comment-title>What if I want to use multiple RNASeq libraries</comment-title>
>
> To get the best possible annotation, it is adviced to use multiple RNASeq libraries, sequences from different tissues in different conditions. To use them, you can map each one individually using STAR, just as in this tutorial. You will get one BAM file per RNASeq library, and you can then easily merge them into a single BAM file by using the {% tool [MergeSamFiles](toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_MergeSamFiles/2.18.2.1) %} tool. This single BAM file can then be used by Funannotate like we do in the next steps.
{: .comment}

Before we move on to the next step, we need to make sure that the mapping went well. Have a look at the `log` output of `RNA STAR`.

> <question-title></question-title>
>
> What proportion of reads were correctly mapped to the genome? Do you think it is enough to continue with this tutorial?
>
> > <solution-title></solution-title>
> >
> > Look for `Uniquely mapped reads %`: you should get ~96%. This is a very good score, because while reducing the size of the dataset for this tutorial, we have kept mostly reads properly mapping.
> > Anyway, with real data, you expect to have a vast majority of reads mapping uniquely on the genome. If it's not the case, check that you're using the correct RNASeq files, with the correct genome sequence.
> >
> {: .solution}
>
{: .question}

# Structural annotation

We can now run **Funannotate predict annotation** to perform the structural annotation of the genome.

We need to input the genome sequence, the mapped RNASeq data, and the proteins to align on the genome. We also specify the name of the species and strain (they can be used later for submission to NCBI).

There are other parameters to finely tune how Funannotate will run ab-initio predictors to predict genes, and to filter the final results based on various criteria. As Funannotate uses [BUSCO](http://busco.ezlab.org/) (Benchmarking Universal Single-Copy Orthologs) for initial training of ab-initio predictors, we select datasets close to the species we are annotating: `mucorales (orthodb 10)` and `rhizopus_oryzae`.

Funannotate is also able to use GeneMark to predict new genes, but to due to licensing restrictions, this software is not available on every Galaxy instance. We will ignore this for this tutorial, it will not impact the results too much.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [Funannotate predict annotation](toolshed.g2.bx.psu.edu/repos/iuc/funannotate_predict/funannotate_predict/1.8.15+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Assembly to annotate"*: `genome_masked.fasta` (Input dataset)
>    - *"Funannotate database"*: select the latest version available
>    - In *"Organism"*:
>        - *"Name of the species to annotate"*: `Mucor mucedo`
>        - *"Strain name"*: `muc1`
>        - *"Is it a fungus species?"*: `No`
>    - In *"Evidences"*:
>        - {% icon param-file %} *"RNA-seq mapped to genome to train Augustus/GeneMark-ET"*: `mapped.bam` (output of **RNA STAR** {% icon tool %})
>        - *"Select protein evidences"*: `Custom protein sequences`
>            - {% icon param-file %} *"Proteins to map to genome"*: `SwissProt_subset.fasta` (Input dataset)
>    - In *"Busco"*:
>        - *BUSCO models to align"*": `mucorales (orthodb 10)`
>        - *"Initial Augustus species training set for BUSCO alignment"*: `rhizopus_oryzae`
>    - *"Which outputs should be generated"*: Select all
>
{: .hands_on}

> <comment-title>on parameters</comment-title>
>
> - For *"Select protein evidences"* we select `Custom protein sequences` to reduce the computing time, but for real data analysis, you should select the default value: `Use UniProtKb/SwissProt (from selected Funannotate database)`.
> - It is possible to enable the *"Is it a fungus species?"* option in Funannotate: it launches an additional ab initio predictor (CodingQuerry) dedicated to fungi genomes. However it has proved to be unstable on the genome studied in this tutorial, and it can create a lot of fragmented gene models depending on the RNASeq data available. For this tutorial we leave this option to `No`. You can test it with real data, but be sure to compare the result with and without this option.
> - For real data analysis you can consider enabling the *"Augustus settings (advanced)"* > *"Run 'optimize_augustus.pl' to refine training (long runtime)"*. If you have enough data, you might get better results as there will be an additional training step for augustus (at the cost of a longer runtime).
{: .comment}

> <comment-title>Don't wait</comment-title>
>
> This step will take a bit of time to run. While it runs, we can already schedule the following functional annotation steps. Galaxy will run them automatically as soon as the structural annotation is ready.
{: .comment}

This tool produces several output dataset, in particular:

- the full structural annotation in Genbank, GFF3 or NCBI tbl formats: these files contain the position of all the genes that were found on the genome.
- the CDS, transcript and protein sequences of all the genes predicted by Funannotate (fasta files)
- some statistics and reports

Let's have a closer look at the output of our annotation. First display the `stats` dataset: the first par of the file contains some information on how funannotate was launched. If you go to the bottom, you'll find a few interesting numbers in the `annotation` section:

- the total number of genes and mRNA
- the average length of genes, exons, proteins
- the number of single/multiple exon transcripts

These number alone are interesting, but not fully informative on the quality of the annotation. For example, for the number of genes, you want a 'good' number based on what you expect for this species (a too big number can mean that genes are fragmented, and a too small number can mean that some genes were not annotated at all). These numbers can help when comparing an annotation with other ones performed with other parameters or tools. You can also compare these numbers with annotation of closely related species.

To get a better picture of the quality of the result, we will run BUSCO in the next step.

Before moving on, have a quick look at the `tbl2asn error summary report` output: it lists a few potential problems that were identified by Funannotate in the results it generated. For example, Funannotate can tell you when it predicted genes that contain very short exons, or that use a rare splice site sequence. You can have a detailed list of identified potential problems in the `tbl2asn genome validation report` dataset. It does not mean that each listed gene is wrong, but it means that you might want to give a closer look at these ones. If you have time to manually check each gene, Apollo can help you in doing this, see the note in the conclusion for this.

{% include topics/genome-annotation/tutorials/functional/content.md short=false %}

# Submission to NCBI

If you plan to submit the final genome sequence and annotation to NCBI, there are a few steps to follow.

> <warning-title>Please do not submit this genome to NCBI!</warning-title>
> In this tutorial we will **not** perform a real submission, as we don't want to duplicate the annotation in Genbank every time someone follows this tutorial!
>
> But here's a description of the main steps, and how Funannotate can help you in this process. NCBI provides a complete documentation for [genome and annotation submission](https://www.ncbi.nlm.nih.gov/genbank/genomesubmit/).
{: .warning}

- First, you should have created a BioProject and a BioSample [on the NCBI portal](https://submit.ncbi.nlm.nih.gov/subs/bioproject/), corresponding to your scientific project, and the sample(s) you have sequenced
- The raw reads used for the assembly, and the RNASeq ones should be deposited on SRA, and taggued with the BioProject and BioSample ids
- Ideally you should start submitting the assembly *before* performing the annotation. The reason is simple: NCBI performs some validation on the genome sequence before accepting it into GenBank. It means that you may be forced to make modifications to the genome sequence (e.g. remove contigs, split contigs where you have adapter contamination, etc). You will save time and trouble doing the annotation on a fully validated and frozen genome sequence.
- You should get a `locus_tag` for your genome: it is the unique prefix that is used at the beginning of each gene name. By default, Funannotate uses `FUN_` (e.g. `FUN_000001`), but to submit to NCBI you need to have a prefix specific to your genome. NCBI should provide it to you when you create your BioProject.
- You then need to prepare a file containing a few metadata. This can be done online on [https://submit.ncbi.nlm.nih.gov/genbank/template/submission/](https://submit.ncbi.nlm.nih.gov/genbank/template/submission/). You need to fill the form with some basic information, like that for example (for real data, you should of course write real information!):

![NCBI submission template](../../images/ncbi_template.png "Example of a filled form to generate an NCBI submission template.")

When filled, you can click on the **Create template** button: you will be able to download a `.sbt` file that you should then upload to your Galaxy history.

The next step will show you how Funannotate can use this `.sbt` file, and the `locus_tag`, to generate the needed file for the submission of the annotation to the NCBI. These files will need to be submitted on the [NCBI genome submission portal](https://submit.ncbi.nlm.nih.gov/subs/genome/).

# Integrating structural and functional annotation

Now we have a structural annotation, and the results of both **EggNOG Mapper** and **InterProScan**. Each one is in a separate file, so we will now combine all this data into a single file that will contain both the structural *and* the functional annotation. This will be the final output of our annotation pipeline, ready to be submitted to the NCBI reference database.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [Funannotate functional](toolshed.g2.bx.psu.edu/repos/iuc/funannotate_annotate/funannotate_annotate/1.8.15+galaxy1) %} with the following parameters:
>    - *"Input format"*: `GenBank (from 'Funannotate predict annotation' tool)`
>        - {% icon param-file %} *"Genome annotation in genbank format"*: `annotation (genbank)` (output of **Funannotate predict annotation** {% icon tool %})
>    - *"Funannotate database"*: select the latest version available
>    - {% icon param-file %} *"NCBI submission template file"*: `template.sbt` (the file downloaded from NCBI website, or leave empty if you didn't generate it)
>    - {% icon param-file %} *"Eggnog-mapper annotations file"*: `annotations` (output of **eggNOG Mapper** {% icon tool %})
>    - {% icon param-file %} *"InterProScan5 XML file"*: `InterProScan XML` (output of **InterProScan** {% icon tool %})
>    - {% icon param-file %} *"BUSCO models"*: `mucorales (orthodb 10)`
>    - *"Strain name"*: `muc1`
>    - *"locus_tag from NCBI to rename GFF gene models with"*: `MMUCEDO_` (we consider NCBI has assigned us this `locus_tag`)
>    - *"Which outputs should be generated"*: Select all
>
{: .hands_on}

This tool produces several output dataset, in particular:

- The full structural and functional annotation in Genbank and GFF3 format
- NCBI tbl and sqn (`NCBI Sequin genome`) formats: these files can be used for submission to NCBI
- The CDS, transcript and protein sequences of all the genes predicted by Funannotate
- Some statistics and reports

If you display the GFF3 output, you will notice that the functional information, including gene names, is now stored in this file.

## Evaluation with **Busco**

We now have a complete annotation, including functional annotation, but it's time to evaluate the quality of this annotation. [BUSCO](http://busco.ezlab.org/) (Benchmarking Universal Single-Copy Orthologs) is a tool allowing to evaluate the quality of a genome assembly or of a genome annotation. By comparing genomes from various more or less related species, the authors determined sets of ortholog genes that are present in single copy in (almost) all the species of a clade (Bacteria, Fungi, Plants, Insects, Mammalians, ...). Most of these genes are essential for the organism to live, and are expected to be found in any newly sequenced and annotated genome from the corresponding clade. Using this data, BUSCO is able to evaluate the proportion of these essential genes (also named BUSCOs) found in a set of (predicted) transcript or protein sequences. This is a good evaluation of the "completeness" of the annotation.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.4.6+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Sequences to analyse"*: `protein sequences` (output of **Funannotate functional** {% icon tool %})
>    - *"Mode"*: `annotated gene sets (protein)`
>    - *"Auto-detect or select lineage?"*: `Select lineage`
>        - *"Lineage"*: `Mucorales`
>    - In *"Advanced Options"*:
>        - *"Which outputs should be generated"*: `shortsummary text` and `summary image`
>
{: .hands_on}

> <question-title></question-title>
>
> How many BUSCO genes were found complete in the annotation? Do you think the quality of the annotation is good?
>
> > <solution-title></solution-title>
> >
> > On a total of 2449, you should find ~2312 BUSCO genes identifed as complete in the annotation, with 2281 being in single copy, and 31 being duplicated.
> >
> > That's a quite good result as running BUSCO on the genome itself gives a very close number (2327 Complete BUSCOs) (see Flye assembly tutorial). It means the annotation process was able to detect most of the genes it was supposed to find.
> >
> > To improve the result you can consider using more RNASeq data, and using the *"Augustus settings (advanced)"* > *"Run 'optimize_augustus.pl' to refine training (long runtime)"* option.
> {: .solution}
>
{: .question}

# Visualisation with a genome browser

With Galaxy, you can visualize the annotation you have generated using JBrowse genome browser. This allows you to navigate along the chromosomes of the genome and see the structure of each predicted gene. We also add an RNASeq track, using the BAM file created with **RNA STAR** {% icon tool %}.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: `genome_masked.fasta` (Input dataset)
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Annotation`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `gff3` (output of **Funannotate functional** {% icon tool %})
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `RNASeq`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BAM Pileups`
>                        - {% icon param-file %} *"BAM Track Data"*: `mapped.bam` (output of **RNA STAR** {% icon tool %})
>                        - *"Autogenerate SNP Track"*: `Yes`
>
{: .hands_on}

Click on the newly created dataset's eye to display it. You will see a JBrowse genome browser. You can have a look at the [JBrowse tutorial]({% link topics/visualisation/tutorials/jbrowse/tutorial.md %}) for a more in-depth description of JBrowse.

Enable the annotation track, and the SNPs/Coverage tracks on the left side of JBrowse, then navigate along the genome. You will see the different gene models predicted by Funannotate.

If you click on a gene, a popup will appear with detailed information on the selected gene: position and sequence, but also the functional information retrieved from the **EggNOG mapper** and **InterProScan** results.

If you zoom to a specific gene, and look at the RNASeq tracks, you will see light grey regions corresponding to portions of the genomes where RNASeq were mapped, and darker grey regions corresponding to introns (= regions were some reads were found to match both the end of an exon, and the start of the next one). You can enable the other RNASeq track to display each individual read that was mapped on the genome.

If you navigate along the genome, you will find genes with very low RNASeq coverage: this demonstrates how Funannotate is able to predict genes not only with RNASeq, but also by comparing to protein sequences (from SwissProt) and using ab initio predictors, trained using RNASeq data.

# Comparing annotations

Earlier, we have seen how general statistics and BUSCO results can help to evaluate the quality of an annotation. But when annnotating a new genome, you might want to try different annotation methods and parameters. We will see now how to compare multiple annotations between them. As an example, we will compare the annotation we have generated, with an alternate one, that you have uploaded from Zenodo at the beginning of the tutorial. Comparison is only possible for annotations performed on the same genome sequence.

## Comparing with **AEGeAn ParsEval**

Let's run **AEGeAn ParsEval** first: it compares two annotations in GFF3 format.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [AEGeAn ParsEval](toolshed.g2.bx.psu.edu/repos/iuc/aegean_parseval/aegean_parseval/0.16.0) %} with the following parameters:
>    - {% icon param-file %} *"Reference GFF3 file"*: `gff3` (output of **Funannotate functional** {% icon tool %})
>    - {% icon param-file %} *"Prediction GFF3 file"*: `alternate_annotation.gff3` (Input dataset)
>    - *"Select the output type"*: `HTML`
>
{: .hands_on}

**AEGeAn ParsEval** {% icon tool %} compares gene loci spread all along the genome sequence, each locus containing gene(s) on the "reference" (=the annotation we have generated) and/or the "prediction" (=the alternate one) annotation.

The output is a web page where you can see how many loci or genes are identical or different between the two annotations. By clicking on the `(+)` links, or on the scaffold names, you can see the results more in detail, even at the gene level.

> <hands-on-title></hands-on-title>
>
> 1. Display the report
> 2. Click on `scaffold_11`
> 3. Click on the first locus (between positions 637 and 7730)
>
{: .hands_on}

> <question-title></question-title>
>
> What can you tell from the comparison at this gene locus?
>
> > <solution-title></solution-title>
> >
> > You should see something like this:
> >
> > ![Scaffold 11 - 637..7730](../../images/aegean_scaffold_11_637-7730.png "Comparison of the two annotations on Scaffold 11 - 637..7730")
> >
> > As you can see, at this position, the alternate annotation as found only one big gene, while the annotation you have generated has found 4 different genes, not all on the same strand.
> >
> > This can be explained by the way the alternate annotation was performed: it was done by running Funannotate without any RNASeq data, and choosing a wrong value (`insecta`) in the busco parameter. The annotation you have generated is probably the good one here.
> {: .solution}
>
{: .question}

## Comparing with **Funannotate compare**

**Funannotate compare** {% icon tool %} is another tool to compare several annotations. It uses genbank files, as generated by **Funannotate functional** {% icon tool %}.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [Funannotate compare](toolshed.g2.bx.psu.edu/repos/iuc/funannotate_compare/funannotate_compare/1.8.15+galaxy1) %} with the following parameters:
>    - {% icon param-files %} *"Genome annotations in genbank format"*: `alternate_annotation.gbk` (Input dataset) and `gbk` (output of **Funannotate functional** {% icon tool %})
>    - *"Funannotate database"*: select the latest version available
>
{: .hands_on}

The output is a web page with different tabs:

- `Stats`: general statistics on the two annotations.
- `Orthologs`: orthology relations between genes of the two annotations.
- `InterPro` and `PFAM`: count the number of times each InterPro/PFAM signature is identified in each annotation. You can download a PDF file at the top, showing an {NMDS} analysis of this data.
- `Merops` and `CAZymes`: count the number of members of each Merops/CAZymes family in each annotation. You can download PDF files at the top, showing colorful representations of this data.
- `GO`: list Gene Ontology terms that were found to be over or under represented in each annotation.

> <question-title></question-title>
>
> What can you tell from this report?
>
> > <solution-title></solution-title>
> >
> > The alternate annotation contains less genes than the one you generated, and misses a lot of functional annotations. It confirms that this alternate annotation have a lower quality than the one you performed.
> {: .solution}
>
{: .question}

# Conclusion


Congratulations for reaching the end of this tutorial! Now you know how to perform a structural and functional annotation of a new eukaryotic genome, using Funannotate, EggNOG mapper and InterProScan. You also learned how Funannotate can help you in the submission process to NCBI. And you learned how to visualise your new annotation using JBrowse, and how to compare it with another annotation.

An automatic annotation of an eukaryotic genome is unfortunately rarely perfect. If you inspect some predicted genes (or look at the `tbl2asn genome validation report` output of Funannotate), you may find some mistakes made by Funannotate, or potential problems, e.g. wrong exon/intron limits, splitted genes, or merged genes. Setting up a manual curation project using [Apollo](http://genomearchitect.org/) can help a lot to manually fix these errors. Check out the [Apollo tutorial]({% link topics/genome-annotation/tutorials/apollo-euk/tutorial.md %}) for more details.


topic: genome-annotation
tutorial: gene-centric
title: Comparative gene analysis in unannotated genomes
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: []
content_hash: 0ba8180122a89b5b15c814e626fa7f29
last_modified: 2026-02-26T21:12:16.595733
zenodo_link: https://zenodo.org/record/7034885
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/gene-centric/tutorial.html

content:
Despite the rapidly increasing number of fully assembled genomes few genomes are well annotated. This is especially true for large eukaryotic genomes with their complex gene structure and abundance of pseudogenes. And of course do not forget about the [Murthy's law](https://en.wikipedia.org/wiki/Murphy%27s_law): if you are interested in a particular gene the chances are that it will not be annotated in your genome of interest. In this tutorial we will demonstrate how to compare gene structures across a set of vertebrate genomes. So ...

> > <code-in-title>What I have</code-in-title>
> > - I know the gene's name
> > - I know which species I'm interested in
> > - I know where to find genomes of these species
> {: .code-in}
>
> > <code-out-title>What I want</code-out-title>
> > - I work with a gene _X_
> > - I would like to compare the structure of gene _X_ across _N_ genomes
> {: .code-out}
>
{: .code-2col}

> <tip-title>What I will get</tip-title>
> - Interactive graphs showing location of the gene across your species of choice. These will allow you to see the absence/presence of the genes across genomes, to detect potential duplications, predogenization events, re-arrangements etc.
> - Phylogenetic trees for individual exons of the gene. The trees will give you an idea of potential unusual evolutionary dynamics for the gene.
{: .tip}

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Logic

The analysis follow the following logic (also see the following figure):

1. Pick a gene and select genomes to analyze
1. Get amino acid translations for all exons of my gene of interest
1. Identify genomes of interest
1. Extract amino acid sequences and genome coordinates for all possible open reading frames (ORFs) from all genomes in my set

    The following steps 5, 6, and 7 are performed using a single Galaxy workflow (grey area in the following figure).

1. Align - Find matches between exons of the gene of interest and ORFs
1. Intersect - Compute genome coordinates of matches
1. Build phylogenetic trees for individual exons

    The final step is performed using Jupyter notebook

1. Create a comparative plot showing distribution of matches across genomes of interest

![A schematic diagram of the above workflow](../../images/gene-centric/gene_analysis.svg)

------

# Example history

This [example history](https://usegalaxy.org/u/cartman/h/genecomparisonxbp1) contains results of the analysis described in this tutorial.

{% snippet faqs/galaxy/histories_import.md %}

You can use this history to understand the input datasets as well as outputs of the entire analysis. The key items in the history are labelled with <kbd>tags</kbd>:

> <code-in-title>dataset in the example history</code-in-title>
> - <kbd>EXONS</kbd> - amino acid translation of exons of the gene of interest (*XBP-1*)
> - <kbd>ORF_BED</kbd> - coordinates of predicted ORFs in the genomes of interest
> - <kbd>DiamondDB</kbd> - database and amino acid translations of predicted ORFs in the genomes of interest
{: .code-in}

> <code-out-title>in the sample history</code-out-title>
> - <kbd>PlottingData</kbd> - summary necessary for plotting comparative genome graphs
> - <kbd>Trees</kbd> - phylogenetic trees for each exon
{: .code-out}

> <warning-title>A suggestion!</warning-title>
> Importing and looking around this history is very helpful for understanding how this analysis works!
{: .warning}

-------

# Practice

## Step 1: Pick a gene and select genomes to analyze

In this example we will compare structure of X-box protein 1 gene (*XBP-1*) across a set of five vertebrate genomes sequenced by [VGP](https://vertebrategenomesproject.org/) consortium ({% cite Rhie2021-ks %}).

## Step 2: Get amino acid translations for all exons of my gene of interest

This step is manual and is performed outside Galaxy.

The best annotated vertebrate is ... human. To obtain amino acid translation of individual exons of *XBP-1* you can use [UCSC Genome browser](https://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&lastVirtModeType=default&lastVirtModeExtraState=&virtModeType=default&virtMode=0&nonVirtPosition=&position=chr22%3A28794555%2D28800597&hgsid=1433476153_N1Iet0RvAaPQIpPuVl2qbi74BrC3). The "UCSC annotations of RefSeq RNAs (NM_\* and NR_\*)" track shows two alternative transcripts produced by *XBP-1* : spliced and unspliced (here 'spliced' and 'unspliced' do not refer to the "normal' splicing process. To learn more about this fascinating gene read here).

![XBP-1](../../images/gene-centric/xbp-1.svg)

Clicking on the names of the transcripts open a UCSC page specific to that transcript. From there you can obtain genomic sequences corresponding to all exons. You can then translate these sequences using any available online tool (e.g., [NCBI ORFinder](https://www.ncbi.nlm.nih.gov/orffinder/)).

In this particular case we did not create translation of all exons. Instead, we created translation of exon 1  and two terminal exons. The first exon is shared between the two transcripts. It will allow us to anchor the beginning of the gene. The two terminal exons are different between the two transcripts. Because of this we create two alternative translation per exon: `s-p2` and `s-p12` for the spliced version and `u-p1` and `u-p2` for unspliced transcript. A FASTA file containing all translation is shown below:

```
>xbp-1u-p1
GNEVRPVAGSAESAALRLRAPLQQVQAQLSPLQNISPWILAVLTLQIQ
>xbp-1u-p2
LISCWAFWTTWTQSCSSNALPQSLPAWRSSQRSTQKDPVPYQPPFLCQWG
RHQPSWKPLMN
>xbp-1s-p12
GNEVRPVAGSAESAAGAGPVVTPPEHLPMDSGGIDSSDSE
>xbp-1s-p3
SDILLGILDNLDPVMFFKCPSPEPASLEELPEVYPEGPSSLPASLSLSVG
TSSAKLEAINELIRFDHIYTKPLVLEIPSETESQANVVVKIEEAPLSPSE
NDHPEFIVSVKEEPVEDDLVPELGISNLLSSSHCPKPSSCLLDAYSDCGY
GGSLSPFSDMSSLLGVNHSWEDTFANELFPQLISV
>xbp1_ex1
MVVVAAAPNPADGTPKVLLLSGQPASAAGAPAGQALPLMVPAQRGASPEA
ASGGLPQARKRQRLTHLSPEEKALR
```

## Step 3: Identify and upload genomes of interest

This tutorial has been initially designed for the analysis of data produced by the Vertebrate Genome Consortium [VGP](https://vertebrategenomesproject.org/). However, it is equally suitable for any genomic sequences (including prokaryotic ones).

In this section we first show how to upload sample datasets. These datasets were intentionally made small. All subsequent steps of this tutorial are performed using these sample data. We then demonstrate how to upload full size genomes from the VGP data repository called [GenomeArk](https://vgp.github.io/genomeark/).

### Uploading sample data

> <hands-on-title>Sample Data upload</hands-on-title>
>
> - Create a new history for this tutorial
> - Import the files from [Zenodo]({{ page.zenodo_link }}) using the following URLs:
>
>   ```
>   https://zenodo.org/record/7034885/files/aGasCar1.fa
>   https://zenodo.org/record/7034885/files/bTaeGut2.fa
>   https://zenodo.org/record/7034885/files/fScoJap1.fa
>   https://zenodo.org/record/7034885/files/mCynVol1.fa
>   https://zenodo.org/record/7034885/files/mHomSapT2T.fa
>   ```
>   {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}

These correspond to fragments of genomes from:

 - `aGasCar1` - _Gastrophryne carolinensis_ (Eastern narrow-mouth toad)
 - `bTaeGut` - _Taeniopygia guttata_ (Zebra finch)
 - `fScoJap` - _Scomber japonicus_ (Chub mackerel)
 - `mCynVol` - _Cynocephalus volans_ (Philippine flying lemur)
 - `mHomSap` - _Homo sapiens_ (Human)

### Uploading VGP data from GenomeArk

Galaxy provides a direct connection to [GenomeArk](https://vgp.github.io/genomeark/) from its "Upload Data" tool. To access GenomeArk data you need to:

> <hands-on-title>Upload VGP data from GenomeArk</hands-on-title>
>
> - Create a new history for this tutorial
> - Import genome assembly FASTA files from GenomeArk:
>
>    - Open the file {% icon galaxy-upload %} __Upload Data__ menu
>    - Click on __Choose remote files__ button at the bottom of the Upload interface
>    - Choose __Genome Ark__
>    - Pick assembles you would like to include in your analysis (use Search box on top)
>
>
> Generally for a given species you want the assembly with the highest version number. For example, Human (_Homo sapiens_) had three assemblies at the time of writing: `mHomSap1`, `mHomSap2`, and `mHomSap3`. Pick `mHomSap3` in this situation.
>
> Inside each assembly folder you will see typically see different sub-folders. The one you should generally be interested in will be called `assembly_curated`. Inside that folder you will typically see separate assemblies for maternal (indicated with `mat`) and paternal (indicated with `pat`) haplotypes. In the analyses here we are choosing as assembly representing heterogametic haplotype (`pat` in the case of Human).
>
> - Repeat this process for all assemblies you are interested in.
>
{: .hands_on}

### Combine uploaded sequences into a dataset collection

After uploading data from GenomeArk you will end up with a number of FASTA datasets in your history. To proceed with the analysis you will need to combine these into a single dataset collection:

> <hands-on-title>Combine genome assemblies into a dataset collection</hands-on-title>
>
> To create a collection from genome assemblies you just uploaded into GenomeArk follow the steps in the video below. Obviously, you only want to click check-boxes next to FASTA files (assemblies) you want to include and leave everything else out.
>
> {% snippet faqs/galaxy/collections_build_list.md %}
>
{: .hands_on}

## Step 4: Extract amino acid sequences and genome coordinates for all ORFs

### Extract ORFs from genomic sequences

Now we can go ahead and find all ORFs in all genome assemblies we have bundled into a collection. For this we will use tool called [ORFiPy](https://github.com/urmi-21/orfipy):

> <hands-on-title>Find ORFs witrh ORFiPy</hands-on-title>
>
> Run {% tool [ORFiPy](toolshed.g2.bx.psu.edu/repos/iuc/orfipy/orfipy/0.0.4+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Find ORFs in"*: The collection containing genome assemblies we just created (click "Dataset collection" {% icon param-collection %} button on left-side of this input field)
>    - *"Select options"*: check the box next to `BED` and `Peptides`
>    - *"Minimum length of ORFs"*: type the number `99`. We do not want ORFs to be too short as their number will be too high
>    - *"Output ORFs bound by Stop codons?"*: set this option to `Yes`.
>
{: .hands_on}

Because this operation is performed on a dataset collection. it will produce a dataset collection as an output. Actually, it will produce two because we selected `BED` and `Peptides` options. One collection will contain amino acid sequences for all ORFs (it will typically be called `ORFs on collection ... (FASTA format)`) and another their coordinates (called `ORFs on collection ... (BED format)`)
)

This will produce two new dataset collections in your history: one containing coordinates of ORFs and the other containing their amino acid translations. Because genomes are large these will likely be tens of millions of ORFs identified this way.

### Creating __Diamond__ database

Because we will be using the [Diamond](https://github.com/bbuchfink/diamond) tool to find matches between our gene of interest and ORF translations we need to convert FASTA files into Diamond database using __Diamond makedb__ tool:

> <hands-on-title>Create Diamond database</hands-on-title>
>
> Run {% tool [Diamond makedb](toolshed.g2.bx.psu.edu/repos/bgruening/diamond/bg_diamond_makedb/2.0.15+galaxy0) %} on the collection containing amino acid translations of ORFs generated using __ORFiPy__:
>    - {% icon param-collection %} "Input reference file in FASTA format"*: Select collections containing amino acid (FASTA Protein) output of __ORFiPy__
>
{: .hands_on}

At this point we have three input datasets that would allow us to find and visualize matches between the gene of interest and genome asseblies:

1. Diamond database of ORF translations from genome assemblies
2. Coordinates and frame information about ORFs in BED format
3. Amino acid translation of exons from the gene of interest

## Steps 5, 6, and 7: Finding matches and building trees

To find location of genes, [we will use the following workflow]({% link topics/genome-annotation/tutorials/gene-centric/workflows/index.md %}) that is available as a part of this tutorial. To use this workflow you need to import it into your Galaxy instance.

![WF](../../images/gene-centric/wf.png)

The workflow takes three inputs:

> <code-in-title>Workflow inputs</code-in-title>
>
> 1. <kbd>EXONS</kbd> - Amino acid translation of exons from the gene of interest ([Step 2](#step-2-get-amino-acid-translations-for-all-exons-of-my-gene-of-interest) of this tutorial)
> 1. <kbd>DiamondDB</kbd> - Diamond database of ORF translations from genome assemblies ([Step 4](#step-4-extract-amino-acid-sequences-and-genome-coordinates-for-all-orfs) of this tutorial)
> 1. <kbd>ORF BED</kbd> - Coordinates and frame information about ORFs in BED format ([Step 4](#step-4-extract-amino-acid-sequences-and-genome-coordinates-for-all-orfs)  of this tutorial)
>
{: .code-in}

It produces two primary outputs:

> <code-out-title>Results</code-out-title>
>
> 1. <kbd>Trees</kbd> - Phylogenetic trees for each input exon as [Newick](https://en.wikipedia.org/wiki/Newick_format) file
> 1. <kbd>PlottingData</kbd> - A summary table of exon matches for each genome
>
{: .code-out}

The overall logic of the workflow is as follows:

1. Perform [__Diamond__](https://github.com/bbuchfink/diamond) search to identify matches between the gene of interest (<kbd>EXONS</kbd>) and ORF sets from each genomes (<kbd>DiamondDB</kbd>).
2. Intersect information about the matches with BED file containing ORF coordinates (<kbd>ORF BED</kbd>). This allows us to know genomic position of the ORFs and their frames (1, 2, 3 or -1, -2, -3).
3. Extract matching parts of the sequences and generated multiple alignments using [__MAFFT__](https://mafft.cbrc.jp/alignment/software/). This is done for each amino acid sequence in <kbd>EXONS</kbd>. Thus in the case of *XBP-1* there will be five sets of alignments.
4. Build phylogenetic tree for each alignment produced in the previous step using the [Neighbor Joining method](https://en.wikipedia.org/wiki/Neighbor_joining) - a simple and quick way to obtain a rough phylogeny.
5. Use information from step 1 to compute genome coordinates of matches. This is done by combining the information about genomic positions of ORFs from the <kbd>ORF BED</kbd> file and local alignment information generated during step 1.

## Step 7: Looking at the trees

After running the workflow phylogenetic trees will be saved into a collection named `Join neighbors on ...`. This collection will also be labelled with tag <kbd>Trees</kbd>. To visualize the trees:

> <hands-on-title>Visualize the trees</hands-on-title>
>
> - Expand the collection by clicking on it
> - Click on any dataset
> - Click the {% icon galaxy-barchart %} icon:
>     ![Graph icon](../../images/gene-centric/display_tree.png)
> - A selection of tree visualization tools will appear in the center pane of the interface
> - Select `Phylogenetic Tre Visualization` and play with it:
>     ![The tree](../../images/gene-centric/tree.png)
{: .hands_on}

## Step 8: Generating and interpreting the comparative genome graph

Another workflow output will represent a single file summarizing genomic location of matches between each of the genomes in our dataset and amino acid translation of exons from the gene of interest. It will be called `Mapping report` and will have tag <kbd>PlottingData</kbd> associated with it. To plot the data contained in this file we will use external Jupyter notebook (note that Jupyter can be run directly from Galaxy, but to make this tutorial runnable on any Galaxy instance we will use an internal notebook server).

### Starting notebook

{% include _includes/cyoa-choices.html option1="UseGalaxy.org" option2="Galaxy Europe, Australia, and Others" default="UseGalaxy.org"
       text="If you are on a server with Interactive Tool support (e.g. Galaxy Europe, Australia, and many others), you can choose that option for a more integrated Jupyter Notebook experience." %}


<div class="UseGalaxyorg" markdown="1">

The notebook can be accessed [from here](https://colab.research.google.com/drive/1smTpRejBb7c02LiIxMPNVDjOucLAsD81?usp=sharing). You do need to have a Google Account to be able to use the notebook.

</div>
<div class="Galaxy-Europe-Australia-and-Others" markdown="1">

{% snippet faqs/galaxy/interactive_tools_jupyter_launch.md %}
{% capture ipynbpath %}{{ site.url }}{{ site.baseurl }}/{{ page.path | replace: "tutorial.md", "gene_image.ipynb" }}{% endcapture %}

1. Once the notebook has started, open a Terminal in Jupyter **File → New → Terminal**

2. Run the following command:

   ```
   wget {{ ipynbpath }}
   ```

3. Switch to the Notebook that appears in the list of files on the left

</div>


### Proving data

The data is provided to the notebook by setting the `dataset_url` parameter in this cell:

```python
# Paste link to the dataset here
dataset_url = 'https://usegalaxy.org/api/datasets/f9cad7b01a472135a8abd43f91f8d3cf/display?to_ext=tabular'
```

This is a URL pointing to one of the workflow outputs: `Mapping report` with the <kbd>PlottingData</kbd> tag. To copy the URL click on the {% icon galaxy-link %} icon adjacent to the dataset:

![Copying the link](../../images/gene-centric/copy_link.png)

Running the notebook will generate two graphs explained in the next section.

## Interpreting the graphs

Analysis of sample data associated with this tutorial will produce the genome graph shown below. In this graph the Y-axis represents ORFs on positive (1, 2, 3 in red color) and negative (-1, -2, -3 in blue color) strands. The X-axis is genomic coordinates.  Boxes represent matches between amino acid sequences of exons and ORFs they are superimposed to. The color of boxes reflect the extent of amino acid identity. The color key is shown in the left upper corner of the plot. The image is interactive so you can zoom in and out.

<html>
<head>
  <script src="https://cdn.jsdelivr.net/npm/vega@5.22.1"></script>
  <script src="https://cdn.jsdelivr.net/npm/vega-lite@5.2.0"></script>
  <script src="https://cdn.jsdelivr.net/npm/vega-embed@6.20.8"></script>
</head>
<body>
  <div id="vis"/>
  <script>
    const spec = {
  "config": {
    "view": {"continuousWidth": 400, "continuousHeight": 300},
    "title": {"anchor": "start", "fontSize": 10}
  },
  "vconcat": [
    {
      "layer": [
        {
          "mark": {"type": "rule", "strokeWidth": 2},
          "encoding": {
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {
              "field": "start",
              "scale": {"domain": [52, 10726]},
              "title": null,
              "type": "quantitative"
            },
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "selection": {
            "selector001": {
              "type": "interval",
              "bind": "scales",
              "encodings": ["x", "y"]
            }
          },
          "title": "aGasCar1.fascaffold_1"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "start", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "aGasCar1.fascaffold_1"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "end", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "aGasCar1.fascaffold_1"
        },
        {
          "mark": {"type": "rule", "opacity": 0.9, "strokeWidth": 10},
          "encoding": {
            "color": {
              "field": "id",
              "scale": {"scheme": "blueorange"},
              "type": "quantitative"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "matchStart", "type": "quantitative"},
            "x2": {"field": "matchEnd"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "aGasCar1.fascaffold_1"
        },
        {
          "mark": {"type": "text", "yOffset": -10},
          "encoding": {
            "color": {"value": "black"},
            "text": {"field": "exon", "type": "nominal"},
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "midpoint", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "aGasCar1.fascaffold_1"
        }
      ],
      "data": {"name": "data-4c922bb95694b1514f353a1acd2ab409"},
      "height": 100,
      "width": 800
    },
    {
      "layer": [
        {
          "mark": {"type": "rule", "strokeWidth": 2},
          "encoding": {
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {
              "field": "start",
              "scale": {"domain": [-228, 3860]},
              "title": null,
              "type": "quantitative"
            },
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "selection": {
            "selector002": {
              "type": "interval",
              "bind": "scales",
              "encodings": ["x", "y"]
            }
          },
          "title": "bTaeGut2.fascaffold_18"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "start", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "bTaeGut2.fascaffold_18"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "end", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "bTaeGut2.fascaffold_18"
        },
        {
          "mark": {"type": "rule", "opacity": 0.9, "strokeWidth": 10},
          "encoding": {
            "color": {
              "field": "id",
              "scale": {"scheme": "blueorange"},
              "type": "quantitative"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "matchStart", "type": "quantitative"},
            "x2": {"field": "matchEnd"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "bTaeGut2.fascaffold_18"
        },
        {
          "mark": {"type": "text", "yOffset": -10},
          "encoding": {
            "color": {"value": "black"},
            "text": {"field": "exon", "type": "nominal"},
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "midpoint", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "bTaeGut2.fascaffold_18"
        }
      ],
      "data": {"name": "data-cd3fa6c014d625fc2f85cfeb0119b9e0"},
      "height": 100,
      "width": 800
    },
    {
      "layer": [
        {
          "mark": {"type": "rule", "strokeWidth": 2},
          "encoding": {
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {
              "field": "start",
              "scale": {"domain": [314, 3976]},
              "title": null,
              "type": "quantitative"
            },
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "selection": {
            "selector003": {
              "type": "interval",
              "bind": "scales",
              "encodings": ["x", "y"]
            }
          },
          "title": "fScoJap1.fascaffold_13"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "start", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "fScoJap1.fascaffold_13"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "end", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "fScoJap1.fascaffold_13"
        },
        {
          "mark": {"type": "rule", "opacity": 0.9, "strokeWidth": 10},
          "encoding": {
            "color": {
              "field": "id",
              "scale": {"scheme": "blueorange"},
              "type": "quantitative"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "matchStart", "type": "quantitative"},
            "x2": {"field": "matchEnd"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "fScoJap1.fascaffold_13"
        },
        {
          "mark": {"type": "text", "yOffset": -10},
          "encoding": {
            "color": {"value": "black"},
            "text": {"field": "exon", "type": "nominal"},
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "midpoint", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "fScoJap1.fascaffold_13"
        }
      ],
      "data": {"name": "data-33f25281b1f78907be251e80305165ce"},
      "height": 100,
      "width": 800
    },
    {
      "layer": [
        {
          "mark": {"type": "rule", "strokeWidth": 2},
          "encoding": {
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {
              "field": "start",
              "scale": {"domain": [-71, 6632]},
              "title": null,
              "type": "quantitative"
            },
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "selection": {
            "selector004": {
              "type": "interval",
              "bind": "scales",
              "encodings": ["x", "y"]
            }
          },
          "title": "mHomSapT2T.fachr22"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "start", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mHomSapT2T.fachr22"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "end", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mHomSapT2T.fachr22"
        },
        {
          "mark": {"type": "rule", "opacity": 0.9, "strokeWidth": 10},
          "encoding": {
            "color": {
              "field": "id",
              "scale": {"scheme": "blueorange"},
              "type": "quantitative"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "matchStart", "type": "quantitative"},
            "x2": {"field": "matchEnd"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mHomSapT2T.fachr22"
        },
        {
          "mark": {"type": "text", "yOffset": -10},
          "encoding": {
            "color": {"value": "black"},
            "text": {"field": "exon", "type": "nominal"},
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "midpoint", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mHomSapT2T.fachr22"
        }
      ],
      "data": {"name": "data-b9417f0d40ef923551b8b4d9ff9101ea"},
      "height": 100,
      "width": 800
    },
    {
      "layer": [
        {
          "mark": {"type": "rule", "strokeWidth": 2},
          "encoding": {
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {
              "field": "start",
              "scale": {"domain": [315, 2819]},
              "title": null,
              "type": "quantitative"
            },
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "selection": {
            "selector005": {
              "type": "interval",
              "bind": "scales",
              "encodings": ["x", "y"]
            }
          },
          "title": "mCynVol1.fascaffold_2"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "start", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mCynVol1.fascaffold_2"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "end", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mCynVol1.fascaffold_2"
        },
        {
          "mark": {"type": "rule", "opacity": 0.9, "strokeWidth": 10},
          "encoding": {
            "color": {
              "field": "id",
              "scale": {"scheme": "blueorange"},
              "type": "quantitative"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "matchStart", "type": "quantitative"},
            "x2": {"field": "matchEnd"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mCynVol1.fascaffold_2"
        },
        {
          "mark": {"type": "text", "yOffset": -10},
          "encoding": {
            "color": {"value": "black"},
            "text": {"field": "exon", "type": "nominal"},
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "midpoint", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mCynVol1.fascaffold_2"
        }
      ],
      "data": {"name": "data-4ee088ff88102dae61c31ea38e9d47e2"},
      "height": 100,
      "width": 800
    },
    {
      "layer": [
        {
          "mark": {"type": "rule", "strokeWidth": 2},
          "encoding": {
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {
              "field": "start",
              "scale": {"domain": [-74, 5866]},
              "title": null,
              "type": "quantitative"
            },
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "selection": {
            "selector006": {
              "type": "interval",
              "bind": "scales",
              "encodings": ["x", "y"]
            }
          },
          "title": "mCynVol1.fascaffold_3"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "start", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mCynVol1.fascaffold_3"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "end", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mCynVol1.fascaffold_3"
        },
        {
          "mark": {"type": "rule", "opacity": 0.9, "strokeWidth": 10},
          "encoding": {
            "color": {
              "field": "id",
              "scale": {"scheme": "blueorange"},
              "type": "quantitative"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "matchStart", "type": "quantitative"},
            "x2": {"field": "matchEnd"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mCynVol1.fascaffold_3"
        },
        {
          "mark": {"type": "text", "yOffset": -10},
          "encoding": {
            "color": {"value": "black"},
            "text": {"field": "exon", "type": "nominal"},
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "midpoint", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mCynVol1.fascaffold_3"
        }
      ],
      "data": {"name": "data-729a5f2f10472b0a903d29284fd22ba5"},
      "height": 100,
      "width": 800
    }
  ],
  "resolve": {"scale": {"y": "shared"}},
  "$schema": "https://vega.github.io/schema/vega-lite/v4.17.0.json",
  "datasets": {
    "data-4c922bb95694b1514f353a1acd2ab409": [
      {
        "genome": "aGasCar1.fa",
        "chr": "scaffold_1",
        "start": 552,
        "end": 684,
        "orf": "aGasCar1.fascaffold_1_ORF.2",
        "frame": 1,
        "strand": "+",
        "midpoint": null,
        "matchStart": null,
        "matchEnd": null,
        "exon": null,
        "id": null,
        "genome_chr": "aGasCar1.fascaffold_1",
        "clade": "a"
      },
      {
        "genome": "aGasCar1.fa",
        "chr": "scaffold_1",
        "start": 762,
        "end": 1029,
        "orf": "aGasCar1.fascaffold_1_ORF.3",
        "frame": 1,
        "strand": "+",
        "midpoint": null,
        "matchStart": null,
        "matchEnd": null,
        "exon": null,
        "id": null,
        "genome_chr": "aGasCar1.fascaffold_1",
        "clade": "a"
      },
      {
        "genome": "aGasCar1.fa",
        "chr": "scaffold_1",
        "start": 8880,
        "end": 9045,
        "orf": "aGasCar1.fascaffold_1_ORF.31",
        "frame": 1,
        "strand": "+",
        "midpoint": "8970",
        "matchStart": "8904",
        "matchEnd": "9036",
        "exon": "xbp-1u-p1",
        "id": "75.6",
        "genome_chr": "aGasCar1.fascaffold_1",
        "clade": "a"
      },
      {
        "genome": "aGasCar1.fa",
        "chr": "scaffold_1",
        "start": 8880,
        "end": 9045,
        "orf": "aGasCar1.fascaffold_1_ORF.31",
        "frame": 1,
        "strand": "+",
        "midpoint": "8923",
        "matchStart": "8904",
        "matchEnd": "8943",
        "exon": "xbp-1s-p12",
        "id": "85.7",
        "genome_chr": "aGasCar1.fascaffold_1",
        "clade": "a"
      },
      {
        "genome": "aGasCar1.fa",
        "chr": "scaffold_1",
        "start": 9270,
        "end": 9432,
        "orf": "aGasCar1.fascaffold_1_ORF.32",
        "frame": 1,
        "strand": "+",

tags_json:
["evolution", "annotation", "vgp", "cookbook", "eukaryote", "prokaryote", "microgalaxy", "biodiversity"]


topic: genome-annotation
tutorial: annotation-with-maker
title: Genome annotation with Maker
difficulty: advanced
time_estimation: 4h
content: {% include topics/genome-annotation/tutorials/annotation-with-maker/content.md short=false %}
tools_json: []
requirements_json: []
tags_json: ["gmod", "eukaryote", "jbrowse1", "maker"]
content_hash: 3ac46864e10bda17a1cfd1c2c83e9061
last_modified: 2026-02-26T21:12:16.189276
zenodo_link: https://doi.org/10.5281/zenodo.4406623
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/annotation-with-maker/tutorial.html


topic: genome-annotation
tutorial: genome-annotation
title: Genome Annotation
difficulty: introductory
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: ["prokaryote"]
content_hash: f1f384d1e62e05ae1d954fa1a7c2b569
last_modified: 2026-02-26T21:12:16.596466
zenodo_link: https://doi.org/10.5281/zenodo.1250793
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/genome-annotation/tutorial.html

content:
Genome annotation is the process of attaching biological information to sequences.
It consists of three main steps:

 - identifying portions of the genome that do not code for proteins
 - identifying elements on the genome, a process called gene prediction, and
 - attaching biological information to these elements.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Introduction into File Formats

## FASTA

DNA and protein sequences are written in FASTA format where you have in the first line a ">" followed by the description. In the second line the sequence starts.

![FASTA file](../../images/fasta_format.png)

## GFF3

The general feature format (gene-finding format, generic feature format, GFF) is a file format used for describing genes and other features of DNA, RNA and protein sequences.

<img src="../../images/gff3_format.png" alt="GFF3 overview" width="70%">

## GENBANK

The genbank sequence format is a rich format for storing sequences and associated annotations.

![genbank file](../../images/gb_full.png)

# Structural Annotation

For the genome annotation we use a piece of the *Aspergillus fumigatus* [genome sequence](https://zenodo.org/record/1250793/files/Aspergillus_sequence.fasta) as input file.

## Sequence Features

First we want to get some general information about our sequence.

> <hands-on-title>Sequence composition</hands-on-title>
>
> 1. Count the number of bases in your sequence (**compute sequence length**)
> 2. Check for sequence composition and GC content (**geecee**).
> 3. Plot the sequence composition as bar chart.
>
> <img src="../../images/barchart_sequencecomposition.png" alt="Bar chart output of the sequence" width="30%">
>
{: .hands_on}

## Gene Prediction

At first you need to identify those structures of the genome which code for proteins. This step of annotation is called “structural annotation”. It contains the identification and location of open reading frames (ORFs), identification of gene structures and coding regions, and the location of regulatory motifs. Galaxy contains several tools for the structural annotation. Tools for gene prediction are **Augustus** (for eukaryotes and prokaryotes) and **glimmer3** (only for prokaryotes).

> <hands-on-title>Gene prediction</hands-on-title>
>
> We use **Augustus** for gene prediction.
> 1. Use the genome sequence (FASTA file) as input.
> 2. Choose the right *model organism*, *gff* format output.
> 3. Select all possible output options.
>
> ![augustus](../../images/augustus.png)
>
> Augustus will provide three output files: *gff3*, *coding sequences* (CDS) and *protein sequences*.
>
>    > <question-title></question-title>
>    >
>    > How many genes are predicted?
>    >
>    > > <solution-title></solution-title>
>    > > Check the output: [augustus_output](../../images/augustus_out.png)
>    > {: .solution }
> {: .question}
>
{: .hands_on}

> <hands-on-title>tRNA and tmRNA Prediction</hands-on-title>
>
> Use **Aragorn** for tRNA and tmRNA prediction.
> 1. As input file use the *Aspergillus* genome sequence. You can choose the genetic code (e.g. bacteria).
> 2. Select the topology of your genome (circular or linear).
>
>    > <question-title></question-title>
>    >
>    > Are there tRNAs or tmRNAs in the sequence?
>    >
>    {: .question}
>
{: .hands_on}

> <details-title>Aragorn in depth</details-title>
>
> read more about **Aragorn** [in this publication](https://nar.oxfordjournals.org/content/32/1/11.full.pdf+html).
{: .details}

# Functional Annotation

## Similarity Searches (BLAST)

Functional gene annotation means the description of the biochemical and biological function of proteins. Possible analyses to annotate genes can be for example:

- similarity searches
- gene cluster prediction for secondary metabolites
- identification of transmembrane domains in protein sequences
- finding gene ontology terms
- pathway information

For similarity searches we use *NCBI BLAST+ blastp* to find similar proteins in a protein database.

> <hands-on-title> Similarity search</hands-on-title>
>
> 1. {% icon tool %} As input file, select the protein sequences from Augustus.
> 2. Choose the protein BLAST database *SwissProt* and the output format *xml*.
>
> <img src="../../images/blastP.png" alt="blastp tool interface and parameters" width="70%">
>
> 3. Parsing the xml output (**Parse blast XML output**) results in changing the format style into tabular.
>
>    > <question-title></question-title>
>    >
>    > What information do you see in the BLAST output?
>    >
>    {: .question}
>
>
> From BLAST search results we want to get only the best hit for each protein.
> 4. {% icon tool %} Therefore apply the tool **BLAST top hit descriptions** with *number of descriptions =1* on the xml output file.
>
>    > <question-title></question-title>
>    >
>    > For how many proteins we do not get a BLAST hit?
>    >
>    {: .question}
>
> 5. {% icon tool %} Choose the tool **Select lines that match an expression** and enter the following information: *Select lines from* [select the BLAST top hit descriptions result file]; *that* [not matching]; *the pattern* [gi].
>
>    <img src="../../images/selectlines.png" alt="Select lines that match an expression tool interface and parameters" width="50%">
>
>    > <comment-title>Results file</comment-title>
>    > The result file will contain all proteins which do not have an entry in the second column and therefore have no similar protein in the SwissProt database.
>    {: .comment}
>
>    > <comment-title>Obtaining unannotated proteins for analysis</comment-title>
>    > For functional description of those proteins we want to search for motifs or domains which may classify them more. To get a protein sequence FASTA file with only the not annotated proteins, use the tool **Filter sequences by ID from a tabular file** and select for *Sequence file to filter on the identifiers* [Augustus protein sequences] and for *Tabular file containing sequence identifiers* the protein file with not annotated sequences. The output file is a FASTA file with only those sequences without description.
>    {: .comment}
>
{: .hands_on}

This file will be the input for more detailed analysis:

- **Interproscan** is a functional prediction tool. Select all applications and run it on your protein file.

- **WolfPSort** predicts eukaryote protein subcellular localization. Filter the result file for the best ranked localization hit. Use **Filter data on any column using simple expressions** with *c4==1*. The parameter c4==1 means: filter and keep all results where in column 4 is a "1".

- **TMHMM** finds transmembrane domains in protein sequences. The number of amino acids in transmembrane helices should be >18. This information can be found in column 3. Filter the result file *c3>17.99*.

- **BLAST2GO** maps BLAST results to GO annotation terms.

### BLAST Programs

![BLAST programs](../../images/blastprograms.png)

![BLAST databases](../../images/blast_database.png)

> <details-title>Organism not available in a BLAST database</details-title>
>
> If you have an organism which is not available in a BLAST database, you can use its genome sequence in FASTA file for BLAST searches "sequence file against sequence file". If you need to search in these sequences on a regularly basis, you can create a own BLAST database from the sequences of the organism. The advantage of having a own database for your organism is the duration of the BLAST search which speeds up a lot.
{: .details}

**NCBI BLAST+ makeblastdb** creates a BLAST database from your own FASTA sequence file. Molecule type of input is protein or nucleotide.

> <details-title>Further Reading about BLAST Tools in Galaxy</details-title>
>
> Cock et al. (2015): [NCBI BLAST+ integrated into Galaxy](http://biorxiv.org/content/early/2015/05/04/014043.full-text.pdf+html)
>
> Cock et al. (2013): [Galaxy tools and workflows for sequence analysis with applications in molecular plant pathology](https://peerj.com/articles/167/)
{: .details}

## More Similarity Search Tools in Galaxy

* **VSEARCH**: For processing metagenomic sequences, including searching, clustering, chimera detection, dereplication, sorting, masking and shuffling. VSEARCH stands for vectorized search, as the tool takes advantage of parallelism in the form of SIMD vectorization as well as multiple threads to perform accurate alignments at high speed. VSEARCH uses an optimal global aligner (full dynamic programming Needleman-Wunsch), in contrast to USEARCH which by default uses a heuristic seed and extend aligner. This results in more accurate alignments and overall improved sensitivity (recall) with VSEARCH, especially for alignments with gaps.

> <details-title>`vsearch` in depth</details-title>
>
> Documentation for vsearch available via their [GitHub repository](https://github.com/torognes/vsearch).
{: .details}

* **Diamond**: Diamond is a high-throughput program for aligning a file of short reads against a protein reference database such as NR, at 20,000 times the speed of Blastx, with high sensitivity.

> <details-title>Diamond in depth</details-title>
>
> [Buchfink et al. (2015): Fast and sensitive protein alignment using Diamond.](https://www.nature.com/nmeth/journal/v12/n1/abs/nmeth.3176.html)
{: .details}

* **Kraken**: Kraken BLAST is a highly scalable, extremely fast, commercial, parallelized implementation of the NCBI BLAST application.

## Identification of Gene Clusters

For identification of gene clusters, **antiSMASH** is used. The tool uses genbank file as input files and predicts gene clusters. Output files are a html visualization and the gene cluster proteins.

> <hands-on-title>antiSMASH analysis</hands-on-title>
>
> {% icon tool %} Import this [dataset](../../input_data/Streptomyces_coelicolor_part.genbank) into your Galaxy history and run **antiSMASH** to detect gene clusters. The genbank file contains a part of the *Streptomyces coelicolor* genome sequence.
>
{: .hands_on}

> <question-title></question-title>
>
> Which gene clusters are identified?
>
{: .question}

When you have a whole genome **antiSMASH** analysis, your result may look like this:

<img src="../../images/antismash_full.png" alt="The result of antiSMASH" width="80%">

At the end, you can extract a reproducible workflow out of your history. The workflow should look like this:

![GenomeAnnotation Workflow](../../images/work_flow_Screenshot_from_2015-06-23_09-33-23.png)


topic: genome-annotation
tutorial: bacterial-pangenomics
title: Bacterial pangenomics
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: []
content_hash: e9ac617f92e0ada230bed4759722cb5a
last_modified: 2026-02-26T21:12:16.212452
zenodo_link: https://zenodo.org/records/1
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/bacterial-pangenomics/tutorial.html

content:
General introduction about the topic and then an introduction of the
tutorial (the questions and the objectives). It is nice also to have a
scheme to sum up the pipeline used during the tutorial. The idea is to
give to trainees insight into the content of the tutorial and the (theoretical
and technical) key concepts they will learn.

You may want to cite some publications; this can be done by adding citations to the
bibliography file (`tutorial.bib` file next to your `tutorial.md` file). These citations
must be in bibtex format. If you have the DOI for the paper you wish to cite, you can
get the corresponding bibtex entry using [doi2bib.org](https://doi2bib.org).

With the example you will find in the `tutorial.bib` file, you can add a citation to
this article here in your tutorial like this:
{% raw %} `{% cite Batut2018 %}`{% endraw %}.
This will be rendered like this: {% cite Batut2018 %}, and links to a
bibliography section which will automatically be created at the end of the
tutorial.


**Please follow our
[tutorial to learn how to fill the Markdown]({{ site.baseurl }}/topics/contributing/tutorials/create-new-tutorial-content/tutorial.html)**

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Galaxy and data preparation

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 3. {% tool [Import](upload1) %} the contig file from [Zenodo]({{ page.zenodo_link }}) or from Galaxy shared data libraries:
>
>    ```
>    {{ page.zenodo_link }}/files/DRR187559_contigs.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
{: .hands_on}

# Pangenomics

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [PPanGGOLiN all](toolshed.g2.bx.psu.edu/repos/iuc/ppanggolin_all/ppanggolin_all/2.2.1+galaxy1) %} with the following parameters:
>    - {% icon param-collection %} *"Select genome files"*: Input dataset collection
>
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}

# Multi-sequence alignment

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [PPanGGOLiN MSA](toolshed.g2.bx.psu.edu/repos/iuc/ppanggolin_msa/ppanggolin_msa/2.2.1+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input pangenome h5 file"*: `pangenome_h5` (output of **PPanGGOLiN all** {% icon tool %})
>    - *"Partition"*: `Core`
>
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}

# Conclusion

Sum up the tutorial and the key takeaways here. We encourage adding an overview image of the
pipeline used.


topic: genome-annotation
tutorial: bacterial-genome-quality-control
title: Bacterial genome quality control
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["microgalaxy"]
content_hash: b882c6aa7dc71cb8321dd0b795a41052
last_modified: 2026-02-26T21:12:16.212135
zenodo_link: https://zenodo.org/records/1
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/bacterial-genome-quality-control/tutorial.html

content:
General introduction about the topic and then an introduction of the
tutorial (the questions and the objectives). It is nice also to have a
scheme to sum up the pipeline used during the tutorial. The idea is to
give to trainees insight into the content of the tutorial and the (theoretical
and technical) key concepts they will learn.

You may want to cite some publications; this can be done by adding citations to the
bibliography file (`tutorial.bib` file next to your `tutorial.md` file). These citations
must be in bibtex format. If you have the DOI for the paper you wish to cite, you can
get the corresponding bibtex entry using [doi2bib.org](https://doi2bib.org).

With the example you will find in the `tutorial.bib` file, you can add a citation to
this article here in your tutorial like this:
{% raw %} `{% cite Batut2018 %}`{% endraw %}.
This will be rendered like this: {% cite Batut2018 %}, and links to a
bibliography section which will automatically be created at the end of the
tutorial.


**Please follow our
[tutorial to learn how to fill the Markdown]({{ site.baseurl }}/topics/contributing/tutorials/create-new-tutorial-content/tutorial.html)**

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Galaxy and data preparation

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 3. {% tool [Import](upload1) %} the contig file from [Zenodo]({{ page.zenodo_link }}) or from Galaxy shared data libraries:
>
>    ```
>    {{ page.zenodo_link }}/files/DRR187559_contigs.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
{: .hands_on}

# Quality control

It comes first a description of the step: some background and some theory.
Some image can be added there to support the theory explanation:

The idea is to keep the theory description before quite simple to focus more on the practical part.

***TODO***: *Consider adding a detail box to expand the theory*

> <details-title> More details about the theory </details-title>
>
> But to describe more details, it is possible to use the detail boxes which are expandable
>
{: .details}

A big step can have several subsections or sub steps:

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [checkm2](toolshed.g2.bx.psu.edu/repos/iuc/checkm2/checkm2/1.0.2+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Input MAG/SAG datasets"*: Input dataset collection
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}

# Dereplication

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [dRep dereplicate](toolshed.g2.bx.psu.edu/repos/iuc/drep_dereplicate/drep_dereplicate/3.5.0+galaxy1) %} with the following parameters:
>    - {% icon param-collection %} *"Genomes"*: `output` (Input dataset collection)
>    - *"Genome quality filtering"*: `Run checkM`
>    - In *"Genome comparison and clustering"*:
>        - *"Steps in genome comparison"*: `Default: Run MASH clustering and a secondary clustering`
>            - *"Algorithm for secondary clustering comparisons"*: `ANImf: Align whole genomes with nucmer; filter alignment; compare aligned regions - RECOMMENDED`
>    - *"Select outputs"*: ``
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}


# Conclusion

Sum up the tutorial and the key takeaways here. We encourage adding an overview image of the
pipeline used.


topic: genome-annotation
tutorial: amr-gene-detection
title: Identification of AMR genes in an assembled bacterial genome
difficulty: introductory
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: ["gmod", "illumina", "amr", "one-health", "jbrowse1", "microgalaxy"]
content_hash: f25b876d874f66951f79ea66b784961d
last_modified: 2026-02-26T21:12:16.187584
zenodo_link: https://zenodo.org/record/10572227
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/amr-gene-detection/tutorial.html

content:
Antimicrobial resistance (AMR) is a global phenomenon with no geographical or species boundaries, which poses an important threat to human, animal and environmental health. It is a complex and growing problem that compromises our ability to treat bacterial infections.

AMR gene content can be assessed from whole genome sequencing to detect known resistance mechanisms and potentially identify novel mechanisms.

To illustrate the process to identify AMR gene in a bacterial genome, we take an assembly of a bacterial genome (KUN1163 sample) generated by following a [bacterial genome assembly tutorial]({% link topics/assembly/tutorials/mrsa-illumina/tutorial.md %}) from data produced in "Complete Genome Sequences of Eight Methicillin-Resistant *Staphylococcus aureus* Strains Isolated from Patients in Japan" ({% cite Hikichi_2019 %}).

> Methicillin-resistant *Staphylococcus aureus* (MRSA) is a major pathogen
> causing nosocomial infections, and the clinical manifestations of MRSA
> range from asymptomatic colonization of the nasal mucosa to soft tissue
> infection to fulminant invasive disease. Here, we report the complete
> genome sequences of eight MRSA strains isolated from patients in Japan.
{: .quote cite="{% cite_url Hikichi_2019 %}"}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Galaxy and data preparation

Any analysis should get its own Galaxy history. So let's start by creating a new one and get the data (contig file from the assembly) into it.

> <hands-on-title>Prepare Galaxy and data</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 3. {% tool [Import](upload1) %} the contig file from [Zenodo]({{ page.zenodo_link }}) or from Galaxy shared data libraries:
>
>    ```
>    {{ page.zenodo_link }}/files/DRR187559_contigs.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
{: .hands_on}

# Identification of AMR genes

To identify AMR genes in contigs, tools like ABRicate or staramr ({% cite bharat2022correlation %}) can be used. In this tutorial, we use [**staramr**](https://github.com/phac-nml/staramr). **Staramr** scans bacterial genome contigs against the **ResFinder** ({% cite zankari2012identification %}), **PointFinder** ({% cite zankari2017pointfinder %}), and **PlasmidFinder** ({% cite carattoli2014plasmidfinder %}) databases (used by the ResFinder webservice) and compiles a summary report of detected antimicrobial resistance genes.

> <comment-title></comment-title>
>
> If you are interested in learning more about using ABRicate for AMR gene detection, you can check the sub-section on AMR in "Gene based pathogenic identification" of ["Pathogen detection from (direct Nanopore) sequencing data using Galaxy - Foodborne Edition" tutorial]({% link topics/microbiome/tutorials/pathogen-detection-from-nanopore-foodborne-data/tutorial.md %})
>
{: .comment}

> <hands-on-title>Run staramr</hands-on-title>
>
> 1. {% tool [staramr](toolshed.g2.bx.psu.edu/repos/nml/staramr/staramr_search/0.10.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"genomes"*: Contig file
>
{: .hands_on}

**staramr** generates 8 outputs:

1. `summary.tsv`:  A summary of all detected AMR genes/mutations/plasmids/sequence type in each genome, one genome per line and the following columns:

   * __Isolate ID__: The id of the isolate/genome file(s) passed to `staramr`.
   * __Quality Module__: The isolate/genome file(s) pass/fail result(s) for the quality metrics
   * __Genotype__: The AMR genotype of the isolate.
   * __Predicted Phenotype__: The predicted AMR phenotype (drug resistances) for the isolate.
   * __CGE Predicted Phenotype__: The CGE-predicted AMR phenotype (drug resistances) for the isolate (CGE = [Center for Genomic Epidemiology](http://www.genomicepidemiology.org/))

   * __Plasmid__: Plasmid types that were found for the isolate.
   * __Scheme__: The MLST scheme used

      MLST stands for MultiLocus Sequence Typing. It is a technique for the typing of multiple loci, using DNA sequences of internal fragments of multiple housekeeping genes to characterize isolates of microbial species.

      Here, **starmr** uses [mlst](https://github.com/tseemann/mlst) to scan the contig files against traditional [PubMLST](https://pubmlst.org/) typing schemes. The correspondance between the scheme and the bacteria genus and species is accessible in the [map](https://github.com/tseemann/mlst/blob/master/db/scheme_species_map.tab)

   * __Sequence Type__: The sequence type that's assigned when combining all allele types
   * __Genome Length__: The isolate/genome file(s) genome length(s)
   * __N50 value__: The isolate/genome file(s) N50 value(s)
   * __Number of Contigs Greater Than Or Equal To 300 bp__: The number of contigs greater or equal to 300 base pair in the isolate/genome file(s)
   * __Quality Module Feedback__: The isolate/genome file(s) detailed feedback for the quality metrics

    > <hands-on-title>Inspect the staramr output</hands-on-title>
    >
    > 1. View {% icon galaxy-eye %} the `summary.tsv` file
    >
    {: .hands_on}

    > <question-title>Inspect the staramr output</question-title>
    >
    > 1. How many genomes are there?
    > 2. Has the genome failed or passed the quality? Why?
    > 3. What are the predicted AMR drug resistances? Are they similar to the one found for KUN1163 in [Table 1](https://journals.asm.org/doi/10.1128/mra.01212-19#tab1) in {% cite Hikichi_2019 %}?
    > 4. What is the MLST scheme? Does that correspond to the expected species and genus?
    >
    > > <solution-title>Inspect the staramr output</solution-title>
    > >
    > > 1. There is one genome (1 line)
    > > 2. The genome has failed the quality (column 2), because the genome length is not within the acceptable length range (last column).
    > > 3. We can summarize starmr output and Table 1 in {% cite Hikichi_2019 %}:
    > >
    > >    Antibiotic name | Abbreviation | staramr | {% cite Hikichi_2019 %}
    > >    --- | --- | --- | ---
    > >    Amikacin | | Yes |
    > >    Azithromycin | | Yes |
    > >    Cefazolin | CEZ | | Yes
    > >    clindamycin | CLDM | | Yes
    > >    Erythromycin | EM | Yes | Yes
    > >    Gentamicin | GM | Yes | Yes
    > >    Imipenem | IPM | | Yes
    > >    Levofloxacin | LVFX | | Yes
    > >    Oxacillin | MPIPC | | Yes
    > >    Penicillin | | Yes |
    > >    Spectinomycin | | Yes |
    > >    Tetracycline | | Yes |
    > >    Tobramycin | | Yes |
    > >
    > > 4. The scheme is saureus, so *Staphylococcus aureus* (given the [scheme genus map](https://github.com/tseemann/mlst/blob/master/db/scheme_species_map.tab)), which is coherent with MRSA
    > {: .solution}
    >
    {: .question}

2. `detailed_summary.tsv`: A detailed summary of all detected AMR genes/mutations/plasmids/sequence type in each genome, one gene per line and the following columns:

    * __Isolate ID__: The id of the isolate/genome file(s) passed to `staramr`.
    * __Data__: The particular gene detected from ResFinder, PlasmidFinder, PointFinder, or the sequence type.
    * __Data Type__: The type of gene (Resistance or Plasmid), or MLST.
    * __Predicted Phenotype__: The predicted AMR phenotype (drug resistances) found in ResFinder/PointFinder. Plasmids will be left blank by default.
    * __CGE Predicted Phenotype__: The CGE-predicted AMR phenotype (drug resistances) found in ResFinder/PointFinder. Plasmids will be left blank by default.
    * __%Identity__: The % identity of the top BLAST HSP to the gene.
    * __%Overlap__: THe % overlap of the top BLAST HSP to the gene (calculated as __hsp length/total length * 100__).
    * __HSP Length/Total Length__ The top BLAST HSP length over the gene total length (nucleotides).
    * __Contig__: The contig id containing this gene.
    * __Start__: The start of the gene (will be greater than __End__ if on minus strand).
    * __End__: The end of the gene.
    * __Accession__: The accession of the gene from either ResFinder or PlasmidFinder  database.

    > <hands-on-title>Inspect the staramr output</hands-on-title>
    >
    > 1. View {% icon galaxy-eye %} the `detailed_summary.tsv` file
    >
    {: .hands_on}

    > <question-title>Inspect the staramr output</question-title>
    >
    > 1. What are the different sequence types that have been identified?
    > 2. Where are located the plasmid genes?
    > 3. Where are located the resistances genes?
    > 4. By looking for the accession number (M13771) on NCBI ([https://www.ncbi.nlm.nih.gov/nuccore/](https://www.ncbi.nlm.nih.gov/nuccore/)), where does the first resistance come from?
    >
    > > <solution-title>Inspect the staramr output</solution-title>
    > >
    > > 1. There are:
    > >    - 1 MSLT
    > >    - 5 plasmid *rep* genes
    > >    - 7 resistance genes
    > > 2. The plasmid genes are on contig00019 (3 genes - coherent with {% cite Lozano_2012 %}), contig00024, and contig00002.
    > > 3. The resistance genes are:
    > >    - 4/7 on contigs with plasmid genes (contig00002, contig00019, contig00024)
    > >    - 3/7 on contigs without plasmid genes (contig00022, contig00021)
    > > 4. M13771 is a 6'-aminoglycoside acetyltransferase phosphotransferase (AAC(6')-APH(2')) bifunctional resistance protein from *Streptococcus faecalis*. It might come from horizontal transfer from *Streptococcus faecalis* on a plasmid of *Staphylococcus aureus*.
    > {: .solution}
    >
    {: .question}

3. `resfinder.tsv`: A tabular file of each AMR gene and additional BLAST information from the **ResFinder** database, one gene per line and the following columns:

   * __Isolate ID__: The id of the isolate/genome file(s) passed to `staramr`.
   * __Gene__: The particular AMR gene detected.
   * __Predicted Phenotype__: The predicted AMR phenotype (drug resistances) for this gene.
   * __CGE Predicted Phenotype__: The CGE-predicted AMR phenotype (drug resistances) for this gene.
   * __%Identity__: The % identity of the top BLAST HSP to the AMR gene.
   * __%Overlap__: THe % overlap of the top BLAST HSP to the AMR gene (calculated as __hsp length/total length * 100__).
   * __HSP Length/Total Length__ The top BLAST HSP length over the AMR gene total length (nucleotides).
   * __Contig__: The contig id containing this AMR gene.
   * __Start__: The start of the AMR gene (will be greater than __End__ if on minus strand).
   * __End__: The end of the AMR gene.
   * __Accession__: The accession of the AMR gene in the ResFinder database.
   * __Sequence__: The AMR Gene sequence
   * __CGE Notes__: Any CGE notes associated with the prediction.

4. `plasmidfinder.tsv`: A tabular file of each AMR plasmid type and additional BLAST information from the **PlasmidFinder** database, one plasmid type per line with the following columns:

   * __Isolate ID__: The id of the isolate/genome file(s) passed to `staramr`.
   * __Plasmid__: The particular plasmid type detected.
   * __%Identity__: The % identity of the top BLAST HSP to the plasmid type.
   * __%Overlap__: The % overlap of the top BLAST HSP to the plasmid type (calculated as    __hsp length/total length * 100__).
   * __HSP Length/Total Length__ The top BLAST HSP length over the plasmid type total length (nucleotides).
   * __Contig__: The contig id containing this plasmid type.
   * __Start__: The start of the plasmid type (will be greater than __End__ if on minus strand).
   * __End__: The end of the plasmid type.
   * __Accession__: The accession of the plasmid type in the PlasmidFinder database.

5. `mlst.tsv`: A tabular file of each multi-locus sequence type (MLST) and it's corresponding locus/alleles, one genome per line with the following columns:

   * __Isolate ID__: The id of the isolate/genome file(s) passed to `staramr`.
   * __Scheme__: The scheme that `MLST` has identified.
   * __Sequence Type__: The sequence type that's assigned when combining all allele types
   * __Locus #__: A particular locus in the specified MLST scheme.

6. `settings.txt`: The command-line, database versions, and other settings used to run `staramr`.
7. `results.xlsx`: An Excel spreadsheet containing the previous files as separate worksheets.
8. `hits`: A collection with fasta files of the BLAST HSP nucleotides for the entries listed in the **resfinder.tsv** and **pointfinder.tsv** files. There are up to two files per input genome, one for ResFinder and one for PointFinder.

# Getting more information about ARG via the CARD database

To get more information about the antibiotic resistant genes (ARG), we can check the [CARD database](https://card.mcmaster.ca) (**C**omprehensive **A**ntibiotic **R**esistance **D**atabase) ({% cite Jia2016 %})

CARD can be very helpful to check all the resistance genes and check if it is logical to find the resistance gene in a specific bacteria.

> <question-title></question-title>
>
> 1. To what family does [mecA](https://card.mcmaster.ca/ontology/36911) belong?
> 2. Do you expect to find this gene in this MRSA strain and why?
> 3. Is the accession number of the entry related to the accession reported by staramr?
>
> > <solution-title></solution-title>
> >
> > 1. [Methicillin resistant PBP2](https://card.mcmaster.ca/ontology/37589)
> > 2. The strain we use is a Methicillin(multi) resistant Staphylococcus aureus. As `mecA` has a perfect resistome mach with *S. aureus*, and the AMR Gene Family is methicillin resistant PBP2, we expect to see mecA in MRSA.
> > 3. No, these are completely unrelated. Unfortunately this is a **very** common issue in bioinformatics. Everyone builds their own numbering system for entries in their database (usually calling them 'accessions'), and then someone else needs to build a service to link these databases.
> >
> {: .solution}
{: .question}

# Visualization of the ARGs and plasmid genes in their genomic context

We would like to look at the ARGs and plasmid genes in their genomic context. To do that, we will usie [JBrowse](https://jbrowse.org/jb2/) ({% cite diesh2023jbrowse %}) with several information:

1. Assembly as the reference
2. ARGs location
3. Contigs annotation (genes, etc)
4. Coverage of the contigs from the raw reads

## Extraction of the ARG and plasmid genes location

The first step is to extract the location of the ARGs and plasmid genes on the contigs. This information is available on the `detailed_summary.tsv` output of **starmar**. The genes and their location are on the lines with a decimal value on column 6 or 7. So to only get this information, we need to select lines with a decimal value (`###.##`) followed by a tab character, the column separator in Galaxy. As a result, any lines without an identity or overlap value will be filtered out.

> <hands-on-title>Extract ARGs and plasmid genes</hands-on-title>
> 1. {% tool [Select lines that match an expression](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `detailed_summary.tsv` **staramr** output
>    - *"that"*: Matching
>    - *"the pattern"*: `[0-9]+\.[0-9]+\t`
{: .hands_on}

> <question-title></question-title>
>
> How many genes have been kept?
>
> > <solution-title></solution-title>
> >
> > 12
> >
> {: .solution}
{: .question}

This table can not be used directly in JBrowse. It first needs to be transformed in a standard format: GFF3, a file format used for describing genes and other features of DNA, RNA and protein sequences.


> <comment-title>GFF3 file format</comment-title>
>
> A GFF is a tab delimited file with 9 fields per line:
> 1. **seqid**: The name of the sequence where the feature is located.
> 2. **source**: The algorithm or procedure that generated the feature. This is typically the name of a software or database.
> 3. **type**: The feature type name, like "gene" or "exon". In a well structured GFF file, all the children features always follow their parents in a single block (so all exons of a transcript are put after their parent "transcript" feature line and before any other parent transcript line). In GFF3, all features and their relationships should be compatible with the standards released by the Sequence Ontology Project.
> 4. **start**: Genomic start of the feature, with a 1-base offset. This is in contrast with other 0-offset half-open sequence formats, like BED.
> 5. **end**: Genomic end of the feature, with a 1-base offset. This is the same end coordinate as it is in 0-offset half-open sequence formats, like BED.
> 6. **score**: Numeric value that generally indicates the confidence of the source in the annotated feature. A value of "." (a dot) is used to define a null value.
> 7. **strand**: Single character that indicates the strand of the feature. This can be "+" (positive, or 5'->3'), "-", (negative, or 3'->5'), "." (undetermined), or "?" for features with relevant but unknown strands.
> 8. **phase**: phase of CDS features; it can be either one of 0, 1, 2 (for CDS features) or "." (for everything else). See the section below for a detailed explanation.
> 9. **attributes**: A list of tag-value pairs separated by a semicolon with additional information about the feature.
{: .comment}

> <hands-on-title>Create a GFF file</hands-on-title>
> 3. {% tool [Table to GFF3](toolshed.g2.bx.psu.edu/repos/iuc/tbl2gff3/tbl2gff3/1.2) %}
>    - {% icon param-file %} *"Table"*: output of the above **Select lines** {% icon tool %} step
>    - *"Record ID column or value"*: `9`
>    - *"Start column or value"*: `10`
>    - *"End column or value"*: `11`
>    - *"Type column or value"*: `3`
>    - *"Score column or value"*: `6`
>    - *"Source column or value"*: `3`
>    - {% icon param-repeat %} *"Insert Qualifiers"*
>        - *"Name"*: `name`
>        - *"Qualifier value column or raw text"*: `2`
>    - {% icon param-repeat %} *"Insert Qualifiers"*
>        - *"Name"*: `phenotype`
>        - *"Qualifier value column or raw text"*: `4`
>    - {% icon param-repeat %} *"Insert Qualifiers"*
>        - *"Name"*: `accession`
>        - *"Qualifier value column or raw text"*: `12`
>
{: .hands_on}

> <question-title></question-title>
>
> How many genes have been found per contigs?
>
> > <solution-title></solution-title>
> >
> > - Contig 19: 3 plasmid genes, 1 ARG
> > - Contig 24: 1 plasmid gene, 2 ARGs
> > - Contig 2: 1 plasmid gene, 1 ARG
> > - Contig 22: 2 ARGs
> > - Contig 21: 1 ARG
> >
> {: .solution}
{: .question}

## Annotation of the contigs

In addition to the ARGs and plasmid genes, it would be good to have extra information about other genes on the contigs. Several tools exists to do that: Prokka ({% cite seemann2014prokka %}), Bakta ({% cite schwengers2021bakta %}), etc. Here, we use **Bakta** as recommended by Torsten Seeman as the successor of Prokka.

{% include _includes/cyoa-choices.html option1="No" option2="Yes" default="No" text="Have you already run Bakta on this dataset before?"%}

<div class="No" markdown="1">

**Bakta**  is a tool for the rapid & standardized annotation of bacterial genomes and plasmids from both isolates and MAGs.

> <hands-on-title>Annotate the contigs</hands-on-title>
>
> 1. {% tool [Bakta](toolshed.g2.bx.psu.edu/repos/iuc/bakta/bakta/1.8.2+galaxy0) %} with the following parameters:
>    - In *"Input/Output options"*:
>        - *"The bakta database"*: latest one
>        - *"The amrfinderplus database"*: latest one
>        - {% icon param-file %} *"Select genome in fasta format"*: Contig file
>    - In *"Optional annotation"*:
>        - *"Keep original contig header (--keep-contig-headers)"*: `Yes`
{: .hands_on}

> <question-title></question-title>
>
> How many features have been identified?
>
> > <solution-title></solution-title>
> >
> > 51k+ (number of lines in the GFF)
> >
> {: .solution}
{: .question}

</div>

<div class="Yes" markdown="1">

If you already ran **Bakta** on your contigs, you can just upload here the GFF generated by **Bakta**.

> <hands-on-title>Import annotation GFF</hands-on-title>
>
> 1. {% tool [Import](upload1) %} the GFF file generated by **Bakta**
>
{: .hands_on}

![Bakta - circular plot of the annotation, with the contigs on the outside ordered clockwise by length size.](./images/bakta.svg)

</div>

## Mapping the raw reads on the contigs

To get information about the coverage of the contigs and genes, we map the reads used to build the contigs to the contigs using **Bowtie2** ({% cite langmead2009ultrafast %}).

> <hands-on-title>Annotating the Genome</hands-on-title>
> 1. {% tool [Import](upload1) %} the reads (after quality control) from another history, from [Zenodo]({{ page.zenodo_link }}) or from Galaxy shared data libraries:
>
>    ```
>    {{ page.zenodo_link }}/files/DRR187559_after_fastp_1.fastq.gz
>    {{ page.zenodo_link }}/files/DRR187559_after_fastp_2.fastq.gz
>    ```
>    
> 2. Create a paired collection named `Paired Reads`
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
> 3. {% tool [Bowtie2](toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.4+galaxy0) %} with the following parameters:
>    - *"Is this single or paired library"*: `Paired-end`
>        - {% icon param-collection %} *"FASTQ Paired Dataset"*: `Paired Reads`
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from the history and build index`
>        - {% icon param-file %} *"Select reference genome"*: Contig file
>    - *"Save the bowtie2 mapping statistics to the history"*: `Yes`
>
{: .hands_on}

> <question-title></question-title>
>
> What is the alignment rate? What does that mean for the assembly?
>
> > <solution-title></solution-title>
> >
> > The alignment rate is 99.77%. Most of the reads have been mapped on the contigs. They have been then used to build the contigs.
> >
> {: .solution}
{: .question}

## Visualisation of the ARGs

We can now visualize the contigs, the mapping coverage, and the genes, using JBrowse and different information track.

> <hands-on-title>Visualize the Genome</hands-on-title>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: Contig file
>    - *"Genetic Code"*: `11. The Bacterial, Archaeal and Plant Plastid Code`
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Bakta`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `Annotation_and_sequences` (output of **Bakta** {% icon tool %})
>                        - *"JBrowse Track Type [Advanced]"*: `Neat Canvas Features`
>                        - *"Track Visibility"*: `On for new users`
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `ARGs and plasmid genes`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: Output of **Table to GFF3**
>                        - *"JBrowse Track Type [Advanced]"*: `Neat Canvas Features`
>                        - *"Track Visibility"*: `On for new users`
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Coverage`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BAM Pileups`
>                        - {% icon param-file %} *"BAM Track Data"*: Bowtie2's output
>                        - *"Autogenerate SNP Track"*: `Yes`
>
> 3. View the output of JBrowse
>
{: .hands_on}

In the output of the JBrowse you can view the mapped reads and the found genes against the reference genome. With the search tools you can easily find genes of interest. JBrowse can handle many inputs and can be very useful. Using the Bowtie2 mapping output, low coverage regions can be detected. This SNP detection can also give a clear view of where the data was less reliable or where variations were located.

If it takes too long to build the JBrowse instance, you can view an embedded one here. (**Warning**: feature name search will not work.)

{% snippet topics/visualisation/faqs/visualizations_jbrowse.html datadir="data" loc="contig00019:81..23760" tracks="DNA,878c908bab1edf92fb1577fad80f7bae_0,a7fe8304cddfe5fbececdc7a2bd7ed18_0," %}

> <question-title></question-title>
>
> 1. What is the name for the gene found by Bakta corresponding to the rep16 gene found by staramr?
> 2. What are the NCBI Protein id and UniRef id for aac(6')-aph(2'')?
>
> > <solution-title></solution-title>
> >
> > 1. By zooming at the beginning of the contig, the name for rep16 is DUF536 domain-containing protein
> > 2. aac(6')-aph(2'') is on contig00019:20311..21749. The corresponding gene found by Bakta is bifunctional aminoglycoside N-acetyltransferase AAC(6')-Ie/aminoglycoside O-phosphotransferase APH(2'')-Ia. When clicking on it, we can find the dbxref attributes:
> >   - NCBI Protein: WP_001028144.1
> >   - UniRef: UniRef50_P0A0C2
> {: .solution}
{: .question}

# Conclusion

In this tutorial, contigs were scaned for AMR and plasmid genes. The genes were then visualized in their genomic context after contig annotation.


topic: genome-annotation
tutorial: braker3
title: Genome annotation with Braker3
difficulty: intermediate
time_estimation: 8h
tools_json: []
requirements_json: []
tags_json: ["eukaryota", "Braker3", "jbrowse1", "biodiversity"]
content_hash: 592fa6887f72693b6782db7919205f11
last_modified: 2026-02-26T21:12:16.263825
zenodo_link: https://zenodo.org/records/14770765
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/braker3/tutorial.html

content:
Annotating the eukaryotic genome represents a somewhat more complex challenge than that of prokaryotes, mainly due to the generally larger size of eukaryotic genomes and their greater number of genes. This annotation can be carried out at different levels of precision, ranging from simple identification of coding and non-coding parts to detailed structural labeling, including for example the precise location of exons, introns and other regulatory elements.

In this tutorial we will use a software tool called Braker3 to annotate the genome sequence of a small eukaryote: [*Mucor mucedo*](https://en.wikipedia.org/wiki/Mucor_mucedo) (a fungal plant pathogen).

[Braker3](https://github.com/Gaius-Augustus/BRAKER) is an automated bioinformatics tool that uses RNA-seq and protein data to annotate genomes. It integrates GeneMark-ETP and AUGUSTUS software to predict genes with a high degree of precision. By combining the results of these two tools, Braker3 generates a final file containing gene annotations with strong extrinsic support (i.e. based on external experimental data).

Braker3 facilitates genome annotation by leveraging transcriptomic and protein data to produce more reliable and robust gene predictions.

In this tutorial, you will learn how to perform structural annotation of the genome and assess its quality. To simplify learning and reduce analysis time, we will use a lightweight dataset, which may result in a less complete annotation than in the context of an analysis under real conditions.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data upload

To annotate our genome using Braker3, we will use the following files:

- The **genome sequence** in fasta format. For best results, the sequence should be soft-masked beforehand. You can learn how to do it by following the [RepeatMasker tutorial]({% link topics/genome-annotation/tutorials/repeatmasker/tutorial.md %}). For this tutorial, we will try to annotate the genome assembled in the [Flye assembly tutorial]({% link topics/assembly/tutorials/flye-assembly/tutorial.md %}). The size of the genome has been reduced to reduce Braker3 execution time.
- Some **RNAseq data** in bam format. These reads are already aligned on the genome, and Braker3 will use it as evidences to annotate genes.
- A set of **protein sequences**, like UniProt/SwissProt. It is important to have good quality, curated sequences here, and the UniProt/SwissProt databank fits very well. For this tutorial, we have prepared a subset of this databank to speed up computing, but you should use UniProt/SwissProt for real life analysis.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/14770765/files/genome.fasta
>    https://zenodo.org/records/14770765/files/RNASeq.bam
>    https://zenodo.org/records/14770765/files/protein_sequences.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

# Preparing RNAseq data

In this tutorial, the alignments from RNA-seq are already prepared.
This data was obtained using the RNA STAR tool, as explained in detail in the [Funannotate]({% link topics/genome-annotation/tutorials/funannotate/tutorial.md %}) tutorial.

Note that when using Braker3 with STAR to align RNA-Seq data, the **--outSAMstrandField intronMotif** parameter must be added.
This parameter adds specific intron information to the alignment files (BAM).
This information is necessary for Braker3 to correctly understand and use the alignments to annotate genes.
Without this parameter, Braker3 may not function correctly or may produce incomplete results.

Here's how to run STAR with this specific parameter (**"Read alignement tags to include in the BAM output"** option), before annotating with Braker3:

> <hands-on-title></hands-on-title>
>
> 1. {% tool [RNA STAR](toolshed.g2.bx.psu.edu/repos/iuc/rgrnastar/rna_star/2.7.8a+galaxy0) %} with the following parameters:
>    - *"Single-end or paired-end reads"*: `Paired-end (as individual datasets)`
>        - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file, forward reads"*: `rnaseq_R1.fq.gz` (Input dataset)
>        - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file, reverse reads"*: `rnaseq_R2.fq.gz` (Input dataset)
>    - *"Custom or built-in reference genome"*: `Use reference genome from history and create temporary index`
>        - {% icon param-file %} *"Select a reference genome"*: `genome_masked.fasta` (Input dataset)
>        - *"Length of the SA pre-indexing string"*: `11`
>    - In *"BAM output format specification"*:
>        - In *"Read alignement tags to include in the BAM output"*:
>             - Select `XS (strand flag, see parameter help below)`
>
{: .hands_on}

# Structural annotation

We now have the following required inputs:

- A masked genome in FASTA format
- RNAseq sequence alignments in BAM format
- Protein annotation in FASTA format

With these files, We can run [**Braker3**](https://github.com/Gaius-Augustus/BRAKER) to perform the structural annotation of the genome.


> <hands-on-title>Genome annotation with Braker3</hands-on-title>
>
> 1. {% tool [Braker3](toolshed.g2.bx.psu.edu/repos/iuc/braker3/braker3/3.0.8+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Assembly to annotate"*: `genome_masked.fasta` (Input dataset)
>    - *"Species name"*: `Mucor mucedo`
>    - {% icon param-file %} *"RNA-seq mapped to genome to train Augustus/GeneMark"*: `rnaseq.bam` (Input dataset)
>    - {% icon param-file %} *"Proteins to map to genome"*: `SwissProt_subset.fasta` (Input dataset)
>    - {% icon param-file %} *"Fungal genome"*: select `Yes`
>    - *"Output format"*: `GFF3`
>
{: .hands_on}

> <comment-title>On parameters</comment-title>
>
> - If you are working with fungi, we recommend that you activate the **--fungus** parameter. This enables Braker3 to run the GeneMark-ETP algorithm with a branch point model.
{: .comment}

> <comment-title>Don't wait</comment-title>
>
> - This step will take a bit of time to run: it could take around 2 hours for this tutorial. While it runs, we can already schedule the following functional annotation steps. Galaxy will run them automatically as soon as the structural annotation is ready.
> - There are several reasons for this: the size of the genome, the quality and quantity of the RNA-Seq data, the supervised learning model for prediction, and more.
{: .comment}

Braker3 generates an output file in GTF format by default. But it is possible to generate the output file in GFF3 output. This file contains informations on gene locations, exons, introns, etc.
GFF3 files are generally preferred for complex and varied annotations, while GTF is often used for transcript annotations in RNA-seq studies.

The GFF3 format is a standard bioinformatics format for storing genome annotations. Each row describes a genomic entity, with columns detailing its identifier, location, score and other attributes.

## Evaluation with **Busco**

[BUSCO](http://busco.ezlab.org/) (Benchmarking Universal Single-Copy Orthologs) is a widely used tool to evaluate the quality of a genome assembly and annotation. By comparing genomes from
various related and distantly related species, the authors determined sets of ortholog genes that are present in single copy in (almost) all the species of a clade (Bacteria, Fungi, Plants, Insects, Mammalians, …).
Most of these genes are essential for the organism to live, and are expected to be found in any newly sequenced and annotated genome from the corresponding clade. Using this data, BUSCO evaluates the "completeness" of genome annotation by assessing the proportion of these essential genes (also named BUSCOs) found in a set of (predicted) transcript or protein sequences.

We want to run BUSCO on the protein sequences predicted from gene sequences of the Braker3 annotation.

So first generate these sequences:

> <hands-on-title>Extract protein sequences</hands-on-title>
>
> 1. {% tool [GFFread](toolshed.g2.bx.psu.edu/repos/devteam/gffread/gffread/2.2.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input GFF3 or GTF feature file"*: output of {% tool [Braker](toolshed.g2.bx.psu.edu/repos/iuc/braker3/braker3/3.0.8+galaxy0) %}
>    - In *"Reference Genome"* select: `From your history` (Input dataset)
>    - *"Genome Reference Fasta"*: `masked genome` (Input dataset)
>    - In *"Select fasta outputs"* select: `fasta file with spliced exons for each GFF transcript (-y)`
>    - *"full GFF attribute preservation (all attributes are shown)"*: `Yes`
>    - *"decode url encoded characters within attributes"*: `Yes`
>    - *"warn about duplicate transcript IDs and other potential problems with the given GFF/GTF records"*: `Yes`
>
{: .hands_on}

The parameters for running BUSCO on these protein sequences:

> <hands-on-title>BUSCO in proteome mode</hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Sequences to analyse"*: `gffread: pep.fa`
>    - *"Mode"*: `annotated gene sets (protein)`
>    - *"Auto-detect or select lineage?"*: `Select lineage`
>        - *"Lineage"*: `Mucorales`
>    - In *"Advanced Options"*:
>        - *"Which outputs should be generated"*: select all outputs
{: .hands_on}

Several output files are generated:

- short summary: statistical summary of the quality of genomic assembly or annotation, including total number of genes evaluated, percentage of complete genes, percentage of partial genes, etc.
- full table: list of universal orthologs found in the assembled or annotated genome, with information on their completeness, location in the genome, quality score, etc.
- missing buscos: list of orthologs not found in the genome, which may indicate gaps in assembly or annotation.
- summary image: graphics and visualizations to visually represent the results of the evaluation, such as bar charts showing the proportion of complete, partial and missing genes.
- GFF: contain information on gene locations, exons, introns, etc.

This gives information about the completeness of the Braker3 annotation. A good idea is to compare this first result with the one you get on the initial genome sequence,and see if the annotation tool found all the genes that BUSCO finds in the raw genome sequence.

The parameters for running BUSCO on the masked genome:

> <hands-on-title>BUSCO in genome mode</hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Sequences to analyse"*: `masked genome` (Input dataset)
>    - *"Mode"*: `Genome assemblies (DNA)`
>    - *"Auto-detect or select lineage?"*: `Select lineage`
>        - *"Lineage"*: `Mucorales`
>    - In *"Advanced Options"*:
>        - *"Which outputs should be generated"*: select all outputs
{: .hands_on}


> <comment-title>What can we deduce from these results?</comment-title>
>
> - Completeness: The results of 50.3% complete BUSCOs for the genome and 43.9% for proteins are relatively low. These results can be explained by the reduction in file size in order to speed up analyses with Braker3.
> - Duplication and fragmentation: The duplication rate is low and the proportion of fragmented genes is small, suggesting that the annotation is correct overall, but incomplete.
{: .comment}

## Evaluation with **OMArk**

[OMArk](https://github.com/DessimozLab/OMArk) is proteome quality assessment software.
It provides measures of proteome completeness, characterises the consistency of all
protein-coding genes with their homologues and identifies the presence of contamination
by other species. OMArk is based on the OMA orthology database, from which it exploits
orthology relationships, and on the OMAmer software for rapid placement of all proteins
in gene families.

OMArk's analysis is based on HOGs (Hierarchical Orthologous Groups), which play a central
role in its assessment of the completeness and coherence of gene sets. HOGs make it possible
to compare the genes of a given species with groups of orthologous genes conserved across a
taxonomic clade.

> <hands-on-title>OMArk on extracted protein sequences</hands-on-title>
>
> 1. {% tool [OMArk](toolshed.g2.bx.psu.edu/repos/iuc/omark/omark/0.3.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Protein sequences"*: `gffread: pep.fa`
>    - *"OMAmer database*: select `LUCA-v2.0.0`
>    - In *"Which outputs should be generated"*: select `Detailed summary`
>
{: .hands_on}

The OMArk tool generated an output file in .txt format containing detailed information on
the assessment of the completeness, consistency and species composition of the proteome
analysed. This report includes statistics on conserved genes, the proportion of duplications,
missing genes and the identification of reference lineages.

> <comment-title>What can we deduce from these results?</comment-title>
>
> - Number of conserved HOGs: OMArk has identified a set of 5622 HOGs which are thought to be conserved in the majority of species in the Mucorineae clade.
> - However, 54.43% of these HOGs were absent from the analysed proteome, which may indicate a potential incompleteness of the annotation. This observation could be linked to the reduction in the number of RNA sequences aligned on the genome when Braker was run, a choice made to optimise analysis time.
> - 80.27% of genes are complete, so the annotation is of good quality in terms of genomic completeness.
> - Number of proteins in the whole proteome: 5642. Of which 80.27% are present and 15.37% of the proteome does not share sufficient similarities with known gene families.
> - No contamination detected.
> - The OMArk analysis is based on the Mucorineae lineage, a more recent and specific clade than that used in the BUSCO assessment, which selected the Mucorales as the reference group.
{: .comment}

# Visualisation with a genome browser

You can visualize the annotation generated using a genomic browser like [JBrowse](https://jbrowse.org/jbrowse1.html).
This browser enables you to navigate along the chromosomes of the genome and view the structure of each predicted gene.

> <hands-on-title>JBrowse visualisation</hands-on-title>
>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: `genome_masked.fasta` (Input dataset)
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Annotation`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `gff3` (output of **Braker3** {% icon tool %})
{: .hands_on}

Click on the newly created dataset's eye to display it. You will see a JBrowse genome browser. You can have a look at the [JBrowse tutorial]({% link topics/visualisation/tutorials/jbrowse/tutorial.md %}) for a more in-depth description of JBrowse.

We have embedded a copy of the resulting JBrowse here, if something went wrong during one of the steps you can always just check this output:


{% capture jbtracks %}DNA%2Caa6aca08d6fd82b94c7a86712a632760_0{% endcapture %}
{% snippet topics/visualisation/faqs/visualizations_jbrowse.html datadir="data" tracks=jbtracks %}

Check that **Braker3 annotation** is selected in the top left-hand corner. If not, select it to view the annotation.

# Conclusion

Congratulations on reaching the end of this tutorial! You now know how to perform a structural annotation of a new eukaryotic genome, using Braker3. And you've learned how to evaluate its quality and how to visualize it using JBrowse.

If you'd like to complete this annotation, we recommend you to follow the tutorial on [functional annotation]({% link topics/genome-annotation/tutorials/functional/tutorial.md %}) with EggNOG Mapper and InterProScan. You can follow it with the protein sequences we generated earlier with gffread.


topic: genome-annotation
tutorial: apollo-euk
title: Refining Genome Annotations with Apollo (eukaryotes)
difficulty: intermediate
time_estimation: 3h
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"internal\""]
tags_json: ["gmod", "eukaryote", "cyoa", "jbrowse1", "apollo2", "biodiversity"]
content_hash: 4939d62b8c7fe392fc900c269bfbf481
last_modified: 2026-02-26T21:12:16.191223
zenodo_link: https://zenodo.org/record/6920962
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/apollo-euk/tutorial.html

content:
After automatically annotating your genome using [Funannotate](../funannotate/tutorial.html) or [Maker](../annotation-with-maker/tutorial.html) for example, it is important to visualize your results so you can understand what your organism looks like, and then to manually refine these annotations along with any additional data you might have. This process is most often done as part of a group, smaller organisms may be annotated individually though.

> <warning-title>Only works on UseGalaxy.eu</warning-title>
> Currently this tutorial requires an Apollo server to be deployed by the administrator. This will currently only work on UseGalaxy.eu, hopefully this list will expand in the future.
{: .warning}

[Apollo](https://github.com/gmod/apollo) {% cite Dunn2019 %} provides a platform to do this. It is a web-based, collaborative genome annotation editor. Think of it as "Google Docs" for genome annotation, multiple users can work together simultaneously to curate evidences and annotate a genome.

This demo is inspired by the [Apollo User's Guide](https://genomearchitect.readthedocs.io/en/latest/UsersGuide.html), which provides additional guidance.

> <comment-title>Ask your instructor!</comment-title>
> This tutorial shows you how to use Galaxy to set up a genome curation project by loading a new organism in Apollo. If you only want to focus on the usage of Apollo, your instructor might already have created for you an organism in Apollo. In this case select the `Use a pre-created organism` option below (you will skip a few steps in Galaxy), and select the organism prepared for you by your instructor.
>
> If you're doing this tutorial on your own, or if the instructor has not set up a specific Apollo organism for you, select `Create your own organism`.
{: .comment}

{% include _includes/cyoa-choices.html option1="Create your own organism" option2="Use a precreated organism" default="Create your own organism"
       text="You can decide to create an organism or use a pre-created one." %}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Preparing for Apollo

<div class="Create-your-own-organism" markdown="1">

## Data upload


To annotate a genome using Apollo, we need the reference genome sequence in FASTA format, and any evidence tracks we want to refine into our annotations. "Evidence tracks" can be any data like:

- A set of prior gene predictions or other genomic feature predictions
- The output of a bioinformatics analysis like BLAST or InterProScan
- Sequencing reads from RNA-Seq or another HTS analysis
- If you are not doing a *de novo* annotation, then a previous released {OGS}.

In this tutorial we use the same data as in the [Funannotate](../funannotate/tutorial.html) tutorial: the genome sequence, the annotation produced by **Funannotate** {% icon tool %}, and the mapping of RNA-Seq data along the genome using **RNA star** {% icon tool %}. We will visualise this data and use it to make some manual annotations in order to familiarise you with the process.

> <comment-title>Real Data: Unreal Circumstances</comment-title>
> While the data for this tutorial is sourced from publicly available databases, and is all related to different experiments on *Mucor mucedo*, this is not necessarily the data *you* might use to annotate your genomes. You probably know best what data you should be using in your own circumstances, for the specific features on which you are focused.
{: .comment}

### Get data


> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/record/6920962/files/genome.fasta
>    https://zenodo.org/record/6920962/files/annotation.gff3
>    https://zenodo.org/record/6920962/files/rnaseq.bam
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

## Using Apollo for Annotation


Refining genome annotations happens in multiple steps:

- Create a JBrowse instance from the reference genome FASTA file and evidence tracks
- Import this data into Apollo
- Refine the annotations
- Export the refined genome annotations

In this tutorial we will focus more on the practical portions than the theoretical part of genome annotation, that are covered in other tutorials. When you've completed this tutorial you should be comfortable manipulating genomic data in Galaxy and Apollo.

> <details-title>Why bother?</details-title>
>
> Automated annotation programs continue to improve, however a simple score may not provide evidence necessary to confirm an accurate prediction.
> Therefore, it is necessary to both visually inspect the results and manually fix any issues with the predictions.
>
> Additionally, many times assemblies are less than perfect or read depth and quality may be insufficient, leading to imperfect automatic annotation.
{: .details}

### Build the JBrowse Instance

Let's begin by building a JBrowse instance with all the data we have for this genome.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: `genome.fasta` (Input dataset)
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Annotation`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-files %} *"GFF/GFF3/BED Track Data"*: `annotation.gff3` (Input dataset)
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `RNA-Seq`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BAM Pileups`
>                        - {% icon param-files %} *"BAM Track Data"*: `rnaseq.bam` (Input dataset)
>                        - *"Autogenerate SNP Track"*: `yes`
>
>    > <comment-title>JBrowse is highly configurable</comment-title>
>    >
>    > JBrowse is highly configurable, we have set a very basic configuration but there are many more advanced features available to you, if you need them. You can choose precisely how data is displayed, and even what menu options are available when users click on features. If your features have some external identifiers like an NCBI Gene ID, you can even configure JBrowse that when the user clicks on the feature, it should show the gene page for that feature in a new tab. These sort of features are incredibly helpful for building very rich experiences.
>    >
>    > A static genome browser like this (just JBrowse, not in Apollo) is very useful for summarising results of a genomics workflow, where the next step is simply interpretation and not annotation. Have a look at the [JBrowse tutorial]({% link topics/visualisation/tutorials/jbrowse/tutorial.md %}) for more information.
>    >
>    > Currently we have built a standalone genome browser (data + the html page and user interface and javascript), but it's possible to just compile the data directory if you intend to send this data to Apollo, and don't need to view the static data in Galaxy.
>    {: .comment}
{: .hands_on}

This tool will take some time to run dependent on data size. All of the inputs need to be pre-processed by JBrowse into a form that it can render and visualise easily. Once this is complete, you can click on the {% icon galaxy-eye %} eyeball to view the JBrowse instance. This is a static view into the data, JBrowse does not let you make any annotations or save any changes. We will convert it into a dynamic view where we can make persistent annotations and share these with our colleagues.

### Sending data to Apollo

Now that we have a good looking static JBrowse instance, it is time to load it into Apollo to turn it into a dynamic view where you can make modifications to the genes.

> <hands-on-title>Import to Apollo</hands-on-title>
>
> 1. {% tool [Create or Update Organism](toolshed.g2.bx.psu.edu/repos/gga/apollo_create_or_update/create_or_update/4.2.5) %} with the following parameters:
>    - {% icon param-file %} *"JBrowse HTML Output"*: output of **JBrowse** {% icon tool %}
>    - *"Organism Common Name Source"*: `Direct Entry`
>        - *"Organism Common Name"*: `Mucor mucedo`
>    - *"Genus"*: `Mucor`
>    - *"Species"*: `mucedo`
>
> 2. {% tool [Annotate](toolshed.g2.bx.psu.edu/repos/gga/apollo_iframe/iframe/4.2.5) %} with the following parameters:
>    - {% icon param-file %} *"Apollo Organism Listing"*: output of **Create or Update Organism** {% icon tool %}
>
> 3. View {% icon galaxy-eye %} the output of the Annotate tool, when it is ready.
>
{: .hands_on}

Viewing the output will open a view into Apollo in the main panel. Here you can interact with your genome and make annotations. This "Annotate" output is a quick link to that specific genome, and while Apollo allows you to manage and annotate multiple genomes, this dataset will always take you back to that specific genome. You can additionally access the Apollo server outside of Galaxy. While the URL will be different for each Galaxy server that supports Apollo, UseGalaxy.eu's Apollo server is available at [https://usegalaxy.eu/apollo](https://usegalaxy.eu/apollo).

</div>
<div class="Use-a-precreated-organism" markdown="1">

While the URL will be different for each Galaxy server that supports Apollo, to access UseGalaxy.eu's Apollo server:

- First login on [https://usegalaxy.eu](https://usegalaxy.eu)
- Then go to [https://usegalaxy.eu/apollo](https://usegalaxy.eu/apollo).

It is important to first login on [https://usegalaxy.eu](https://usegalaxy.eu) before connecting to Apollo as it will set up everything to make Apollo recognize your user account.

Your instructor should give you the name of the organism you will work on, you need to select it in the drop down box at the top of the right panel of Apollo. If multiple users are using the same organism, you will experiment the true collaborative aspect of Apollo: all the actions performed on the gene models by anyone will be seen by everyone instantly.
</div>
# Apollo
<div class="Create-your-own-organism" markdown="1">

![Screenshot of Apollo in Galaxy, apollo takes up most of the space in the center iframe with the history shown on the right.](../../images/apollo/ui.png "This is the Apollo interface, when viewed from inside Galaxy. The Annotate tool produces a special output which acts like a link to the Apollo server. It is <b>not</b> sufficient to download this file to save a copy of your annotations, unlike other datasets in Galaxy. In the main panel Apollo is displayed, with two panels of its own: the Annotation Window (left), and the Annotator Panel (right). The annotation window is the main view into our genome and here we will see evidence, reconcile it, and make our annotations. The right panel allows us to show or hide individual evidence tracks, switch between organisms, and navigate around the genome.")

</div>
<div class="Use-a-precreated-organism" markdown="1">

![Screenshot of Apollo.](../../images/apollo/ui_apollo.png "This is the Apollo interface, with two panels of its own: the Annotation Window (left), and the Annotator Panel (right). The annotation window is the main view into our genome and here we will see evidence, reconcile it, and make our annotations. The right panel allows us to show or hide individual evidence tracks, switch between organisms, and navigate around the genome.")

> <details-title>Organism naming</details-title>
>
> At the top of the right panel, you can switch from one organism to another. Depending on the server you are using, the organism names can be suffixed with `gx[number]` or an email address: this is to avoid possible confusion between organisms with the same name created by multiple users.
{: .details}

</div>

From the Apollo user manual:

> The major steps of manual annotation using Apollo can be summarized as follows:
>
> 1. Locate a chromosomal region of interest.
> 2. Determine whether a feature in an existing evidence track provides a reasonable gene model to start annotating.
> 3. Drag the selected feature to the ‘User Annotation’ area, creating an initial gene model.
> 4. Use editing functions to edit the gene model if necessary.
> 5. Check your edited gene model for consistency with existing homologs by exporting the FASTA formatted sequence and searching a protein sequence database, such as UniProt or the NCBI Non Redundant (NR) database, and by conducting preliminary functional assignments using the Gene Ontology (GO) database.
>
{: .quote}

The first four steps are generally the process of structural annotation (the process of identifying the correct gene model), and the last includes functional annotation (the process of assigning a putative function to a gene in your annotations).

## Evidence tracks

Let's start by looking at the tracks available to us, and then turning on the annotation track so we can start exploring our data.

> <hands-on-title>Visualize the annotation</hands-on-title>
>
> 1. In the right hand panel at the top click on **Tracks** to open the track listing
>
>    ![Zoomed in screenshot of the track menu with the Annotation set of tracks expanded showing Augustus and NCBI AnnotWriter genes.](../../images/apollo/tracks_eu.png)
>
> 2. In the **Annotation** group, select the `annotation.gff3` track.
>
>    You can either activate tracks in bulk, by clicking on the checkbox to the right of the group name ("Annotation"), or by clicking on the group name to expand the section, and then selecting individual tracks.
>
> 3. Zoom to a region of the genome.
>
>    1. In the left hand Annotation Window, at the top navigation bar you will find a textbox which shows the current location on the genome.
>    2. Edit this and enter `scaffold_1:3000..12000`
>    3. Press *Go* or use <kbd>Enter</kbd> on your keyboard.
>
>    ![Screenshot of Jbrowse header with the control panel with magnifying glasses and zoom location.](../../images/apollo/location_euk.png)
>
{: .hands_on}

> <comment-title>Track names</comment-title>
> Apollo uses the dataset names from your Galaxy history to name the tracks in the track list.
{: .comment}

We can now see an evidence track: `annotation.gff3` is the output of the [Funannotate tutorial](../funannotate/tutorial.html). In a *de novo* annotation project, we probably will only have the outputs of various gene callers, and potentially some expression evidence like RNA-Seq.

For this organism, we only have one track in the **Annotation** group, but an organism can have multiple annotations coming from different sources.

## Adding new genes

With the selected track, if you look along the genome, you will see many genes that were predicted by Funannotate. Each of them as a name assigned by Funannotate based on automatic functional annotation. If you right click on gene, an select `View details`, you can get access to detailed information on the predicted gene, including its unique name (e.g. `FUN_000002`), functional annotation (e.g. `InterPro:IPR006594`, `GO:0005515`) or the coding sequence (CDS).

> <details-title>Gene colors</details-title>
> Each gene color corresponds to an open reading frame on the genome. This allows to quickly see if two genes that are very close are on the same open reading frame, which could mean that they can be merged into a single gene, if other evidences support this.
{: .details}

What we want now is first to check that the structure of the genes predicted by Funannotate are in good shape. To do this, we will display an additional evidence tracks.

> <hands-on-title>Display RNA-Seq track</hands-on-title>
>
> 1. In the right hand panel at the top click on **Tracks** to open the track listing
>
> 2. In the **RNA-Seq** group, select the `rnaseq.bam - SNPs/Coverage` track and then the `rnaseq.bam` track.
>
{: .hands_on}

These new tracks represent the result of aligning RNA-Seq reads along the genome, with two different representation. It should look like that:

![JBrowse with RNA-Seq data shown, genes appear as red green and blue stripes with a pointed end, RNA-Seq appear as a coverage plot along the genome, and as individual reads as red and blue stripes](../../images/apollo/rnaseq_euk.png)

The first track (`rnaseq.bam - SNPs/Coverage`) is a coverage xy-plot, representing the number of reads aligned on the genome at each position. The little coloured teardrops represent positions where mismatches where found between some reads and the reference sequence (SNPs or sequenceing errors). Light grey regions corresponding to portions of the genomes where RNASeq were mapped, and darker grey regions corresponding to introns (= regions were some reads were found to match both the end of an exon, and the start of the next one).

The second track (`rnaseq.bam`) represent the alignment of each individual RNA-Seq read. Red reads were aligned in the forward strand, and blue ones in the reverse strand.

Let's suppose you want to improve the annotation of one of the gene. To do it, you can either click on one of its intron and drag and drop it to the User-created Annotations track (yellow background, at the top), or if you prefer, right click on one of its intron and select **Create new annotation** > **gene**. It should appear shortly in the **User-created Annotation** track.

<div class="Use-a-precreated-organism" markdown="1">
> <warning-title>Working with others</warning-title>
> Maybe you're not doing this tutorial alone, if you see that other people have started working on this region of the genome, it is best to move to a different region to avoid disturbing them by modifying the same genes at the same time.
{: .warning}
</div>

![One gene is added to the User-created Annotation track](../../images/apollo/added_genes_euk.png)

You can also see that it appears now in the list of genes in the right panel, at the gene and the mRNA level (as a gene can have multiple isoforms).

![Gene list showing gene FUN_000005 which is expanded to reveal a child mRNA](../../images/apollo/gene_list_euk.png)

The **User-created Annotation** track is where you can make modifications to genes, like changing their coordinates, or their name and functional annotation. If you right click on a gene in this track, you will see all the possibilities offered by apollo.

![Gene contextual menu showing way too many options, the mouse hovers over Open Annotation (alt-click)](../../images/apollo/gene_menu_euk.png)

Currently, the gene we added has a meaningless name. Let's improve that: right click on the gene, and click on **Open Annotation (alt-click)**.

![Gene details menu in apollo showing type set to gene, the name field has been filled with FUN_000004-T1 and a number of other empty fields like symbol, aliases, description.](../../images/apollo/gene_details_euk.png)

One way to give a better name is to find a similarity with genes of another species: if you right click on the gene, you can get the protein sequence by clicking on **Get Sequence**, and then compare it with a sequence database with Blast for example (using Galaxy, or the [NCBI blast form](https://blast.ncbi.nlm.nih.gov/Blast.cgi) for example).

In this case, the blast result suggests that this name is similar to `alpha/beta-hydrolase` genes in other fungi species. This is not a very strong evidence, but for this exercise, let's use this as a name: write `alpha/beta-hydrolase` in the **Name** field (type it manually and select it from the drop down list).

> <details-title>Naming genes</details-title>
>
> Giving a proper name to a gene is not always easy. Should it include "Putative" or not? What if multiple names can apply? Should it be lowercase or uppercase?
> The important thing is to always use the same naming rules when working on a full annotation, and to agree on these rules with other collaborators.
> Usually, big annotation consortiums have naming guidelines that you are supposed to follow.
{: .details}

We have just edited the gene name, but Apollo allows to edit information at the mRNA level too. Click on the **Sync name with transcript** button to copy the gene name to the mRNA name. It should now display in the **User-created Annotation** track. To check what you can edit at the mRNA level, just click on the corresponding mRNA in the list above:

![very zoomed screenshot showing clicking on the transcript below the gene to open the mRNA info.](../../images/apollo/mrna_level_euk.png)

You should see `alpha/beta-hydrolase` in the **Name** field.

> <comment-title>Saving your work</comment-title>
>
> You do not need to do anything specific to *Save* your work in Apollo. Just like Google Docs, each modification is immediately saved, and any other user working on the same genome will instantly see the changes you make.
{: .comment}

## Editing a gene structure

Apollo allows to edit the whole structure of a gene. If you zoom to the 5' end of the gene, you will notice two things:

- a few RNA-Seq reads were mapped a little before the 5' limit of our gene
- there is another potential start codon a few bases after the one that was predicted by Funannotate

Let's modify the 5' of the gene to add a 5' untranslated region. Although we don't have enough evidences to be sure that the start codon should be changed, for this exercise will we do it too. To do it, all you need to do is click on the 5' limit of the gene and drag it to the desired position. You will notice that the structure of the gene will be shortly changed. Changing the start codon can be done by right clicking on the new position and clicking on "Set Translation Start"

![Gif showing modifying the 5' region of a gene by dragging it.](../../images/apollo/change_start_euk.gif)

The new white region corresponds to the newly create 5' untranslated region.

This kind of modifications is very common when using Apollo, and you can perform it at the gene level, and at the exon/intron level (an exclaim mark appears when Apollo detects that an exon/intron junction doesn't match with the cannonical acceptor/donor sequences).

![Gif showing modifying a exon/intron junction of a gene by dragging it.](../../images/apollo/exon_edit.gif)

To guide you doing these changes, you should look at all the tracks available for the genome you study. RNA-Seq track are very helpful to determine the limits of coding sequences on the genome. Other tracks can be helpful, like alignements of transcripts or proteins from closely related species (or even big databanks like Swissprot or NR).

## Working with isoforms

In the eukaryote world, with the process of alternative splicing, multiple mRNA sequences (isoforms) can be generated for a single gene, based on the use of various splice site combinations. This process has a complex regulation mechanism allowing to express or not each isoform depending on multiple conditions (tissue, developmental stage, environmental conditions, ...).

Often automatic annotation softwares produce gene models composed of many exons, but does not give the detail of the sequence of each potential isoforms. However, in some cases, the RNA-Seq data and the scientific knowledge on specific gene families can be used to annotate each isoform of a gene.

Apollo allows to do that: just duplicate an existing gene model at the same position. Each copy will be considered as an isoform of the same gene. You can then make all the modifications you want to each one independently.

![Gif showing creating two isoforms and removing exons from them.](../../images/apollo/isoforms.gif)

The isoforms will appear in the gene list, under their common gene.

![Gene list showing the three isoforms of the same gene.](../../images/apollo/isoform_list.png)

As isoforms regulation depends heavily on multiple conditions (tissue, developmental stage, environmental conditions, ...), it is important when annotating a new genome to have access to a lot of RNA-Seq data sequenced in multiple conditions and from different tissues.

## Viewing and reverting changes

Everything you do in Apollo is tracked in a database. If you right click on the gene, and select **Show History**, you have access to the full list of all the actions that were performed on it.

![History of changes on a gene showing many changes to Set Name and a final Set Exon Boundaries change](../../images/apollo/history.png)

When you click on one of the steps, you can see below the list a preview of how the gene looked at the time. And you can revert to a specific version of the gene by clicking on the arrow button on the right.

## Adding more functional annotation

Sometimes you'll want to modify a gene that was predicted by FunAnnotate, just to add functional annotation to it, like Gene Ontology terms, of references to external databases (UniProt, InterPro, ...). This can be done using Apollo, have a look at the [Apollo tutorial for prokaryotes]({% link topics/genome-annotation/tutorials/apollo/tutorial.md %}) for more details.

# Exporting and collaborating

<div class="Create-your-own-organism" markdown="1">
{% include topics/genome-annotation/tutorials/apollo/collab.md eukaryote=true %}
</div>
<div class="Use-a-precreated-organism" markdown="1">
It is of course possible to export all your annotation from Apollo in standard formats, directly from its interface (`Ref Sequence` tab), or from Galaxy. If you create your own organism in Apollo, you can choose to make it open to the public, or to restrict it to only your collaborators.

Click on **Create your own organism** at the beginning to learn more.

</div>

# Conclusion


Congratulations, you finished this tutorial! By using Apollo and JBrowse, you learned how to manually refine predicted annotations<span class="Create-your-own-organism"> and export them to Galaxy for future analyses. You also learn how to give access to your project at any other researcher, making it a real collaborative solution</span><span class="Use-a-pre-created-organism">. Although you use a pre-created organism, remember that you can use Galaxy to add your own organism to Apollo, and give access to your project at any other researcher, making it a real collaborative solution</span>.

A [similar tutorial for prokaryote genomes]({% link topics/genome-annotation/tutorials/apollo/tutorial.md %}) exists, using different types of evidence tracks, feel free to have a look at it to learn more.

When refinement is sufficient an updated or new version of the annotation may be exported as GFF3 as well as published as a new JBrowse directory for inspection.

# What's next?

After generating your refined annotation, you'll want to merge it back into the official gene sets. A future tutorial will show you how to do it within Galaxy.

If a *de novo* set, you can export it as GFF3 and load it into a tool like [Tripal](http://tripal.info) to provide visualization.


topic: genome-annotation
tutorial: annotation-with-maker-short
title: Genome annotation with Maker (short)
difficulty: intermediate
time_estimation: 2h
content: {% include topics/genome-annotation/tutorials/annotation-with-maker/content.md short=true %}
tools_json: []
requirements_json: []
tags_json: ["gmod", "eukaryote", "maker", "jbrowse1"]
content_hash: f1a705520163589b3c417cb47f67fbad
last_modified: 2026-02-26T21:12:16.188340
zenodo_link: https://doi.org/10.5281/zenodo.4406623
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/annotation-with-maker-short/tutorial.html


topic: genome-annotation
tutorial: helixer
title: Genome annotation with Helixer
difficulty: intermediate
time_estimation: 4h
tools_json: []
requirements_json: []
tags_json: ["eukaryota", "helixer", "jbrowse1", "biodiversity"]
content_hash: 99a04e80a479d24ba096af6a79dfe38a
last_modified: 2026-02-26T21:12:16.596933
zenodo_link: https://zenodo.org/record/7867921
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/helixer/tutorial.html

content:
Annotating the eukaryotic genome represents a somewhat more complex challenge than that of prokaryotes, mainly due to the generally larger size of eukaryotic genomes and their greater number of genes, but also to the complexity of eukaryotic genes structure (e.g. exons and {UTR}). This annotation can be carried out at different levels of precision, ranging from simple identification of coding and non-coding parts to detailed structural labeling, including for example the precise location of exons, introns and other regulatory elements.


In this tutorial we will use a software tool called Helixer to annotate the genome sequence of a small eukaryote: [*Mucor mucedo*](https://en.wikipedia.org/wiki/Mucor_mucedo) (a fungal plant pathogen).

[Helixer](https://github.com/weberlab-hhu/Helixer) is an annotation software with a new and different approach: it performs evidence-free predictions (no need for RNASeq data or sequence aligments), using {GPU}, with a much faster execution time. The annotation is based on the development and use of a cross-species deep learning model. The software is used to configure and train models for *ab initio* prediction of gene structure. In other words, it identifies the base pairs in a genome that belong to the UTR/CDS/Intron genes.

In this tutorial, you'll learn how to perform a structural annotation of the genome and how to assess its quality.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data upload

To annotate our genome using Helixer, we will use the following files:

- The **genome sequence** in fasta format. For this tutorial, we will try to annotate the genome assembled in the [Flye assembly tutorial]({% link topics/assembly/tutorials/flye-assembly/tutorial.md %}). (Note: Helixer will ignore soft-masking. Hard-masking is not recommnded for Helixer either, as it does not ignore the hard-masked regions, but will get less information from them, which could influence your predictions in a negative way.)

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/record/7867921/files/genome_masked.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

# Structural annotation

We can run [**Helixer**](https://github.com/weberlab-hhu/Helixer) to perform the structural annotation of the genome.

We need to input the genome sequence we want to annotate.

We also need to choose between 4 different lineages: *invertebrate*, *vertebrate*, *land plant* or *fungi*. Select the one that fits the best to the species you're studying: *fungi* in our case. Helixer is shipped with these 4 models that were trained specifically to annotate genes from each of these lineages. Advanced users can upload their own lineage model in .h5 format with the *"Lineage model"* option.

As an option, we can also enter a species name.


> <hands-on-title></hands-on-title>
>
> 1. {% tool [Helixer](toolshed.g2.bx.psu.edu/repos/genouest/helixer/helixer/0.3.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Genomic sequence"*: `genome_masked.fasta` (Input dataset)
>    - In *"Available lineages"*: "*select*" `fungi`
>    - In *"Species name"*: `Mucor mucedo`
>
{: .hands_on}

> <comment-title>Advanced parameters</comment-title>
>
> Depending on the lineage, the parameters *"Subsequence length"*, *"Overlap offset"* and *"Overlap corelength"* are adjusted to corresponding default values (listed in the help of each option).
>
> This is due in particular to the size of the genomes. Indeed, it is recommended to increase the value of *"Subsequence length"* for genomes containing large genes. This is particularly important for vertebrates and invertebrates.
>
> The default values used by Galaxy are the ones recommended by Helixer authors. If you wish to modify these default values, you can do so by entering your values in the *"Subsequence length"*, *"Overlap offset"* and *"Overlap corelength"* parameters.
>
{: .comment}

> <comment-title>Don't wait</comment-title>
>
> This step can take a bit of time to run: although Helixer runs much faster than many other annotation tools (typically <20min for this tutorial), it requires specific hardware (GPU) that is often available in limited quantity on computing systems. It means your job can be placed in queue for a longer time than a more standard Galaxy job.
>
> While it runs, we can already schedule the following steps. Galaxy will run them automatically as soon as the Helixer annotation is ready.
{: .comment}

Helixer produces a single output dataset: a GFF3 file. The GFF3 format is a standard bioinformatics format for storing genome annotations. Each row describes a genomic entity, with columns detailing its identifier, location, score and other attributes.

# Quality evaluation

## General statistics

Genome Annotation Statistics is a program designed to analyze and provide statistics on genomic annotations. This software performs its analyses from a GFF3 file.

> <hands-on-title>Genome Annotation Statistics</hands-on-title>
>
> 1. {% tool [Genome Annotation Statistics](toolshed.g2.bx.psu.edu/view/iuc/jcvi_gff_stats/jcvi_gff_stats/0.8.4) %} with the following parameters:
>    - {% icon param-file %} *"Annotation to analyse "*: `GFF3 file` (Output of **Helixer**)
>    - In *"Reference genome"*: select `Use a genome from history`
>    - In *"Corresponding genome sequence"*: `genome_masked.fasta` (Input dataset)
{: .hands_on}

Two output files are generated:
- a file containing graphs in pdf format
- a summary in txt format

> <comment-title>What can we deduce from these results?</comment-title>
>
> - The summary file provides statistics on the genome annotation and gives a complete overview of the genomic structure and characteristics of the genes, exons and introns in the analysed genome.
> - We can see that there are 19,299 genes, 77% of which are multi-exons (i.e. 14,860) and 23% single-exons (i.e. 4,439).
> - We can obtain other information such as the average size of exons, the percentage in GC or the average size of transcripts.
>
{: .comment}

These statistics are interesting on their own: you often have a rough idea of the expected number of genes or mean length when annotating a new genome, by comparing with published similar species. You can also use them to compare the quality of annotations produced by different tools.

## Evaluation with **Busco**

[BUSCO](http://busco.ezlab.org/) (Benchmarking Universal Single-Copy Orthologs) is a tool allowing to evaluate the quality of a genome assembly or of a genome annotation. By comparing genomes from various more or less related species, the authors determined sets of ortholog genes that are present in single copy in (almost) all the species of a clade (Bacteria, Fungi, Plants, Insects, Mammalians, ...). Most of these genes are essential for the organism to live, and are expected to be found in any newly sequenced and annotated genome from the corresponding clade. Using this data, BUSCO is able to evaluate the proportion of these essential genes (also named BUSCOs) found in a set of (predicted) transcript or protein sequences. This is a good evaluation of the "completeness" of the annotation.

As an alternative for genomes only one can use [**compleasm**](https://github.com/huangnengCSU/compleasm) with the same BUSCO gene sets, as compleasm is a bit more sensitive and thus allows finding slightly more conserved genes. 

We want to run BUSCO on the protein sequences predicted from gene sequences of the Helixer annotation. So first generate these sequences:

> <hands-on-title>Extract protein sequences</hands-on-title>
>
> 1. {% tool [GFFread](toolshed.g2.bx.psu.edu/repos/devteam/gffread/gffread/2.2.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input GFF3 or GTF feature file"*: output of {% tool [Helixer](toolshed.g2.bx.psu.edu/repos/genouest/helixer/helixer/0.3.3+galaxy1)) %}
>    - In *"Reference Genome"* select: `From your history` (Input dataset)
>    - *"Genome Reference Fasta"*: `masked genome` (Input dataset)
>    - In *"Select fasta outputs"* select: `protein fasta file with the translation of CDS for each record (-y)`
>    - *"full GFF attribute preservation (all attributes are shown)"*: `Yes`
>    - *"decode url encoded characters within attributes"*: `Yes`
>    - *"warn about duplicate transcript IDs and other potential problems with the given GFF/GTF records"*: `Yes`
>
{: .hands_on}

To run BUSCO on these protein sequences:

> <hands-on-title>BUSCO in proteome mode</hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Sequences to analyse"*: `gffread: pep.fa`
>    - *"Mode"*: `annotated gene sets (protein)`
>    - *"Auto-detect or select lineage?"*: `Select lineage`
>        - *"Lineage"*: `Mucorales`
>    - In *"Advanced Options"*:
>        - *"Which outputs should be generated"*: select all outputs
>
{: .hands_on}

Several output files are generated:

- short summary : statistical summary of the quality of genomic assembly or annotation, including total number of genes evaluated, percentage of complete genes, percentage of partial genes, etc.
- full table : list of universal orthologs found in the assembled or annotated genome, with information on their completeness, location in the genome, quality score, etc.
- missing buscos : list of orthologs not found in the genome, which may indicate gaps in assembly or annotation.
- summary image : graphics and visualizations to visually represent the results of the evaluation, such as bar charts showing the proportion of complete, partial and missing genes.
- GFF : contain information on gene locations, exons, introns, etc.

This gives information about the completeness of the Helixer annotation. A good idea is to compare this first result with the one you get on the initial genome sequence, and see if the annotation tool found all the genes that BUSCO finds in the raw genome sequence. So run BUSCO in genome mode:

> <hands-on-title>BUSCO in genome mode</hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Sequences to analyse"*: `masked genome` (Input dataset)
>    - *"Mode"*: `Genome assemblies (DNA)`
>    - *"Auto-detect or select lineage?"*: `Select lineage`
>        - *"Lineage"*: `Mucorales`
>    - In *"Advanced Options"*:
>        - *"Which outputs should be generated"*: select all outputs
>
{: .hands_on}

> <comment-title>What can we deduce from these results?</comment-title>
>
> - 94.6% of genes are complete, so the annotation is of high quality in terms of genomic completeness.
> - It is a little bit lower than what BUSCO is able to find in genome mode (95.7%), but the difference is quite small so Helixer seems to have generated a quite good result.
> - The duplication rate is low, with 1.3% and 1.5% of genes duplicated.
> - So the Helixer annotation looks like a good one, with high completeness and low duplication.
>
{: .comment}

## Evaluation with **OMArk**

[OMArk](https://github.com/DessimozLab/OMArk) is proteome quality assessment software. It provides measures of proteome completeness, characterises the consistency of all protein-coding genes with their homologues and identifies the presence of contamination by other species. OMArk is based on the OMA orthology database, from which it exploits orthology relationships, and on the OMAmer software for rapid placement of all proteins in gene families.

OMArk's analysis is based on HOGs (Hierarchical Orthologous Groups), which play a central role in its assessment of the completeness and coherence of gene sets. HOGs make it possible to compare the genes of a given species with groups of orthologous genes conserved across a taxonomic clade. 

> <hands-on-title>OMArk on extracted protein sequences</hands-on-title>
>
> 1. {% tool [OMArk](toolshed.g2.bx.psu.edu/repos/iuc/omark/omark/0.3.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Protein sequences"*: `gffread: pep.fa`
>    - *"OMAmer database*: select `LUCA-v2.0.0`
>    - In *"Which outputs should be generated"*: select `Detailed summary`
>
{: .hands_on}

The OMArk tool generated an output file in .txt format containing detailed information on the assessment of the completeness, consistency and species composition of the proteome analysed. This report includes statistics on conserved genes, the proportion of duplications, missing genes and the identification of reference lineages.

> <comment-title>What can we deduce from these results?</comment-title>
>
> - Number of conserved HOGs: OMArk has identified a set of 5622 HOGs which are thought to be conserved in the majority of species in the Mucorineae clade.
> - 85.52% of genes are complete, so the annotation is of good quality in terms of genomic completeness.
> - Number of proteins in the whole proteome: 19 299. Of which 62.83% are present and 30.94% of the proteome does not share sufficient similarities with known gene families.
> - No contamination detected.
> - The OMArk analysis is based on the Mucorineae lineage, a more recent and specific clade than that used in the BUSCO assessment, which selected the Mucorales as the reference group.
{: .comment}




# Visualisation with a genome browser

You can visualize the annotation generated using a genomic browser like [JBrowse](https://jbrowse.org/jbrowse1.html). This browser enables you to navigate along the chromosomes of the genome and view the structure of each predicted gene.

> <hands-on-title>JBrowse visualisation</hands-on-title>
>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: `genome_masked.fasta` (Input dataset)
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Annotation`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `gff3` (output of **Helixer** {% icon tool %})
{: .hands_on}

Click on the newly created dataset's eye to display it. You will see a JBrowse genome browser. You can have a look at the [JBrowse tutorial]({% link topics/visualisation/tutorials/jbrowse/tutorial.md %}) for a more in-depth description of JBrowse.

# Conclusion

Congratulations on reaching the end of this tutorial! You now know how to perform a structural annotation of a new eukaryotic genome, using Helixer. And you've learned how to evaluate its quality and how to visualize it using JBrowse.

If you'd like to complete this annotation, we recommend you to follow the tutorial on [functional annotation]({% link topics/genome-annotation/tutorials/functional/tutorial.md %}) with EggNOG Mapper and InterProScan. You can follow it with the protein sequences we generated earlier with gffread.


topic: genome-annotation
tutorial: apollo
title: Refining Genome Annotations with Apollo (prokaryotes)
difficulty: intermediate
time_estimation: 3h
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"internal\""]
tags_json: ["gmod", "prokaryote", "microgalaxy", "jbrowse1", "biodiversity", "apollo2"]
content_hash: 7ccc4f8efe118d1e3e255bcea72f3dca
last_modified: 2026-02-26T21:12:16.191946
zenodo_link: https://zenodo.org/record/4889110
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/apollo/tutorial.html

content:
After automatically annotating your genome using [Prokka](../annotation-with-prokka/tutorial.html) for example, it is important to visualize your results so you can understand what your organism looks like, and then to manually refine these annotations along with any additional data you might have. This process is most often done as part of a group, smaller organisms may be annotated individually though.

> <warning-title>Only works on UseGalaxy.eu</warning-title>
> Currently this tutorial requires an Apollo server to be deployed by the administrator. This will currently only work on UseGalaxy.eu, hopefully this list will expand in the future.
{: .warning}

[Apollo](https://github.com/gmod/apollo) {% cite Dunn2019 %} provides a platform to do this. It is a web-based, collaborative genome annotation editor. Think of it as "Google Docs" for genome annotation, multiple users can work together simultaneously to curate evidences and annotate a genome.

This demo is inspired by the [Apollo User's Guide](https://genomearchitect.readthedocs.io/en/latest/UsersGuide.html), which provides additional guidance.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data upload

To annotate a genome using Apollo, we need the reference genome sequence in FASTA format, and any evidence tracks we want to refine into our annotations. "Evidence tracks" can be any data like:

- A set of prior gene predictions or other genomic feature predictions
- The output of a bioinformatics analysis like BLAST or InterProScan
- Sequencing reads from RNA-Seq or another HTS analysis
- If you are not doing a *de novo* annotation, then a previous released {OGS}

In this tutorial we have obtained some data from NCBI related to [*Escherichia coli K12 str. MG1655*](https://ecoliwiki.org/colipedia/index.php/Category:Strain:MG1655), and we will visualise this data and use it to make some annotations in order to familiarise you with the process.

> <comment-title>Real Data: Unreal Circumstances</comment-title>
> While the data for this tutorial is sourced from publicly available databases, and is all related to different experiments on *E. coli K12*, this is not necessarily the data *you* might use to annotate your genomes. You probably know best what data you should be using in your own circumstances, for the specific features on which you are focused.
{: .comment}

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 0. Create a new history and give it a good name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 1. Click the upload icon {% icon galaxy-upload %}
>
> 2. Switch to the "Rule-based" tab
>
> 3. Copy & Paste the following table into the Rule-based uploader textbox:
>
>    ```
>    https://zenodo.org/records/4889110/files/augustus.gff3	Augustus	gff3
>    https://zenodo.org/records/4889110/files/blastp_vs_swissprot_2018-01-22.blastxml	Blastp vs swissprot	blastxml
>    https://zenodo.org/records/4889110/files/BWA-MEM_K12_Coverage.bigwig	BWA-MEM K12 Coverage	bigwig
>    https://zenodo.org/records/4889110/files/BWA-MEM_K12_Mapping.bam	BWA-MEM K12 Mapping	bam
>    https://zenodo.org/records/4889110/files/BWA-MEM_O104_Coverage.bigwig	BWA-MEM O104 Coverage	bigwig
>    https://zenodo.org/records/4889110/files/BWA-MEM_O104_Mapping.bam	BWA-MEM O104 Mapping	bam
>    https://zenodo.org/records/4889110/files/E._coli_str_K-12_substr_MG1655_100kb_subset.fasta	Genome	fasta
>    https://zenodo.org/records/4889110/files/K12_Variants.vcf	K12 Variants	vcf
>    https://zenodo.org/records/4889110/files/NCBI_AnnotWriter_Genes.gff3	NCBI AnnotWriter Genes	gff3
>    https://zenodo.org/records/4889110/files/O104_H4_LASTZ_Alignment.bed	O104 H4 LASTZ Alignment	bed
>    https://zenodo.org/records/4889110/files/O104_Variants.vcf	O104 Variants	vcf
>    https://zenodo.org/records/4889110/files/TopHat_SRR1927169_rep1.bam	TopHat SRR1927169 rep1	bam
>    https://zenodo.org/records/4889110/files/TopHat_SRR1927169_rep1_Coverage.bigwig	TopHat SRR1927169 rep1 Coverage	bigwig
>    https://zenodo.org/records/4889110/files/TopHat_SRR1927170_rep2.bam	TopHat SRR1927170 rep2	bam
>    https://zenodo.org/records/4889110/files/TopHat_SRR1927170_rep2_Coverage.bigwig	TopHat SRR1927170 rep2 Coverage	bigwig
>    ```
> 4. Click **Build**
>
> 5. From **Rules** menu select `Add / Modify Column Definitions`
>
>    - Click `Add Definition` button and select `URL`
>      - *"URL"*: `A`
>    - Repeat this again and select `Name` instead.
>      - *"Name"*: `B`
>    - Repeat this again and select `Type` instead.
>      - *"Type"*: `C`
>    - Click `Apply`
>
> 7. Click **Upload**
>
{: .hands_on}

# Using Apollo for Annotation

Refining genome annotations happens in multiple steps:

- Create a JBrowse instance from the reference genome FASTA file and evidence tracks
- Import this data into Apollo
- Refine the annotations
- Export the refined genome annotations

In this tutorial we will focus more on the practical portions than the theoretical part of genome annotation, that are covered in other tutorials. When you've completed this tutorial you should be comfortable manipulating genomic data in Galaxy and Apollo.

> <details-title>Why bother?</details-title>
>
> Automated annotation programs continue to improve, however a simple score may not provide evidence necessary to confirm an accurate prediction.
> Therefore, it is necessary to both visually inspect the results and manually fix any issues with the predictions.
>
> Additionally, many times assemblies are less than perfect or read depth and quality may be insufficient, leading to imperfect automatic annotation.
{: .details}

## Build the JBrowse Instance

Let's begin by building a JBrowse instance with all the data we have for this genome.

> <comment-title>Reduced data</comment-title>
>
> To reduce the size of the data, in this tutorial we will only work on a portion of the *Escherichia coli K12 str. MG1655* genome.
{: .comment}

> <hands-on-title></hands-on-title>
>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: Select the `Genome` fasta file
>    - *"Genetic Code"*: `11. The Bacterial, Archael and Plant Plastid Code`
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Gene Calls`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-files %} *"GFF/GFF3/BED Track Data"*: `Augustus` and `NCBI AnnotWriter Genes`
>                        - In *"JBrowse Styling Options [Advanced]"*
>                            - *"JBrowse style.className"*: `transcript-CDS`
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Sequencing`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BAM Pileups`
>                        - {% icon param-files %} *"BAM Track Data"*: Both BWA-MEM Mappings
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BigWig XY`
>                        - {% icon param-files %} *"BAM Track Data"*: Both of the BWA-MEM Coverage files (**not** the `(as bigwig)` files)
>                        - *"Use XYPlot"*: `Yes`
>                        - *"Show Variance Band"*: `Yes`
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `RNA-Seq`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BAM Pileups`
>                        - {% icon param-files %} *"BAM Track Data"*: Both TopHat Mappings
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BigWig XY`
>                        - {% icon param-files %} *"BAM Track Data"*: Both of the `TopHat ... Coverage` files (**not** the `(as bigwig)` files)
>                        - *"Use XYPlot"*: `Yes`
>                        - *"Show Variance Band"*: `Yes`
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Variation`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `VCF SNPs`
>                        - {% icon param-files %} *"SNP Track Data"*: Both Variants files
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Similarity`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `O104:H4 LASTZ Alignments`
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `Blast XML`
>                        - {% icon param-file %} *"Blast XML Track Data"*: The `blastp` results from swissprot (`Blastp vs swissprot`)
>                        - {% icon param-file %} *"Features used in Blast Search"*: The `NCBI AnnotWriter Genes` file
>                        - *"Minimum Gap Size"*: `3`
>                        - *"Is this a protein blast search?"*: `Yes`
>
>    > <comment-title>JBrowse is highly configurable</comment-title>
>    >
>    > JBrowse is highly configurable, we have set a very basic configuration but there are many more advanced features available to you, if you need them. You can choose precisely how data is displayed, and even what menu options are available when users click on features. If your features have some external identifiers like an NCBI Gene ID, you can even configure JBrowse that when the user clicks on the feature, it should show the gene page for that feature in a new tab. These sort of features are incredibly helpful for building very rich experiences.
>    >
>    > A static genome browser like this (just JBrowse, not in Apollo) is very useful for summarising results of a genomics workflow, where the next step is simply interpretation and not annotation. Have a look at the [JBrowse tutorial]({% link topics/visualisation/tutorials/jbrowse/tutorial.md %}) for more information.
>    >
>    > Currently we have built a standalone genome browser (data + the html page and user interface and javascript), but it's possible to just compile the data directory if you intend to send this data to Apollo, and don't need to view the static data in Galaxy.
>    {: .comment}
{: .hands_on}

This tool will take some time to run dependent on data size. All of the inputs need to be pre-processed by JBrowse into a form that it can render and visualise easily. Once this is complete, you can click on the {% icon galaxy-eye %} eyeball to view the JBrowse instance. This is a static view into the data, JBrowse does not let you make any annotations or save any changes. We will convert it into a dynamic view where we can make persistent annotations and share these with our colleagues.

## Sending data to Apollo

Now that we have a good looking static JBrowse instance, it is time to load it into Apollo to turn it into a dynamic view where you can make modifications to the genes.

> <hands-on-title>Import to Apollo</hands-on-title>
>
> 1. {% tool [Create or Update Organism](toolshed.g2.bx.psu.edu/repos/gga/apollo_create_or_update/create_or_update/4.2.5) %} with the following parameters:
>    - {% icon param-file %} *"JBrowse HTML Output"*: output of **JBrowse** {% icon tool %}
>    - *"Organism Common Name Source"*: `Direct Entry`
>        - *"Organism Common Name"*: `E. coli K12`
>    - *"Genus"*: `Escherichia`
>    - *"Species"*: `coli`
>
> 2. {% tool [Annotate](toolshed.g2.bx.psu.edu/repos/gga/apollo_iframe/iframe/4.2.5) %} with the following parameters:
>    - {% icon param-file %} *"Apollo Organism Listing"*: output of **Create or Update Organism** {% icon tool %}
>
> 3. View {% icon galaxy-eye %} the output of the Annotate tool, when it is ready.
>
{: .hands_on}

Viewing the output will open a view into Apollo in the main panel. Here you can interact with your genome and make annotations. This "Annotate" output is a quick link to that specific genome, and while Apollo allows you to manage and annotate multiple genomes, this dataset will always take you back to that specific genome. You can additionally access the Apollo server outside of Galaxy. While the URL will be different for each Galaxy server that supports Apollo, UseGalaxy.eu's Apollo server is available at [https://usegalaxy.eu/apollo](https://usegalaxy.eu/apollo).


# Apollo


![Screenshot of Apollo in Galaxy, apollo takes up most of the space in the center iframe with the history shown on the right.](../../images/apollo/ui.png "This is the Apollo interface, when viewed from inside Galaxy. The Annotate tool produces a special output which acts like a link to the Apollo server. It is <b>not</b> sufficient to download this file to save a copy of your annotations, unlike other datasets in Galaxy. In the main panel Apollo is displayed, with two panels of its own: the Annotation Window (left), and the Annotator Panel (right). The annotation window is the main view into our genome and here we will see evidence, reconcile it, and make our annotations. The right panel allows us to show or hide individual evidence tracks, switch between organisms, and navigate around the genome.")

From the Apollo user manual:

> The major steps of manual annotation using Apollo can be summarized as follows:
>
> 1. Locate a chromosomal region of interest.
> 2. Determine whether a feature in an existing evidence track provides a reasonable gene model to start annotating.
> 3. Drag the selected feature to the ‘User Annotation’ area, creating an initial gene model.
> 4. Use editing functions to edit the gene model if necessary.
> 5. Check your edited gene model for consistency with existing homologs by exporting the FASTA formatted sequence and searching a protein sequence database, such as UniProt or the NCBI Non Redundant (NR) database, and by conducting preliminary functional assignments using the Gene Ontology (GO) database.
>
{: .quote}

The first four steps are generally the process of structural annotation (the process of identifying the correct gene model), and the last includes functional annotation (the process of assigning a putative function to a gene in your annotations).

## Evidence tracks

Let's start by looking at the tracks available to us, and then turning on the gene call tracks so we can start exploring our data.

> <hands-on-title>Visualize the Gene Calls</hands-on-title>
>
> 1. In the right hand panel at the top click on **Tracks** to open the track listing
>
>    ![Zoomed in screenshot of the track menu with the Gene Calls set of tracks expanded showing Augustus and NCBI AnnotWriter genes.](../../images/apollo/tracks.png)
>
> 2. In the **Gene Calls** group, select the `Augustus` track.
>
>    You can either activate tracks in bulk, by clicking on the checkbox to the right of the group name ("Gene Calls"), or by clicking on the group name to expand the section, and then selecting individual tracks.
>
> 3. Zoom to the first 10kb of the genome.
>
>    1. In the left hand Annotation Window, at the top navigation bar you will find a textbox which shows the current location on the genome.
>    2. Edit this and enter `1..10000`
>    3. Press *Go* or use <kbd>Enter</kbd> on your keyboard.
>
>    ![Screenshot of Jbrowse header with the control panel with magnifying glasses and zoom location.](../../images/apollo/location.png)
>
{: .hands_on}

We can now see an evidence track: `Augustus` is the output of [AUGUSTUS](https://github.com/Gaius-Augustus/Augustus) {% cite Stanke_2008 %}. In a *de novo* annotation project, we probably will only have the outputs of various gene callers, and potentially some expression evidence like RNA-Seq.

We will use the other track in the **Gene Calls** group later in this tutorial, leave it unchecked for now.

## Adding new genes

With the selected track, if you look along the genome, you will see many genes that were predicted by Augustus. Each of them as an unique name assigned by Augustus (e.g. `NC_000913.3.g7.t1`, which means `the 7th gene on the NC_000913.3 chromosome`). If you right click on gene, an select `View details`, you can get access to the coding sequence (CDS).

> <details-title>Gene colors</details-title>
> Each gene color corresponds to an open reading frame on the genome. This allows to quickly see if two genes that are very close are on the same open reading frame, which could mean that they can be merged into a single gene, if other evidences support this.
{: .details}

What we want now is first to check that the structure of the genes predicted by Augustus are in good shape. To do this, we will display an additional evidence tracks.

> <hands-on-title>Display blastp vs swissport track</hands-on-title>
>
> 1. In the right hand panel at the top click on **Tracks** to open the track listing
>
> 2. In the **Similarity** group, select the `Blastp vs swissprot` track.
>
{: .hands_on}

This new track represents the result of aligning sequences from the Swissprot databank along the genome, using Blastp. It should look like that:

![JBrowse with blastp results shown, genes appear as red green and blue stripes with a pointed end, blast results appear as stripes with portions missing where they don't align](../../images/apollo/blastp.png)

Each red box is the alignment of a protein from Swissprot on the genome. You can get more information on the aligned sequence by passing your mouse over the alignement, or clicking on it.

Most of the genes predicted by Augustus look very similar to aligned Swissprot sequences, with the coordinates. Note that our example is an ideal situation that you will probably not see on other genomes: as E. coli is a very studied and well annotated genome, which means Swissprot contains many high quality sequences that can be aligned perfectly by Blastp.

If you look at the region `55000..63000` you will notice however that 2 sequences were aligned on the genome, but Augustus didn't predict any corresponding genes.

![JBrowse showing no genes on the Augustus track, and one blast hit.](../../images/apollo/missing_genes.png)

The blastp alignment looks solid, and the sequences seem to correspond to real proteins references in the litterature. We want to add them to our final annotation, to do it, right click on each gene, and select **Create new annotation** > **gene**. They should appear shortly in the **User-created Annotation** track (yellow background, at the top).

![The two genes are added to the User-created Annotation track](../../images/apollo/added_genes.png)

You can also see that they appear now in the list of genes in the right panel, at the gene and the mRNA level (as a gene can have multiple isoforms, in particular for eukaryotes).

![Gene list showing genes b2g.56.0.0 and b2g.55.0.0 which is expanded to reveal a child mRNA](../../images/apollo/gene_list.png)

The **User-created Annotation** track is where you can make modifications to genes, like changing their coordinates, or their name and functional annotation. If you right click on a gene in this track, you will see all the possibilities offered by apollo.

![Gene contextual menu showing way too many options, the mouse hovers over Open Annotation (alt-click)](../../images/apollo/gene_menu.png)

Currently, the two genes we added have meaningless names. Let's improve that: right click on the leftmost gene, and click on **Open Annotation (alt-click)**.

![Gene details menu in apollo showing type set to gene, the name field has been filled with b2g.55.0.0 and a number of other empty fields like symbol, aliases, description.](../../images/apollo/gene_details.png)

From the blast results, we know that this gene is similar to a Swissprot protein, named `Putative uncharacterized protein YabP`. Let's write this in the **Name** field (type it manually and select it from the drop down list), and `YabP` in the **Symbol** field.

> <details-title>Naming genes</details-title>
>
> Giving a proper name to a gene is not always easy. Should it include "Putative" or not? What if multiple names can apply? Should it be lowercase or uppercase?
> The important thing is to always use the same naming rules when working on a full annotation, and to agree on these rules with other collaborators.
> Usually, big annotation consortiums have naming guidelines that you are supposed to follow.
{: .details}

We have just edited the gene name, but Apollo allows to edit information at the mRNA level. Click on the **Sync name with transcript** button to copy the gene name to the mRNA name. It should now display in the **User-created Annotation** track. To check what you can edit at the mRNA level, just click on the corresponding mRNA in the list above:

![very zoomed screenshot showing clicking on the transcript below the gene to open the mRNA info.](../../images/apollo/mrna_level.png)

You should see `Putative uncharacterized protein YabP` in the **Name** field.

You can do the same for the other gene you created, which is similar to `Uncharacterized protein YabQ` according to Blastp vs Swissprot.

> <comment-title>Saving your work</comment-title>
>
> You do not need to do anything specific to *Save* your work in Apollo. Just like Google Docs, each modification is immediately saved, and any other user working on the same genome will instantly see the changes you make.
{: .comment}

## Editing a gene structure

Apollo allows to edit the whole structure of a gene. If you zoom to the 5' end of `YabP`, you will notice, that a few nucleotides after the start codon, there is another one. Let's change the start of `YabP` to this alternate start codon (even if the blast result suggests that we shouldn't do it, it's an exercise!). To do it, all you need to do is click on the 5' limit of the gene and drag it to the desired position. You will notice that the structure of the gene will be shortly changed.

![Gif showing modifying the 5' limit of a gene by dragging it downstream.](../../images/apollo/change_start.gif)

This kind of modifications is very common when using Apollo, and you can perform it at the gene level, or (for eukaryotes) at the exon/intron level. To guide you doing these changes, you should look at all the tracks available for the genome you study. RNA-Seq track are very helpful to determine the limits of coding sequences on the genome, you can find 2 RNA-Seq libraries in the track list, they were aligned on the genome using TopHat. Other tracks can be helpful, like alignements of transcripts or proteins from closely related species (or even big databanks like Swissprot or NR).

## Viewing and reverting changes

Everything you do in Apollo is tracked in a database. If you right click on the `YabP` gene, and select **Show History**, you have access to the full list of all the actions that were performed on it.

![History of changes on a gene showing many changes to Set Name and a final Set Exon Boundaries change](../../images/apollo/history.png)

When you click on one of the steps, you can see below the list a preview of how the gene looked at the time. And you can revert to a specific version of the gene by clicking on the arrow button on the right.

## Adding more functional annotation

Sometimes you'll want to modify a gene that was predicted by Augustus, just to add functional annotation to it. Navigate to position `3000..5700`, you will see an Augustus gene named **NC_000913.3.g3.t1**. At the same position, there are a few Blastp hits with high scores that correspond to `Threonine synthase`, meaning that the gene found by Augustus is probably an homolog. Let's add this Augustus gene to the **User-created Annotation** track: just drag and drop it there. Now, modify the Name (`Threonine synthase`) and Symbol (`TS`), just as we did earlier, for the gene and mRNA.

If you look at the details of the blast hits, you will notice an identifier looking like that: `gi|11387170|sp|P57289.1|`. In this identifier, `P57289` is the id of an [UniProt record](https://www.uniprot.org/uniprot/P57289). Navigate to this UniProt page and you'll find a lot more details about the protein which was found to be very similar to the gene we are currently annotating. In particular, you can see that 3 [Gene Ontology (GO)](http://www.geneontology.org/) terms are associated with it, in the **GO - Molecular function** and **GO - Biological process** sections. Let's add these terms to our gene with Apollo.


> <comment-title>Gene Ontology (GO) Consortium</comment-title>
>
> The [Gene Ontology Consortium](http://www.geneontology.org/) provides with its Ontology a framework for the model of biology.
> The GO defines concepts/classes used to describe gene function, and relationships between these concepts. It classifies functions along three aspects:
>
>
> molecular function
> :  molecular activities of gene products
>
> cellular component
> :  where gene products are active
>
> biological process
> :  pathways and larger processes made up of the activities of multiple gene products.
>
> [more information](http://geneontology.org/page/ontology-documentation) can be found in the GO website.
>
{: .comment}

With the gene selected, click on the **GO** panel, then click on the **New** button. We will then follow the [GO annotation guidelines](http://geneontology.org/docs/go-annotations/), filling the form like this.

![Adding a GO term with a complex and confusing interface. Aspect is set to MF, all of the parameters are described in the following paragraph.](../../images/apollo/go_form.png)

This form means that our gene will be tagged with the **Go Term** `GO:0030170` (select it in the drop down list while typing `pyridoxal phosphate binding`) from the Molecular Function GO branch (`MF` in **Aspect**). This gene `enables` this `pyridoxal phosphate binding` activity, and we declare it based on the evidence code `ECO:0000250` which means `sequence similarity evidence used in manual assertion` (exactly what we are doing). This similarity is **With** the `UniProt:P57289` record, and we add a **Reference** to this record (we could add a pubmed id to reference a published result for example). Finally we add a **Note** saying that there's a strong similarity.

Before saving, don't forget to click on the two **Add** buttons to save the **With** and **Note** fields.

You can now do the same for the two other GO terms found on the UniProt page: `threonine synthase activity` and `threonine biosynthetic process`. Adding these terms this way allows to save the information in a computing-friendly way, which means other bioinformatics tools will be able to use make this information automatically.

Other tabs are available in the annotation panel, allowing to add **Comments**, external database references (**DbXref**) or **Attributes* to genes, to record more infortmation about them.

## Comparing with the official annotation

In this tutorial we are lucky as we are studying a very well known reference organism. It means we have a [very good quality reference annotation](https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi?mode=Info&id=511145&lvl=3&lin=f&keep=1&srchmode=1&unlock) provided by NCBI. You can display it in Apollo by enabling the corresponding track:

> <hands-on-title>Display NCBI annotation track</hands-on-title>
>
> 1. In the right hand panel at the top click on **Tracks** to open the track listing
>
> 2. In the **Gene Calls** group, select the `NCBI AnnotWriter Genes` track.
>
{: .hands_on}

Now navigate along the genome, and notice the differences between the Augustus annotation, the changes you made in the **User-created Annotation** track, and the reference annotation. This illustrates what happens in real life: you start with an automatic annotation which is not perfect, and using Apollo you improve it and add functional annotation, which allows in the end to release a better annotation to the community.

## Sequence alterations

Until now we have supposed that the quality of the genome sequence is perfect. But you might work on othr genomes where the sequence contains errors (substitutions, insertions, deletions) due to assembly problems for example. In this case, automatic annotation programs will have difficulties producing good looking gene models: there can be frame shifts within genes, or broken start/stop codons which are not detected. In this case, Apollo allows to edit the genome sequence itself.

Navigate to position `42500..44500`, and show the `K12 Variants` track from the `Variation` track group. This track shows variants that were detected in this genome sequence after resequencing it. Drag the `NC_000913.3.g36.t1` gene to the **User-created Annotation** track. Now we will register in Apollo the SNP that was detected on position `43988` (C->T): zoom to this position until you see the 6 open reading frame and the sequence. Then right click on the `C` nucleotide and select **Create Genomic Substitution**.

![Right click menu showing options to toggle strands and create genome insertions, deletions, and substitutions. The mouse hovers over substitution.](../../images/apollo/alteration_menu.png)

Fill the form like this, and the SNP will be saved:

![SNP menu with +strand and -strand (seems to be auto-filled out), and a mandatory comment field set to SNP in K12](../../images/apollo/substitution.png)

You can add other types of alterations like insertions or deletions.

Once you have added some alterations, Apollo will automatically display the effect it has on the overlapping genes.

# Exporting and collaborating

{% include topics/genome-annotation/tutorials/apollo/collab.md eukaryote=false %}

# Conclusion


Congratulations, you finished this tutorial! By using Apollo and JBrowse, you learned how to manually refine predicted annotations and export them to Galaxy for future analyses. You also learn how to give access to your project at any other researcher, making it a real collaborative solution.

A [similar tutorial for eukaryote genomes]({% link topics/genome-annotation/tutorials/apollo-euk/tutorial.md %}) exists, using different types of evidence tracks, feel free to have a look at it to learn more.

When refinement is sufficient an updated or new version of the annotation may be exported as GFF3 as well as published as a new JBrowse directory for inspection.

# What's next?

After generating your refined annotation, you'll want to merge it back into the official gene sets. A future tutorial will show you how to do it within Galaxy.

If a *de novo* set, you can export it as GFF3 and load it into a tool like [Tripal](http://tripal.info) to provide visualization.


topic: genome-annotation
tutorial: tnseq
title: Essential genes detection with Transposon insertion sequencing
difficulty: intermediate
time_estimation: 7H
tools_json: []
requirements_json: []
tags_json: ["bacteria", "tnseq", "essential genes", "microgalaxy"]
content_hash: 846b9717f469e14799f92a8e535a73ca
last_modified: 2026-02-26T21:12:16.602194
zenodo_link: https://doi.org/10.5281/zenodo.2579335
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/tnseq/tutorial.html

content:
In microbiology, identifying links between genotype and phenotype is key to understand bacteria growth and virulence mechanisms, and to identify targets for drugs and vaccines. These analysis are limited by the lack of bacterial genome annotations (*e.g.* 30% of genes for *S. pneumoniae* are of unknown function) and by the fact that genotypes often arose from complex composant interactions.

## Transposon insertion Sequencing

Transposon insertion sequencing is a technique used to functionally annotate bacterial genomes. In this technique, the genome is saturated by insertions of transposons. Transposons are highly regulated, discrete DNA segments that can relocate within the genome. They have a large influence on gene expression and can be used to determine the function of genes.

When a transposon inserts itself in a gene, the gene's function will be disrupted, affecting the fitness (growth) of the bacteria. We can then manipulate transposons for use in insertional mutagenesis, *i.e.* creation of mutations of DNA by the addition of transposons. The genomes can be then sequenced to locate the transposon insertion site and the function affected by a transposon insertion can be linked to the disrupted gene.

![Illustration of tnseq Method](../../images/tnseq/principle_tnseq.png "Transposon insertion sequencing method (from <a href='#Chao2016'> Chao <i>et al.</i> 2016</a>)")

1. **Data production** (**a** in the previous image)

    An initial population of genomes is mutated so that the genome of the bacteria is saturated with transposon insertions. A library is called *saturated* if in the genomes across the whole population of bacteria, each potential insertion site has at least one insertion. The population is then divided into several media containing different growth conditions, to identify the impact of the insertions on the bacteria growth. After growth, the regions flanking the insertion are amplified and sequenced, allowing to determine the location of the insertion.

2. **Analysis** (**b** in the previous image)

    The sequences are aligned to the reference genome to identify the location of the regions flanking the insertions. The resulting data will show a discrete repartition of reads on each site. If a gene present several insertions, like the two leftmost genes in *Condition A*, it means that its disruption has little or no impact to the bacterial growth. On the other hand, when a gene shows no insertions at all, like the rightmost gene in *Condition A*, is means that any disruption in this gene killed the bacteria, meaning it is a gene essential to bacteria survival. If the library is sufficiently saturated, there is a clear threshold between essential and non-essential genes when you analyze the insertion rate per gene.

Two type of transposon insertion methods exist:
- Gene disruption, where we analyze only the disruptions. This will covered by of this tutorial
- Regulatory element insertion, where different promoters are inserted by the transposon, and we analyze the change in gene expression in addition to the disruption. This method will be the subject of another tutorial.

## Building a TnSeq library


Different types of transposons can be used depending of the goal of your analysis.

- Randomly pooled transposon
    - Mariner-based transposons, common and stable transposons which target the "TA" dinucleotides

        The TA are distributed relatively evenly along genome. The Mariner-based transposons can be inserted to impact statistically every gene, with in average more than 30 insertions site per kb. With the low insertion bias, it is easy to build saturated libraries. But local variations means less loci and less statistical power.

    - Tn5-based vectors, which insert at random sites

        This transposon do not require any target sequences. It is useful for specie where it is difficult to build mariner based transposons. But it has a preference for high GC content, causing insertion bias

- Defined sequence transposon

    It can be used to study interactions in pathways of interest, but also more precise targeting (small genes, pathways) for specific analyses

Independently of the transposon choice we need to be careful about the library complexity. With a large complex library, multiple insertion can be found in every potential locus. The higher density of insertion, the greater precision in identifying limits of regions of interest. If the density of the library is too low, some genes might by chance not be disrupted and mistaken for essential. The advantage of a target specific transposon, like the mariner, in opposition of a Tn5-based transposon inserting randomly, it that the limited number of insertion sites makes it easier to build high complexity libraries.

After selection of the type of transposon, we need to modify it to allow insertion site amplification and sequencing to get a library fitting the transposon insertion. Biases could be introduced during the process due to uneven fragment sizes. To avoid that, we can introduce a Type I restriction site to cleave DNA downstream of transposon, and get uniform fragment sizes and therefore avoid a bias in the representation of the insertions.

As we just want to identify the TA site affected by an insertion, we only need the location of the start of the reads and not a good coverage of the entire genome. Long reads are then not so important. On the other hand, a minimum transposon length of 16 bp is necessary for precise mapping on the genome {% cite Kwon2015 %}. We can therefore not use the BsmFI restriction site (11 to 12 bp) but MmeI.

**In this tutorial, we are using mariner transposon targeting TA sequences, in ordered to target the whole genome uniformly,** with two specific regions used to specifically sequence the region upstream of the insertion {% cite Santiago2015 %}

![Structure of the transposon containing several parcodes and adapters](../../images/tnseq/tranposon_structure.png "Structure of the transposon containing several parcodes and adapters (from <a href='#Santiago2015'> Santiago <i>et al.</i> 2015</a>)")

The transposon inserts itself at TA site at the ITR junctions. These ITR junctions have been modified to include a Mme1 restriction site and a NotI restriction site which cut 21 bp upstream the restriction site. These two site are the 5' and 3' limits to the genomic DNA we want to sequence.

1. After digestion by NotI restriction enzyme, the fragments are attached to biotinylated adaptors that link to NotI restriction site. The attached fragment are then digested by MMeI at a site upstream, where an Illumina primer is then linked. The sequencing is then done, adding Illumina adaptors and an additional barcode to the read for multiplexed sequencing.

2. An insertion can sometimes be composed of one or more copies of the transposon (multimer). There is therefore a risk to select plasmid backbone sequence. To solve this problem, an additional NotI has been added in the backbone to create different length construct, that can later be filtrated. Different promoters are added with an additional 3 bp barcode to analyze differential expression impact. This type of complex analysis will be covered in a follow-up tutorial.

Because of this complex transposon structure, the reads obtained after sequencing contain a lot of adapters and foreign sequences used to insert and target the transposon. Several steps of preprocessing are then need to extract only the transposon sequence before finding its location on the genome.

## Tnseq analysis


Once the genomic sequences are extracted from the initial reads (*i.e.* remove non genomic sequences from the reads), they need to be located each on the genome to link them to a TA site and genes. To do that we map them to a reference genome, link them to a specific insertion site, and then count the number of insertion for each TA site and identify essential genes of regions.

We will apply this approach in this tutorial using a subset of TnSeq reads from {% cite Santiago2015 %}.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data upload

Let's start with uploading the data.

> <hands-on-title>Import the data</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import from [Zenodo](https://zenodo.org/record/2579335) or a data library (ask your instructor):
>   - FASTQ file with the Tnseq reads: `Tnseq-Tutorial-reads.fastqsanger.gz`
>   - Set of barcodes to separate reads from different experimental conditions: `condition_barcodes.fasta`
>   - Set of barcodes to separate reads from different transposon constructs: `construct_barcodes.fasta`
>   - Genome file for *Staphylococcus aureus*: `staph_aur.fasta`
>   - Annotation file for *Staphylococcus aureus*: `staph_aur.gff3`
>
>    ```
>    https://zenodo.org/record/2579335/files/Tnseq-Tutorial-reads.fastqsanger.gz
>    https://zenodo.org/record/2579335/files/condition_barcodes.fasta
>    https://zenodo.org/record/2579335/files/construct_barcodes.fasta
>    https://zenodo.org/record/2579335/files/staph_aur.fasta
>    https://zenodo.org/record/2579335/files/staph_aur.gff3
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
> 3. Rename the files
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}


# Remove all non genomic sequences from the sequenced reads

Because of the experimental design for transposon insertion sequencing, the raw reads contain a lot of adapters and foreign sequences used to insert and target the transposon. To obtain the core reads that contain only genomic sequence, the reads have to go through several steps (using the **Cutadapt** tool) to remove them and divide the reads per experimental condition and type of transposon:

![Workflow of data pre-processing](../../images/tnseq/preprocessing.png "Data pre-processing")

1. We separate the reads of each experimental condition based on a 8 bp barcode at the beginning of each read. These barcodes were added to be able to pool different conditions together before the transposon insertion and sequencing.
2. The tail of each set of read is then removed. It immediately follows the 3 bp barcode specific to transposon constructs, and contains illumina adapter sequence and downstream
3. To be sure all our reads have been trimmed correctly we filter out the reads too large.
4. We then separate the reads per transposon construct
5. We remove the remaining transposon sequence containing MmeI.

## Separate reads by experimental conditions

First we divide the initial data set by experimental conditions using the 8 bp barcode added during the Illumina multiplexing protocol.

> <hands-on-title>Inspect condition barcodes</hands-on-title>
>
> 1. Inspect the `condition_barcodes` file
>
{: .hands_on}

This fasta file contains the barcodes for each condition:

```
>control
^CTCAGAAG
>condition
^GACGTCAT
```

We can see 2 barcodes there: one for control and one for condition.

> <comment-title>Barcode symbols used by <b>Cutadapt</b></comment-title>
>
> The `^` at the beginning of the sequence means we want to anchor the barcode at the beginning of the read. To know more about the symbols used by **Cutadapt**, you check the [**Cutadapt** manual](https://cutadapt.readthedocs.io/en/stable/guide.html#adapter-types).
{: .comment}

We would like now to split our Tnseq reads in `Tnseq-Tutorial-reads` given the barcodes.

> <hands-on-title>Split reads by condition barcodes using Cutadapt</hands-on-title>
>
> 1. **Cutadapt** {% icon tool %} with:
>     - *"Single-end or Paired-end reads?"*: `Single-end`
>       - {% icon param-file %} *"FASTQ/A file"*: `Tnseq-Tutorial-reads`
>       - In *"Read 1 Options"*
>         - In *"5' (Front) Adapters"*
>           - Click on *"Insert 5' (Front) Adapters"*
>             - *"Source"*: `File From History`
>               - *"Choose file containing 5' adapters"*: `condition_barcodes` file
>     - In *"Adapter Options"*
>       - *"Maximum error rate"*: `0.15` (to allow 1 mismatch)
>       - *"Match times"*: `3` (to cover cases where barcodes are attached several times)
>     - In *"Output Selector"*, select
>       -  *"Report"*
>       -  *"Multiple output"* (to separate the reads into one file per condition)
>       -  *"Untrimmed reads"* (to write reads that do not contain the adapter to a separate file)
>
>     The output is a collection of the different conditions datasets, here control and condition, and a report text file.
>
> 2. Inspect the report text generated by **Cutadapt** {% icon tool %}
{: .hands_on}

> <question-title></question-title>
>
> 1. What percentage of reads has been trimmed for the adapter?
> 2. How many reads have been trimmed for each condition?
> 3. What should we change if our barcodes were at the end of the reads ?
>
> > <solution-title></solution-title>
> >
> > 1. 100% of the reads were trimmed (line `Reads with adapters`)
> > 2. 2,847,018 for control (`CTCAGAAG` barcode) and 2,862,108 for condition (`GACGTCAT` barcode)
> > 3. We should use the option *"Insert 3' (End) Adapters"* and anchored them at the end of the read with the symbol `$` in our fasta file containing the barcodes.
> {: .solution }
>
{: .question}


## Remove Adapter sequence

Our reads are now divided by condition. We need to trim their tail containing the Illumina adapters, used for initiating the sequencing (`CGTTATGGCACGC`, here). To do so, we remove the adapter and everything downstream, using the end adapter option of **Cutadapt** and not anchor the sequence anywhere. To eliminate reads that might not have been trimmed because of too many mismatches or other reasons, we filter the reads by size, given the known approximate size of the remaining sequences (**how is it computed??**).

> <hands-on-title>Remove Adapter with Cutadapt</hands-on-title>
>
> 1. **Cutadapt** {% icon tool %} to remove adapters with:
>    - *"Single-end or Paired-end reads?"*: `Single-end`
>      - {% icon param-collection %} *"FASTQ/A file"*: collection output of the previous **Cutadapt**
>      - In *"Read 1 Options"*
>        - In *"3' (End) Adapters"*
>          - Click on *"Insert 3' (End) Adapters"*
>            - *"Source"*: `Enter custom Sequence`
>              - *"Enter custom 3' adapter sequence"*: `CGTTATGGCACGC`
>    - In *"Adapter Options"*
>      - *"Match times"*: `3` (to cover cases where barcodes are attached several times)
>    - In *"Output Selector"*, select
>      -  *"Report"*
>
>    > <question-title></question-title>
>    > What are the outputs at this step?
>    > > <solution-title></solution-title>
>    > > The outputs are two collections: one containing the reads in both conditions, and one containing the **Cutadapt** reports for each condition.
>    > {: .solution }
>    {: .question}
>
> 2. Inspect the generated report files
>
>    > <question-title></question-title>
>    > What percentage of the reads contained the adapter?
>    > > <solution-title></solution-title>
>    > > More than 99% of the reads contained the adapter in both conditions (line `Reads with adapters` in the reports)
>    > {: .solution }
>    {: .question}
>
> 3. **Cutadapt** {% icon tool %} to filter reads based on length with:
>    - *"Single-end or Paired-end reads?"*: `Single-end`
>      - {% icon param-collection %} *"FASTQ/A file"*: collection output of the previous **Cutadapt**
>    - In *"Filter Options"*
>      - *"Minimum length"* : `64`
>      - *"Maximum length"* : `70`
>    - In *"Output Selector"*, select
>      -  *"Report"*
>
> 4. Inspect the generated report files
>
>    > <question-title></question-title>
>    > How many reads where discarded after filtering?
>    > > <solution-title></solution-title>
>    > > Less than 2% of the reads were discarded in both conditions (lines `Reads that were too short` and `Reads that were too long` in the reports)
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

We can see that is both samples the reads have pass the filtering at more than 98%. If this percentage is very low, it means the previous trimming steps is incomplete or faulty.

## Separate reads from different transposon constructs

The constructs used in this experiment contain different strengths and directions of promoters. We use the different constructs as replicates, so we need now to separate the reads based on the construct specific 3 bp barcodes.

> <comment-title></comment-title>
> In addition of disrupting a gene at the location of the insertion, such constructs can modify the expression of either upstream or downstream regions. The analysis of such modification will be studied in another training material, but for now we consider that the construct does not impact the essentiality analysis.
{: .comment}

> <hands-on-title>Inspect construct barcodes</hands-on-title>
>
> 1. Inspect the `construct_barcodes` file
>
>    > <question-title></question-title>
>    >
>    > What does the `$` mean in the barcode sequence file?
>    >
>    > > <solution-title></solution-title>
>    > > It means the barcode is anchored at the end of the reads.
>    > {: .solution}
>    {: .question}
{: .hands_on}

> <hands-on-title>Barcode split with Cutadapt</hands-on-title>
>
> 1. **Cutadapt** {% icon tool %} with:
>    - *"Single-end or Paired-end reads?"*: `Single-end`
>      - {% icon param-collection %} *"FASTQ/A file"*: collection output of the previous **Cutadapt**
>      - In *"Read 1 Options"*
>        - In *"3' (End) Adapters"*
>          - Click on *"Insert 3' (End) Adapters"*
>            - *"Source"* : `File From History`
>               - {% icon param-file %} *"Choose file containing 5' adapters"*: `construct_barcodes` file
>    - In *"Adapter Options"*
>       - *"Match times"*: `3` (to cover cases where barcodes are attached several times)
>    - In *"Output Selector"*, select
>       -  *"Report"*
>       - *"Multiple output"* (to separate the reads into one file per condition)
>
> 2. Inspect the report files
>
>    > <question-title></question-title>
>    >
>    > Are the reads equally divided between constructs?
>    >
>    > > <solution-title></solution-title>
>    > > When you look at the reports, you can see that most of the reads have been assigned to the *blunt* construct (`TAC` barcode): the blunt construct is the control and does not contain any promoters. This means that there is less negative selective pressure on blunt than the other ones, that have affected flanking region in addition to the disrupted gene at the insertion site. This won't be a problem here as the tool we use for the essentiality prediction consider the sum of reads in the replicates.
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

You can notice that the output of this split is a *nested collection*, a collection of collection.

## Remove the remaining transposon sequence

The last remaining transposon sequence in the reads is the linker with the MmeI restriction site (`ACAGGTTGGATGATAAGTCCCCGGTCTATATTGAGAGTAACTACATTT`).

> <hands-on-title> Remove Linker with Cutadapt</hands-on-title>
>
> 1. **Cutadapt** {% icon tool %} with:
>    - *"Single-end or Paired-end reads?"*: `Single-end`
>      - {% icon param-collection %} *"FASTQ/A file"*: collection output of the previous **Cutadapt**
>      - In *"Read 1 Options"*
>        - In *"3' (End) Adapters"*
>          - Click on *"Insert 3' (End) Adapters"*
>            - *"Source"*: `Enter custom Sequence`
>              - *"Enter custom 3' adapter sequence"*: `ACAGGTTGGATGATAAGTCCCCGGTCTATATTGAGAGTAACTACATTT`
>    - In *"Adapter Options"*
>      - *"Maximum error rate"*: `0.15`
>    - In *"Output Selector"*, select
>      -  *"Report"*
>
> 2. Inspect the report files and check that the majority of the read have been trimmed
{: .hands_on}

Now that we isolated the genomic sequences from the initial reads, we want to align them to count how many insertion have been retained at each TA site.

# Count the number of insertion per TA sites

## Align the reads to a reference genome

To identify the location of each TA site to the count them, the first step is to map the reads on the reference genome. We use the **Bowtie**.

> <comment-title></comment-title>
> We could also use **Bowtie2** with an end-to-end option, but **Bowtie** is more suitable for very short reads like ours (16-17 bp).
{: .comment}


> <hands-on-title>Map reads with Bowtie</hands-on-title>
>
> 1. **Map with Bowtie for Illumina** {% icon tool %} with:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use one from the history`
>      - {% icon param-file %} *"Select the reference genome"*: `staph_aur.fasta` file.
>    - *"Is this library mate-paired?"*: `Single-end`
>      - {% icon param-collection %} *"FASTQ file"*: collection output of the last **Cutadapt**
>      - *"Bowtie settings to use"*: `Full parameters list`
>      - *"Skip the first n reads (-s)"*: `0`
>      - *"Maximum number of mismatches permitted in the seed (-n)"*: `0`
>      - *"Seed length (-l)"*: `17`
>      - *"Whether or not to make Bowtie guarantee that reported singleton alignments are 'best' in terms of stratum and in terms of the quality values at the mismatched positions (--best)"*: `Use best`
>      - *"Whether or not to try as hard as possible to find valid alignments when they exist (-y)"* : `Try Hard`
>
>    > <question-title></question-title>
>    > Why are we strictly enforcing no mismatch mapping?
>    > > <solution-title></solution-title>
>    > > Our reads being very short, the smallest size needs precise mapping, allowing even one mismatch would risk having reads mapping in wrong positions.
>    > {: .solution }
>    {: .question}
>
> 2. Rename your collection for better clarity
>
>    {% snippet faqs/galaxy/collections_rename.md %}
>
{: .hands_on}

## Compute coverage of the genome

We have now our reads mapped on the reference genome. We can compute the genome coverage, *i.e.* how much the nucleotides of the genome are covered by reads. This information will be later crossed with our TA sites position.

In our case, the sequenced reads cover the  5' region flanking region of the TA site where the transposon inserted. We should not then extract the coverage across the whole reads, as it could cover several TA sites, but only the coverage at the 3' end of the read:

![A read align to the genome with its 3' end covering half of the TA site](../../images/tnseq/Map_cov.png "Mapping read and TA site coverage")

> <hands-on-title>Compute genome coverage</hands-on-title>
>
> 1. **bamCoverage** {% icon tool %} with:
>    - {% icon param-collection %} *"BAM/CRAM file"*: collection output of the last **Map with Bowtie for Illumina**
>    - *"Bin size in bases"*: `1`
>    - *"Scaling/Normalization method"*: `Do not normalize or scale`
>    - *"Coverage file format"*: `bedgraph`
>    - *"Show advanced options"*: `yes`
>      - *"Ignore missing data?"*: `Yes` (to get only region with read counts)
>      - *"Offset inside each alignment to use for the signal location"*: `-1` (to read the signal of the coverage only at the 3' end of the read)
>
{: .hands_on}

## Identify TA sites positions

In order to get the coverage on each TA site we need to prepare a file containing the position of each TA site.

As you can see on the [previous figure](#figure-4), the read can cover one side or the other of the TA depending on the direction of insertion of the transposon. Depending on the direction of insertion the coverage will be counted on the leftmost position of the TA site or on the rightmost. As we are not considering strand separately in this analyses, we will consider both count as attached to the leftmost base of the TA site. To do that we will create two list of TA site positions, listing the 5' end of each TA site for forward and reverse strand, and then merge them to get a global count per TA site.

> <hands-on-title>Get TA sites coordinates</hands-on-title>
>
> 1. **Nucleotide subsequence search** {% icon tool %} with:
>    - {% icon param-file %} *Nucleotide sequence to be searched"*: `staph_aur.fasta`
>    - *"Search with a"*: `user defined pattern`
>    - *"Search pattern"*: `TA`
>    - *"Search pattern on"*: `forward strand`
> 4. Inspect the generated file
{: .hands_on}

**Nucleotide subsequence search** generate one BED file with the coordinates of each TA on the genome:
1. Chrom
2. Start
3. End
4. Name
5. Strand

The only information we need here are the positions of the 5' end of each TA site for each strand.

> <hands-on-title>Get forward and reverse strand positions</hands-on-title>
>
>
> 1. **Cut columns from a table** {% icon tool %} with:
>    - Cut columns : `c2`
>    - *"Delimited by"*: `tab`
>    - {% icon param-file %} *"From"*: output of **Nucleotide subsequence search**
>
> 4. Add the `#forward` tag to the output
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
> 2. **Cut columns from a table** {% icon tool %} with:
>    - Cut columns : `c1,c3`
>    - *"Delimited by"*: `tab`
>    - {% icon param-file %} *"From"*: output of **Nucleotide subsequence search**
>
> 3. **Compute an expression on every row** {% icon tool %} to shift the end position of TA sites  by 1 and get the start of the TA site on the reverse strand with:
>    - *Add expression"*: `c2-1`
>    - {% icon param-file %} *"as a new column to"*: output of **cut**
>    - *"Round result?"*: `yes`
>
>
> 4. **Cut columns from a table** {% icon tool %} with:
>    - Cut columns : `c3`
>    - *"Delimited by"*: `tab`
>    - {% icon param-file %} *"From"*: output of **Compute**
>
> 8. Add the `#reverse` tag to the output
>
{: .hands_on}


## Merge overall coverage and positions of TA sites to get the coverage of each TA sites

We have now 2 files containing the coordinates of our TA sites for both strands. We should cross them with the coverage files to get the coverage on both sides of each TA site.

> <hands-on-title> Get coverage of TA sites</hands-on-title>
>
> 1. **Join two files** {% icon tool %} with:
>    - {% icon param-collection %} *1st file"*: output of **bamCoverage**
>    - *"Column to use from 1st file"*: `Column 2` (to select the position of the end of the read)
>    - {% icon param-file %} *"2nd File"*: output of last **cut** with the `forward` tags
>    - *"Column to use from 2nd file"*: `Column 1`
>    - *"Output lines appearing in"*: `Both 1st & 2nd file, plus unpairable lines from 2st file. (-a 2)` (to get all TA sites)
>    - *"First line is a header line"*: `No`
>    - *"Value to put in unpaired (empty) fields"*: `0` (to assign a count of 0 to TA sites not covered in the bamCoverage file)
>
> 2. Add `#forward` tag to the output collections
>
>    {% snippet faqs/galaxy/collections_add_tag.md %}
>
> 3. **Cut columns from a table** {% icon tool %} with:
>    - Cut columns : `c1,c4`
>    - *"Delimited by"*: `tab`
>    - {% icon param-file %} *"From"*: output of**Join two files** with the `forward` tag
>
>
> 5. Repeat the same steps for `reverse`
>
{: .hands_on}

We now have a read count for each nucleotide at the different TA sites. The insertions on the forward strand files are in a different direction than those on the reverse strand file. In our case, we are only interested in studying the gene disruption, and therefore we do not want to consider the directions separately. We can then combine the forward and reverse counts together to get a total count per TA site. In order to do that we need to assign the count of both strand to the same nucleotide, by shifting by one position the count on the reverse strand.

> <hands-on-title>Get total count per TA site</hands-on-title>
>
> 1. **Compute an expression on every row** {% icon tool %} to shift the positions of reverse strand counts with:
>    - *Add expression"*: `c1-1` (to shift the position by 1)
>    - {% icon param-collection %} *"as a new column to"*: **Cut** output collection with the tags `reverse`
>    - *"Round result?"* : `yes`
>
> 2. **Cut columns from a table** {% icon tool %} with:
>    - *"Cut columns"*: `c3,c2`
>    - *"Delimited by"*: `tab`
>    - {% icon param-collection %} *File to cut"*: output of the previous **Compute**
>
> 3. **Sort** {% icon tool %} with:
>    - {% icon param-collection %} *"Sort Query"*: the newest `reverse` collection
>    - *"Number of header lines"*: `0`
>    - In *"Column selections"*
>      - *"on column"*: `Column: 1` (to sort on positions)
>      - *"in"*: `Ascending Order`
>      - *"Flavor"*: `Fast numeric sort (-n)`
>
> 4. Repeat the **Sort** with the newest `forward` collection
>
> 6. **Join two files** {% icon tool %} to merge the two collections and get the forward and reverse count in two different columns with:
>    - {% icon param-collection %} *1st file"* : the `forward` collection from **Sort**
>    - *"Column to use from 1st file"*: `Column 1` to join on positions
>    - {% icon param-collection %} *"2nd File"* : the `reverse` collection from **Sort**
>    - *"Column to use from 2nd file"*: `Column 1`
>    - *"Output lines appearing in"*: `Both 1st & 2nd file` (to get all TA sites)
>    - *"First line is a header line"*: `No`
>    - *"Value to put in unpaired (empty) fields"*: `.`
>
> 7. **Compute an expression on every row** {% icon tool %} to get the total read count with:
>    - *Add expression"*: `c2+c3` to get the total count
>    - {% icon param-collection %} *"as a new column to"*: output of **Join two files** with the tags `reverse` and `forward`
>    - *"Round result?"*: `yes`
>
>
> 2. **Cut columns from a table** {% icon tool %} with:
>    - *"Cut columns"*: `c1,c4`
>    - *"Delimited by"*: `tab`
>    - {% icon param-collection %} *File to cut"*:output of the last **Compute**
>
> 9. **Sort data in ascending or descending order** {% icon tool %} with:
>    - {% icon param-collection %} *Sort Query"*: output of the last **Cut**
>    - *"Number of header lines"*: `0`
>    - In *"Column selections"*
>      - *"on column"*: `Column: 1` (to sort on positions)
>      - *"in"*: `Ascending Order`
>      - *"Flavor"*: `Fast numeric sort (-n)`
>
{: .hands_on}

> <question-title></question-title>
>
> Take a look at the output files:
>
> 1. What are the data contained in the two columns?
> 2. What do the 5 first lines of the blunt sample in the treatment with the condition look like?
>
> > <solution-title></solution-title>
> > 1. The first column contain the position of the TA site, and the second column the number of insertions observed.
> > 2. Your five first lines should look like this :
> >     - One column with the positions : `4,10,16,42,79`
> >     - One column with the number of insertions : `0,3,20,0,11`
> >
> > If your counts are only zeros in one of the blunt sample, you might have made a mistake in merging the counts to the TA sites.
> >
> {: .solution }
{: .question}


# Predicting Essential Genes with Transit

Now that we have the counts of insertions per TA site, we can use them to predict gene essentiality. Several methods exist to identify essential genes of regions. They can be divided in two major categories:

![Different types of TnSeq Analyses](../../images/tnseq/type_of_analyses.png "Methods of TnSeq Analyses (from <a href='#Chao2016'> Chao <i>et al.</i> 2016</a>)")

1. **Annotation dependent method**

    The total read count an/or percentage of disrupted site are computed per annotated regions. The values are then compared to the rest of the genome to classify the genes into the categories *essential* or *non-essential*.

2. **Annotation independent method**

    The total read count and/or disrupted sites are computed independently of annotated regions. One of these methods is using a sliding window. Each window is then classified into the categories *essential* or *non-essential*. After the windows have been classified, they are linked annotations, and the genes/regions can be classified as *essential*, *non-essential*, or *domain essential* according to the classification of the windows they cover. The same classification can be done using HMM based methods instead of sliding windows. In that case, each insertion site will be predicted as *essential* or *non essential* {% cite Chao2016 %}).

To predict the essential genes in our datasets, we will use the Transit tool {% cite dejesus2015transit %}.

## Transit

Transit is a software that can be used to analyse TnSeq Data. It is compatible with Mariner and Tn5 transposon. In total, 3 methods are available to assess gene essentiality in one sample.

### Gumbel method

The Gumbel method performs a gene by gene analysis of essentiality for Mariner data based on the longest consecutive sequence of TA site without insertions in a gene. This allows to identify essential domains regardless of insertion at other location of the gene.

The distribution governing the longest sequence of consecutive non insertion is  characterized by a Gumbel distribution. This distribution is often used to model the distribution of maximums (or minimums) of a variable (See the [Wikipedia article on Gumbel distribution ](https://en.wikipedia.org/wiki/Gumbel_distribution)). The Gumbel distribution is used as a likelihood of non-essentiality, as the longest run of non-insertions should follow what is expected given the global distribution of non-insertion runs. The essential genes, whose runs of empty sites are longer than expected, are modeled through a normalized sigmoid function. This function reflect that any gene can be essential, except the gene were the run of non insertions are too smell to represent a domain.

The total distribution of the maximum run of non-insertion per gene is therefore represented y a bimodal distribution composed of a Gumbel distribution of non-essential genes and a normalized sigmoid function of essential ones. The parameters of these distributions are estimated through a Metropolis–Hastings (MH) sampling procedure {% cite dejesus2013bayesian %}).

Using these two distribution, the posterior probability of each distribution is calculated for each gene. Some genes can be classified as "Unclear" if one probability is not winning over the other. Some other can be classified as "Small" if the space of TA sites covered by the gene is insufficient to categorize it (See the [Transit Manual for the Gumbel method](https://transit.readthedocs.io/en/latest/transit_methods.html#gumbel)).

### HMM method

The HMM method performs a whole genome essentiality analysis for Mariner data. This approach uses the clustering of TA sites along the genome to identify essential regions, and then apply results to the annotation to identify genes containing essential regions. The HMM method provides a classification for each TA site into 4 states : Essential, Non-Essential, Growth Advantage, and Growth Defect.

This method require a well-saturated library and is sensitive to sparse datasets {% cite dejesus2013hidden %}.

### Tn5Gaps method

The Tn5Gaps method is a method dedicated to the identification of essential genes in studies using Tn5 transposons. The analysis is performed on the whole genome to identify regions of non insertion overlapping with genes. It is based on a Gumbel analysis method {% cite griffin2011high %} and adapted to Tn5 transposon specificity. The main difference comes from the fact that Tn5 transposon can insert everywhere, thus
creating libraries with lower insertion rates. The difference from the Gumbel method described above is that the run of non insertion are computer on the whole genome instead of individual genes. The longest run of non insertion considered is not the longest within the gene, but the longest one overlapping the gene.


## Predict the essentiality of genes

In our case, we will use the Gumbel method, because we are using Mariner data and our data look pretty sparse. Gumbel is therefore the safe choice compared to HMM which is more sensitive to sparse data.


In order to use transit, we need to create a an annotation file in the `prot_table` format, specific to the Transit tool. It can be created this file from a GFF3 from GenBank like the one we uploaded earlier.

> <hands-on-title>Hands-on : Create annotation file in prot_table format</hands-on-title>
> 1. Check that the format of `staph_aur.gff3` file is `gff3` and not `gff`
> 2. Change the datatype if needed
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="gff3" %}
>
> 3. **Convert GFF3 to prot_table for TRANSIT** {% icon tool %} with:
>    - {% icon param-file %} *GenBank GFF file"*: the `staph_aur.gff3` file
>
{: .hands_on}


Now that we have prepared the annotation file, we can use the count per TA site to predict essential genes using Transit tool, with some customized parameters:

- The number of sites with a single read could be artefactual. We ignore then counts lower than 2
- A disrupted site that would be very close to the border of the gene may not actually disturb the gene function, and therefore not be an actual signal of disruption. We ignore then insertion near the extremities of the genes.

> <hands-on-title>Predict gene essentiality with Transit</hands-on-title>
>
> 1. **Gumbel** {% icon tool %} with:
>    - *"Operation mode"*: `Replicates`
>      - {% icon param-collection %} *"Input .wig files"*: output of the latest **Sort**
>    - {% icon param-file %} *"Input annotation"*: output of the latest **Convert**
>    - *"Smallest read-count to consider"*: `2` (to ignore single count insertions)
>    - *"Ignore TAs occurring at given fraction of the N terminus."*: `0.1` (to ignore 10% of the insertion at the N-terminus extremity of the gene)
>    - *"Ignore TAs occurring at given fraction of the C terminus."*: `0.1` (to ignore 10% of the insertion at the C-terminus extremity of the gene)
>
{: .hands_on}

The output of Transit is a tabulated file containing the following columns:
- Gene ID
- Name of the gene
- Gene description
- Number of Transposon Insertions Observed within the Gene
- Total Number of TA sites within the Gene
- Length of the Maximum Run of Non-Insertions observed (in number of TA sites)
- Span of nucleotides for the Maximum Run of Non-Insertions
- Posterior Probability of Essentiality
- Essentiality call for the gene
  - E=Essential
  - U=Uncertain
  - NE=Non-Essential
  - S=too short

> <comment-title>More details about Transit</comment-title>
> You can find more information on [Transit manual page](https://transit.readthedocs.io/en/latest/transit_methods.html)
{: .comment}

We can obtain the list of genes predicted as essential by filtering on the essentiality call.

> <hands-on-title>Hands-on : Get table of essential genes</hands-on-title>
>
> 1. **Filter data on any column using simple expressions** {% icon tool %} with:
>    - {% icon param-collection %} *"Filter"*: output of the last **Gumbel**
>    - *With following condition"*: `c9=='E'` (to select essential genes)
>
{: .hands_on}


> <question-title></question-title>
>
> How many genes are essential in each conditions?
>
> > <solution-title></solution-title>
> >
> > 359 genes are essential in control, and 381 in condition.
> {: .solution }
>
{: .question}


## Compare the essential genes between two conditions

Now let's compare the results between out two conditions.

> <hands-on-title>Separate Files from the collection</hands-on-title>
>
> 1. **Extract Dataset from a list** {% icon tool %} to extract 1st file in the collection with:
>    - {% icon param-collection %} *Input List"*: output of the last **Filter**
>    - *How should a dataset be selected?"*: `Select by index`
>      - *Element index:"*: `0` to select the first dataset
>
> 2. Add the `#condition` tag
>
> 3. **Extract Dataset from a list** {% icon tool %} to extract 2nd file in the collection with:
>    - {% icon param-collection %} *Input List"*: output of the last **Filter**
>    - *How should a dataset be selected?"*: `Select by index`
>      - *Element index:"*: `1` to select the second dataset
>
> 4. Add the `#control` tag
{: .hands_on}

We would like now to get the list of essential gene in both conditions

> <hands-on-title>Get gene essential in both conditions</hands-on-title>
>
> 1. **Join two files** {% icon tool %} with:
>    - {% icon param-file %} *1st file"*: output of **Extract Dataset** with `control` tag
>    - *Column to use from 1st file"*: `Column : 1` (to compare on gene ID)
>    - {% icon param-file %} *2nd file"*: output of **Extract Dataset** with `condition` tag
>    - *Column to use from 2nd file"*: `Column : 1` (to compare on gene ID)
>    - *Output lines appearing in"*: `Both 1st and 2nd files` (to get common essential genes)
>
{: .hands_on}


Let's now get the list of essential gene that are different between the two conditions.

> <hands-on-title>Get gene essential only in control</hands-on-title>
>
> 1. **Join two files** {% icon tool %} with:
>    - {% icon param-file %} *1st file"*: output of **Extract Dataset** with `control` tag
>    - *Column to use from 1st file"*: `Column : 1` (to compare on gene ID)
>    - {% icon param-file %} *2nd file"*: output of **Extract Dataset** with `condition` tag
>    - *Column to use from 2nd file"*: `Column : 1` (to compare on gene ID)
>    - *Output lines appearing in"*: `1st but not 2nd`
{: .hands_on}



> <hands-on-title>Hands-on : Get gene essential only in condition</hands-on-title>
>
>  1. **Join two files** {% icon tool %} with:
>   - *1st file"*: output of **Extract Dataset** with `control` tag
>   - *Column to use from 1st file"*: `Column : 1` (to compare on gene ID)
>   - *2nd file"*: output of **Extract Dataset** with `condition` tag
>   - *Column to use from 2nd file"*: `Column : 1` (to compare on gene ID)
>   - *Output lines appearing in"*: `2nd but not 1st`
{: .hands_on}


> <question-title></question-title>
>
> 1. How many genes are essential in both conditions?
> 2. How many genes are essential only in control?
> 3. How many genes are essential only in condition?
>
>
> > <solution-title></solution-title>
> >
> > 1. 320 genes are essential in both conditions.
> > 2. 39 genes are essential only in control.
> > 3. 61 genes are essential only in condition.
> {: .solution }
>
{: .question}

# Conclusion

In this tutorial, we learned how to predict essential genes in different conditions. Although it seems to be a long process, keep in mind that the steps allowing to get the list of TA site does not need to be repeated for each analysis as long as you study the same genome.
For repetitive analyses, the steps are fewer : mapping, coverage, attribution to TA sites, merging of strands and Transit analysis.

This tutorial focused on simple gene essentiality prediction in individual samples. You could be interested in intergenic regions as well, or comparing many samples together. Other methods exist in transit for these goals, that follow the same workflow. It is important to look at your data and to perform quality control to evaluate the coverage and sparsity of your data. This will influence not only the choice of method (HMM does not manage sparse data well), but the choice of parameters for your analysis.

Once you get this list of gene, it is interesting to verify manually the genes classified as essentials. In order to do that, you can visualize the read mapped on your genome by displaying the bam files in IGV.

{% snippet topics/visualisation/faqs/visualizations_igv_add_genome.md %}

{% snippet topics/visualisation/faqs/visualizations_igv_view_mapping.md %}

When looking at the read mapping at the location of predicted essential genes, you may encounter several situation: Some gene will have no read at all mapping to them, some will have an empty region, while other will have no clearly defined empty region but a very low count of reads, possibly indicating a growth defect induced by the gene disruption.

![Example of Essential gene with no reads mapped](../../images/tnseq/empty_gene.png "Example of Essential gene with no reads mapped")
![Example of Essential gene with depleted coverage](../../images/tnseq/low_coverage_gene.png "Example of Essential gene with depleted coverage")

Once you have curated and confirm the genes that appear essential, comparing the annotation of essential genes with the condition of growth will allow you a better understanding of certain mechanism of interest, like the gene involved in resistance to stress for example.

A comparison with published results of essentiality in your organism will be useful but don't expect to find exactly the same results. The conditions of growth differ from one experiment to another, and can highly influence the detection of essential genes.


