# tutorials

topic: epigenetics
tutorial: estrogen-receptor-binding-site-identification
title: Identification of the binding sites of the Estrogen receptor
difficulty: intermediate
time_estimation: 3h
tools_json: []
requirements_json: []
tags_json: ["ChIP-seq"]
content_hash: 47bb80763b739263a4abe1f54a5e06bf
last_modified: 2026-05-27T09:01:48.465016
zenodo_link: https://doi.org/10.5281/zenodo.892432
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/epigenetics/tutorials/estrogen-receptor-binding-site-identification/tutorial.html

content:
This exercise uses the dataset from the Nature publication by [Ross-Inness et al., 2012](https://www.ncbi.nlm.nih.gov/pubmed/22217937).
The goal of this article was to identify the binding sites of the Estrogen receptor, a transcription factor known to be associated with different types of breast cancer.

To this end, ChIP-seq was performed in breast cancer cells from 4 patients of different outcomes (good and poor). For each ChIP-seq experiment there is a matching technical control, i.e. there are 8 samples in total:

Patient | Outcome | Treatment
--- | --- | ---
Patient 1 | Good | ChIP ER
Patient 1 | Good | input (no immunoprecipitation step)
Patient 2 | Good | ChIP ER
Patient 2 | Good | input (no immunoprecipitation step)
Patient 3 | Poor | ChIP ER
Patient 3 | Poor | input (no immunoprecipitation step)
Patient 4 | Poor | ChIP ER
Patient 4 | Poor | input (no immunoprecipitation step)

Half of which are the so-called 'input' samples for which the same treatment as the ChIP-seq samples was done except for the immunoprecipitation step.
The input files are used to identify potential sequencing bias, like open chromatin or GC bias.

Because of the long processing time for the large original files, we have downsampled the original data and provide already processed data for subsequent steps.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Step 1: Quality control and treatment of the sequences

The first step of any ChIP-Seq data analysis is quality control of the raw sequencing data.

> <hands-on-title>Quality control</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import `patient1_input_good_outcome` from [Zenodo](https://zenodo.org/record/892432/files/patient1_input_good_outcome.fastq) or from the data library into the history
>
>    > <tip-title>Importing data via links</tip-title>
>    >
>    > * Copy the link location
>    > * Open the Galaxy Upload Manager
>    > * Select **Paste/Fetch Data**
>    > * Paste the link into the text field
>    > * Press **Start** and **Close** the window
>    > * Click on the `pencil` icon once the file is imported
>    > * Click on **Datatype** in the central panel
>    > * Select `fastqsanger` as **New Type**
>    {: .tip}
>
>    > <tip-title>Importing data from a data library</tip-title>
>    >
>    > * Go into "Data" (top panel) then "Data libraries"
>    > * Click on "Training data" and then "Analyses of ChIP-Seq data"
>    > * Select interesting file
>    > * Click on "Import selected datasets into history"
>    > * Import in the history
>    {: .tip}
>
>    As default, Galaxy takes the link as name, so rename them.
>
>
> 3. Inspect the file by clicking on the `eye` icon
>
>    > <question-title></question-title>
>    >
>    > 1. How are the DNA sequences stored?
>    > 2. What are the other entries?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The DNA sequences are stored in the second line of every 4-line group
>    > > 2. This file is called a [FastQ file](https://en.wikipedia.org/wiki/FASTQ_format). It stores sequence information and quality information. Each sequence is represented by a group of 4 lines with the 1st line being the sequence id, the second the sequence of nucleotides, the third a transition line and the last one a sequence of quality score for each nucleotide.
>    > {: .solution }
>    {: .question}
>
> 4. Run **FastQC** {% icon tool %} with
>    - "Short read data from your current history" to the imported file
>
>    Inspect the generated files
{: .hands_on}

It is often necessary to trim sequenced read, for example, to get rid of bases that were sequenced with high uncertainty (= low quality bases).

> <hands-on-title>Quality control</hands-on-title>
>
> 1. Run **Trim Galore!** {% icon tool %} with
>    - "Is this library paired- or single-end?" to `Single-end`
>    - "Reads in FASTQ format" to the imported file
>    - "Trim Galore! advanced settings" to `Full parameter list`
>    - "Trim low-quality ends from reads" to `15`
>    - "Overlap with adapter sequence required to trim a sequence" to `3`
>
>    > <tip-title>Importing data from a data library</tip-title>
>    >
>    > If your FASTQ files cannot be selected, you might check whether their format is FASTQ with Sanger-scaled quality values (`fastqsanger`). You can edit the data type by clicking on the `pencil` symbol.
>    {: .tip}
>
{: .hands_on}

# Step 2: Mapping of the reads

In order to figure where the sequenced DNA fragments originated from in the genome, the short reads must be aligned to the reference genome. This is equivalent to solving a jigsaw puzzles, but unfortunately, not all pieces are unique. In principle, you could do a BLAST analysis to figure out where the sequenced pieces fit best in the known genome. Aligning millions of short sequences this way may, however, take a couple of weeks.

## Running Bowtie2

Nowadays, there are many read alignment programs for shotgun sequenced DNA, Bowtie2 being one of them.

> <hands-on-title>Mapping</hands-on-title>
>
> 1. **Bowtie2** {% icon tool %} with
>    - "Is this single or paired library" to `Single-end`
>    - "FASTA/Q file" to the Trim Galore! output with the trimmed reads
>    - "Will you select a reference genome from your history or use a built-in index?" to `Use a built-in genome index`
>    - "Select reference genome" to `Human (Homo sapiens): hg18`
>
>    > <question-title></question-title>
>    >
>    > 1. How many reads where mapped?
>    >
>    > > <solution-title></solution-title>
>    > > 1. This information can be accessed by clicking on the resulting history entry. You can see some basic mapping statistics once the alignment is completed. 16676 (66.96%) were mapped exactly 1 time and 7919 (31.80%) aligned >1 times. The overall alignment rate is then 98.76%.
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

The read alignment step with bowtie2 resulted in a compressed, binary file (BAM) that is not human-readable. It's like the zipped version of a text file.

We will show you two ways to inspect the file:

1. Visualization using a Genome Browser
2. Converting the binary format into its text file equivalent


## Visualization using a Genome Browser

> <hands-on-title>Visualization of the reads in IGV</hands-on-title>
>
> 1. Click on the `display with IGV local` to load the reads into the IGV browser
> 2. Zoom at the start of chromosome 11 (or `chr11:1,562,200-1,591,483`)
{: .hands_on}

The reads have a direction: they are mapped to the forward or reverse strand, respectively. When hovering over a read, extra information is displayed

> <question-title></question-title>
>
> 1. Some reads have colored lines included. What is this?
>
> > <solution-title></solution-title>
> > 1. Try to zoom in in one of those lines and you will see the answer!
> {: .solution }
{: .question}

> <comment-title></comment-title>
> Because the number of reads over a region can be quite large, the IGV browser by default only allows to see the reads that fall into a small window. This behaviour can be changed in the IGV preferences panel.
{: .comment}


## Inspection of the SAM format

As mentioned above, you can convert the binary BAM file into a simple (but large!) text file, which is called a [SAM](https://en.wikipedia.org/wiki/SAM_(file_format)) (Sequence Alignment Map) file.

> <hands-on-title>Conversion into a SAM file</hands-on-title>
>
> 1. **BAM-to-SAM** {% icon tool %} with
>    - "BAM File to Convert" to the file generated by Bowtie2
>    - "Header options" to `Include header in SAM output`
>
> 2. Inspect the file by clicking on `eye` icon
>
{: .hands_on}

A SAM file is a file with

- A header with the chromosome names and lengths
- A file content as a tabular file with the location and other information of each read found in the FASTQ file and the mapping information

> <question-title></question-title>
>
> 1. Which information do you find in a SAM/BAM file? What is the additional information compared to a FASTQ file.
>
> > <solution-title></solution-title>
> > 1. Sequences and Quality information, like FASTQ
> > 2. Mapping information; Location of the read on the chromosome; Mapping quality ...
> > 3. For more information please see [https://en.wikipedia.org/wiki/SAM_(file_format)](https://en.wikipedia.org/wiki/SAM_(file_format))
> {: .solution }
{: .question}

# Step 3: ChIP-seq Quality Control

We already checked the quality of the raw sequencing reads in the first step.
Now we would like to test the quality of the ChIP-seq preparation, to know if your ChIP-seq samples are more enriched than the control (input) samples.

## Correlation between samples

To assess the similarity between the replicates of the ChIP and the input, respectively, it is a common technique to calculate the correlation of
read counts on different regions for all different samples.
We expect that the replicates of the ChIP-seq experiments should be clustered more closely to each other than the replicates of the input sample.
That is, because the input samples should not have enriched regions included - remember the immuno-precipitation step was skiped during the sample preparation.

To compute the correlation between the samples we are going to to use the QC modules of deepTools (http://deeptools.readthedocs.io/), a software package for the QC, processing and analysis of NGS data. Before computing the correlation a time consuming step is required, which is to compute the read coverage over a large number of regions from each BAM file. For this we will use the tool **multiBamSummary** {% icon tool %}.

To do that, we need at first to catch up for all our samples and re-run the previous steps (quality control and mapping) on each sample.
To save time, we already did that and we can now work directly on the BAM files of the 8 samples

> <hands-on-title>Correlation between samples</hands-on-title>
>
> 1. Create a new history
> 2. Import the 8 BAM files from [Zenodo](https://doi.org/10.5281/zenodo.892432) or from the data library into the history
>    - [`patient1_ChIP_ER_good_outcome.bam`](https://zenodo.org/record/892432/files/patient1_ChIP_ER_good_outcome.bam)
>    - [`patient1_input_good_outcome.bam`](https://zenodo.org/record/892432/files/patient1_input_good_outcome.bam)
>    - [`patient2_ChIP_ER_good_outcome.bam`](https://zenodo.org/record/892432/files/patient2_ChIP_ER_good_outcome.bam)
>    - [`patient2_input_good_outcome.bam`](https://zenodo.org/record/892432/files/patient2_input_good_outcome.bam)
>    - [`patient3_ChIP_ER_poor_outcome.bam`](https://zenodo.org/record/892432/files/patient3_ChIP_ER_poor_outcome.bam)
>    - [`patient3_input_poor_outcome.bam`](https://zenodo.org/record/892432/files/patient3_input_poor_outcome.bam)
>    - [`patient4_ChIP_ER_poor_outcome.bam`](https://zenodo.org/record/892432/files/patient4_ChIP_ER_poor_outcome.bam)
>    - [`patient4_input_poor_outcome.bam`](https://zenodo.org/record/892432/files/patient4_input_poor_outcome.bam)
>
> 3. **multiBamSummary** {% icon tool %} with
>    - "Sample order matters" to `No`
>    - "Bam file" to the 8 imported BAM files
>    - "Choose computation mode" to `Bins`
>    - "Bin size in bp" to `100`
>
>       This corresponds to the length of the fragments that were sequenced; it is not the read length!
>
>    - "Distance between bins" to `500000` (to reduce the computation time for the tutorial)
>    - "Region of the genome to limit the operation to" to `chr1` (to reduce the computation time for the tutorial)
>
>    Using these parameters, the tool will take bins of 100 bp separated by 500,000. For each bin the overlapping reads in each sample will be computed
>
> 4. **plotCorrelation** {% icon tool %} with
>    - "Matrix file from the multiBamSummary tool" to the generated multiBamSummary output
>
>    To compute and visualize the sample correlation we use plotCorrelation from deepTools. This is a fast process that allows the user to quickly try different color combinations and outputs. Feel free to try different parameters.
>
{: .hands_on}

> <question-title></question-title>
>
> ![Output for plotCorrelation with the correlation scores between the 8 samples](../../images/estrogen-receptor-binding-site-identification/plotCorrelation_output.png "Correlation scores between the 8 samples")
>
> 1. How are your samples clustered? Does that correspond to your expectations?
>
> > <solution-title></solution-title>
> > 1. Badly, this is due to the subsampling of our data. You should expect better results from your real data.
> > 2. Expected is that your inputs and your ChIP samples cluster together.
> {: .solution }
{: .question}

> <comment-title></comment-title>
> More information on these two tools can be found at the [deepTools documentation page](https://deeptools.readthedocs.io/en/latest/content/list_of_tools.html).
{: .comment}

## IP strength estimation

To evaluate the quality of the immuno-precipitation step, we can compute the IP strength. It determines how well the signal in the ChIP-seq sample can be differentiated from the background distribution of reads in the control sample. To do that we take the data for one Patient and compare the input sample and the ChIP-seq sample.

> <comment-title></comment-title>
> For more information on how the IP strength is estimated, you can check the [deepTools documentation page](https://deeptools.readthedocs.io/en/latest/content/list_of_tools.html).
{: .comment}

> <hands-on-title>IP strength estimation</hands-on-title>
>
> 1. **plotFingerprint** {% icon tool %} with
>    - "Sample order matters" to `No`
>    - "Bam file" to `patient1_input_good_outcome` and `patient1_ChIP_ER_good_outcome`
>    - "Region of the genome to limit the operation to" to `chr1`
{: .hands_on}

The plotFingerprint tool generates a fingerprint plot. You need to intepret it to know the IP strength. The [deepTools documentation](https://deeptools.readthedocs.io/en/latest/content/list_of_tools.html) explains it clearly:

![A guide to interpret a fingerprint plot](../../images/estrogen-receptor-binding-site-identification/QC_fingerprint.png "How to interpret a fingerprint plot? Image extracted from the deepTools documentation")

> <question-title></question-title>
>
> ![Output for plotFingerprint with the fingerprint plot to estimate the IP strength](../../images/estrogen-receptor-binding-site-identification/plotFingerprint_output.png "Fingerprint plot for the Patient 1 to estimate the IP strength")
>
> 1. What do you think about the quality of the IP for this experiment?
>
> > <solution-title></solution-title>
> >  The difference between input and ChIP signal is not totally clear. 20% of chromosome 1 are not sequenced
> >  at all and the ChIP signal is only slightly more enrichted then the input.
> {: .solution }
{: .question}

> <hands-on-title>(Optional) IP strength estimation (other samples)</hands-on-title>
>
> 1. Run the same analysis on data of the 3 other patients
{: .hands_on}


# Step 4: Normalization

We would like to know where the binding sites of the estrogen receptor are located. For this we need
to extract which parts of the genome have been enriched (more reads mapped) within the samples that underwent immunoprecipitation.

For the normalization we have two options.

1. Normalization by sequencing depth
2. Normalization by input file


## Generation of coverage files normalized by sequencing depth

We first need to make the samples comparable. Indeed, the different samples have usually a different sequencing depth, i.e. a different number of reads.
These differences can bias the interpretation of the number of reads mapped to a specific genome region.

> <hands-on-title>Coverage file normalization</hands-on-title>
>
> 1. **IdxStats** {% icon tool %} with
>    - "BAM file" to "Multiple datasets": `patient1_input_good_outcome` and `patient1_ChIP_ER_good_outcome`
>
>    > <question-title></question-title>
>    >
>    > 1. What is the output of this tool?
>    > 2. How many reads has been mapped on chr2 for the input and for the ChIP-seq samples?
>    >
>    > > <solution-title></solution-title>
>    > > 1. This tool estimates how many reads mapped to which chromosome. Furthermore, it tells the chromosome lengths and naming convention (with or without 'chr' in the beginning)
>    > > 2. 1,089,370 for ChIP-seq samples and 1,467,480 for the input
>    > {: .solution }
>    {: .question}
>
> 2. **bamCoverage** {% icon tool %} with
>    - "BAM file" to "Multiple datasets": `patient1_input_good_outcome` and `patient1_ChIP_ER_good_outcome`
>    - "Bin size in bases" to `25`
>    - "Scaling/Normalization method" to `Normalize coverage to 1x`
>    - "Effective genome size" to `hg19 (2451960000)`
>    - "Coverage file format" to `bedgraph`
>
>    > <question-title></question-title>
>    >
>    > 1. What are the different columns of a `bedgraph` file?
>    >
>    > > <solution-title></solution-title>
>    > > 1. chrom, chromStart, chromEnd and a data value
>    > {: .solution }
>    {: .question}
>
> 3. **bamCoverage** {% icon tool %} with the same parameters but to generate a `bigWig` output file
> 4. **IGV** {% icon tool %} to inspect both signal coverages (input and ChIP samples) in IGV
>
>    > <question-title></question-title>
>    >
>    > 1. What is a bigWig file?
>    >
>    > > <solution-title></solution-title>
>    > > A bigWig file is a compressed bedgraph file. Similar in relation as BAm to SAM, but this time just for coverage data. This means bigWig and bedgraph
>    > > files are much smaller than BAM or SAM files.
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

## Generation of input-normalized coverage files and their visualization

To extract only the information induced by the immunoprecipitation, we normalize for each patient the coverage file for the sample that underwent immunoprecipitation by the coverage file for the input sample. Here we use the tool bamCompare which compare 2 BAM files while caring for sequencing depth normalization.

> <hands-on-title>Generation of input-normalized coverage files</hands-on-title>
>
> 1. **bamCompare** {% icon tool %} with
>    - "First BAM file (e.g. treated sample)" to `patient1_ChIP_ER_good_outcome`
>    - "Second BAM file (e.g. control sample)" to `patient1_input_good_outcome`
>    - "Bin size in bases" to `50`
>    - "How to compare the two files" to `Compute log2 of the number of reads ratio`
>    - "Coverage file format" to `bedgraph`
>    - "Region of the genome to limit the operation to" to `chr11` (to reduce the computation time for the tutorial)
>
>      > <question-title></question-title>
>      >
>      > 1. What does mean a positive or a negative value in the 4th column?
>      >
>      > > <solution-title></solution-title>
>      > > 1. The 4th column contains the log2 of the number of reads ratio between the ChIP-seq sample and the input sample. A positive value means that the coverage on the portion is more important in the ChIP-seq sample than in the input sample
>      > {: .solution }
>      {: .question}
>
> 2. **bamCompare** {% icon tool %} with the same parameters but to generate a `bigWig` output file
> 3. **IGV** {% icon tool %} to inspect the log2 ratio
>
>    Remember that the bigWig file contains only the signal on chromosome 11!
>
{: .hands_on}

# Step 5: Detecting enriched regions (peak calling)

We can also call the enriched regions, or peaks, found in the ChIP-seq samples.

> <hands-on-title>Peak calling</hands-on-title>
>
> 1. **MACS2 callpeak** {% icon tool %} with
>    - "ChIP-Seq Treatment File" to `patient1_ChIP_ER_good_outcome`
>    - "ChIP-Seq Control File" to `patient1_input_good_outcome`
>    - "Effective genome size" to `H. sapiens (2.7e9)`
>    - "Outputs" to `Summary page (html)`
>
>    > <comment-title></comment-title>
>    > The advanced options may be adjusted, depending of the samples.
>    > If your ChIP-seq experiment targets regions of broad enrichment, *e.g.* non-punctuate histone modifications, select calling of broad regions.
>    > If your sample has a low duplication rate (*e.g.* below 10%), you might keep all duplicate reads (tags). Otherwise, you might use the 'auto' option to estimate the maximal allowed number of duplicated reads per genomic location.
>    {: .comment}
>
> 2. **IGV** {% icon tool %} to inspect with the signal coverage and log2 ratio tracks
>
{: .hands_on}

The called peak regions can be filtered by, *e.g.* fold change, FDR and region length for further downstream analysis.

# Step 6: Plot the signal on the peaks between samples

Plotting your region of interest will involve using two tools from the **deepTools** suite.
+ computeMatrix : Computes the signal on given regions, using the bigwig coverage files from different samples.
+ plotHeatmap : Plots heatMap of the signals using the computeMatrix output.

ptionally, you can also use `plotProfile`to create a profile plot using to computeMatrix output.

## computeMatrix

> <hands-on-title>Visualization of the coverage</hands-on-title>
>
> 1. **UCSC Main** {% icon tool %} with
>    - "assembly" to `hg18`
>    - "track" to `RefSeq genes`
>    - "region" to `position` with `chr11`
>    - "output format" to `BED`
>    - "Send output to" to `Galaxy`
>
> 2. **computeMatrix** {% icon tool %} with
>    - "Regions to plot" to the imported UCSC file
>    - "Score file" to the bigwig file generated by bamCompare
>    - "computeMatrix has two main output options" to `scale-regions`
>
>       This option stretches or shrinks all regions in the BED file (here: genes) to the same length (bp) as indicated by the user
>
>    - "Show advanced options" to `yes`
>    - "Convert missing values to 0?" to `Yes`
>
>    This tool prepares a file with scores per genomic region, which is required as input for the next tool.
>
{: .hands_on}

## plotHeatmap

> <hands-on-title>Visualization of the coverage</hands-on-title>
> 3. **plotHeatmap** {% icon tool %} with
>    - "Matrix file from the computeMatrix tool" to the generated matrix
>    - "Show advanced options" to `yes`
>    - "Did you compute the matrix with more than one groups of regions?" to the correct setting
>
{: .hands_on}

# Conclusion


![Summary of the different steps of the tutorial and the generated files](../../images/estrogen-receptor-binding-site-identification/tutorial-scheme.png "Different steps of the tutorials with the generated files")


# Additional exercise (if you have finished all above)

## Additional Quality control : GC bias assessment

A common problem of PCR-based protocols is the observation that GC-rich regions tend to be amplified more readily than GC-poor regions.
We need to check that our samples do not have more reads from regions of the genome with high GC.

> <comment-title></comment-title>
> GC bias was for many years a big problem, but recent advances in sample preparation have solved this problem to a degree that you can skip this step more often.
{: .comment}

For practical reasons, we will focus here only on one of our BAM files. With real data you should repeat these steps for all your samples.

> <question-title></question-title>
>
> 1. Can you guess why it makes more sense to check the input files for GC bias?
>
> > <solution-title></solution-title>
> > 1. We only want to assess the bias induced by the PCR-based protocols. This is not possible with the ChIP samples, as the enriched regions (binding sites) can have a potential GC enrichment on their own.
> {: .solution }
{: .question}

> <hands-on-title>GC bias assessment</hands-on-title>
>
> 1. **computeGCbias** {% icon tool %} with
>    - "BAM file" to `patient1_input_good_outcome`
>    - "Reference genome" to `locally cached`
>    - "Using reference genome" to `Human (Homo sapiens): hg18`
>    - "Effective genome size" to `hg19 (2451960000)`
>    - "Fragment length used for the sequencing" to `300`
>    - "Region of the genome to limit the operation to" to `chr1` (to reduce the computation time for the tutorial)
>
>    > <question-title></question-title>
>    >
>    > ![Output for computeGCbias with the GC bias estimation](../../images/estrogen-receptor-binding-site-identification/computeGCBias_output.png "Estimation of the GC bias for the input sample for the Patient 1")
>    >
>    > 1. Does this dataset have a GC bias?
>    >
>    > > <solution-title></solution-title>
>    > > 1. There is no significantly more reads in the GC-rich regions.
>    > {: .solution }
>    {: .question}
{: .hands_on}

## Plotting heatmap from multiple samples with clustering

> <hands-on-title>plotting multiple samples</hands-on-title>
>
> 1. Run **bamCompare** {% icon tool %} with  same parameters as above, for all 4 patients:
>   - "First BAM file (e.g. treated sample)" to `patient2_ChIP_ER_good_outcome`, `patient3_ChIP_ER_good_outcome`, `patient4_ChIP_ER_good_outcome`
>   - "Second BAM file (e.g. control sample)" to `patient2_input_good_outcome`, `patient3_input_good_outcome`, `patient4_input_good_outcome`
>
> 2. Perform peak calling again using treatment file : `patient2_ChIP_ER_good_outcome` and control `patient2_input_good_outcome`, using macs2 parameters same as above.
>
> 3. Concatenate the outputs (summits in BED) from `patient1` and `patient2` using `Operate on Genomic Intervals` --> `Concatenate`
>
> 4. Sort the output `Operate on Genomic Intervals` --> `sortBED`
>
> 5. Merge the overlapping intervals using `Operate on Genomic Intervals` --> `MergeBED`
>
> 6. **computeMatrix** {% icon tool %} with the same parameters but:
>    - Regions to plot : select the merged bed from above
>    - Output option : `reference-point`
>    - The reference point for the plotting: `center of region`
>    - Distance upstream of the start site of the regions defined in the region file : `3000`
>    - Distance downstream of the end site of the given regions: `3000`
>
>    With this option, it considers only those genomic positions before (downstream) and/or after (upstream) a reference point (*e.g.* TSS, which corresponds to the annotated gene start in our case)
>
> 7. **plotHeatmap** {% icon tool %} with
>    - "Matrix file from the computeMatrix tool" to the generated matrix
>    - "Show advanced options" to `yes`
>    - "Did you compute the matrix with more than one groups of regions?" to `No, I used only one group`
>    - "Clustering algorithm" to `Kmeans clustering`
> Inspect the output
>
{: .hands_on}


topic: epigenetics
tutorial: methylation-seq
title: DNA Methylation data analysis
difficulty: intermediate
time_estimation: 3h
requirements_json: []
tags_json: []
content_hash: e2dd0eeb590c574463da1df5a0b2e2d9
last_modified: 2026-05-27T09:01:48.466406
zenodo_link: https://zenodo.org/record/557099
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/epigenetics/tutorials/methylation-seq/tutorial.html

content:
We will use a small subset of the original data. If we would do the computation on the orginal data the computation time for a tutorial is too long. To show you all necessary steps for Methyl-Seq we decided to use a subset of the data set. In a second step we use precomputed data from the study to show you different levels of methylation. We will consider samples from normal breast cells (NB), fibroadenoma (noncancerous breast tumor, BT089), two invasive ductal carcinomas (BT126, BT198) and a breast adenocarcinoma cell line (MCF7).

This tutorial is based off of {% cite Lin2015 %}. The data we use in this tutorial is available at [Zenodo](https://zenodo.org/record/557099).


> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Data upload

We will start by loading the example dataset which will be used for the tutorial into Galaxy


> <hands-on-title>Get the data into Galaxy</hands-on-title>
>
>
> 1. Create a new history
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the two example datasets from Zenodo or the shared data library:
>
>    ```
>    https://zenodo.org/record/557099/files/subset_1.fastq
>    https://zenodo.org/record/557099/files/subset_2.fastq
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}


# Quality Control

The first step in any analysis should always be quality control. We will use the [Falco](https://falco.readthedocs.io/en/latest/) tool to asses the quality of our reads and determine if we need to perform any data cleaning before proceeding with our analysis. Falco is an efficiency optimized rewrite of [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/)

> <hands-on-title>Quality Control</hands-on-title>
>
> 1. {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} with the following parameters:
>    - {% icon param-files %} *"Raw read data from your current history"*:  `subset_1.fastq.gz` and `subset_2.fastq.gz`
>
>    {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>
> 2. Go to the web page result page and have a closer look at 'Per base sequence content'
>
>    ![Falco webpage results](../../images/methylation_falco.png)
>
>    > <question-title></question-title>
>    >
>    > 1. Note the GC distribution and percentage of "T" and "C". Why is this so weird?
>    > 2. Is everything as expected?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The attentive audience of the theory part knows: Every C-meth stays a C and every normal C becomes a T during the bisulfite conversion.
>    > > 2. Yes it is. Always be careful and have the specific characteristics of your data in mind during the interpretation of Falco results.
>    > {: .solution }
>    {: .question}
>
{: .hands_on}


# Alignment

> <hands-on-title>Mapping with bwameth</hands-on-title>
>
> We will now map the imported dataset against a reference genome.
>
> 1. {% tool [bwameth](toolshed.g2.bx.psu.edu/repos/iuc/bwameth/bwameth/0.2.7+galaxy0) %} with the following parameters:
>    - *"Select a genome reference from your history or a built-in index?"*: `Use a built-in index`
>        - *"Select a reference genome"*: `Human (hg38full)`
>    - *"Is this library mate-paired"*: `Paired-end`
>        - *"First read in pair"*: `subset_1.fastq`
>        - *"Second read in pair"*: `subset_2.fastq`
>
>    > <comment-title>Long compute times</comment-title>
>    > Please notice that mapping can take some time. If you want to skip this, we provide for you a precomputed alignment. Import `https://zenodo.org/records/557099/files/aligned_subset.bam` to your history.
>    {: .comment}
>
>    > <question-title></question-title>
>    >
>    > Why we need other alignment tools for bisulfite sequencing data?
>    >
>    > > <solution-title></solution-title>
>    > > You may have noticed that all the C's are C-meth's and a T can be a T or a C. A mapper for methylation data needs to find out what is what.
>    > {: .solution }
>    {: .question}
>
{: .hands_on}


# Methylation bias and metric extraction

> <hands-on-title>Methylation bias</hands-on-title>
>
> In this step we will have a look at the distribution of the methylation and will look at a possible bias.
>
> 1. {% tool [MethylDackel](toolshed.g2.bx.psu.edu/repos/bgruening/pileometh/pileometh/0.5.2+galaxy0) %} with the following parameters:
>    - *"Load reference genome from"*: `Local cache`
>        - *"Using reference genome"*: `Human (hg38)`
>    - *"Sorted BAM file"*: output of **bwameth** {% icon tool %}
>    - *"What do you want to do?"*: `Determine the position-dependent methylation bias in the dataset, producing diagnostic SVG images (mbias)`
>    - In *"Advanced options"*
>        - *"Keep singletons"*: {% icon param-toggle %} `Yes`
>        - *"Keep discordant alignmetns"*: {% icon param-toggle %} `Yes`
>
>    ![Methylation bias example](../../images/methylation_bias_example_data.png)
>
>    > <question-title></question-title>
>    >
>    > 1. Consider the `original top strand` output. Is there a methylation bias?
>    > 2. If we would trim, what would be the start and the end positions?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The distribution of the methylation is more or less equal. Only at the start and the end we could trim a bit but a +- 5% variation is acceptable.
>    > > 2. To trim the reads we would include for the first strand only the positions 0 to 145, for the second 6 to 149.
>    > {: .solution }
>    {: .question}
>
>
{: .hands_on}


> <hands-on-title>Methylation extraction with MethylDackel</hands-on-title>
>
>
> We will extract the methylation on the resulting BAM file of the alignment step. We need this to create a methylation level plot in the next step.
>
> 1. {% tool [MethylDackel](toolshed.g2.bx.psu.edu/repos/bgruening/pileometh/pileometh/0.5.2+galaxy0) %} with the following parameters:
>    - *"Load reference genome from"*: `Local cache`
>        - *"Using reference genome"*: `Human (hg38)`
>    - *"Sorted BAM file"*: output of **bwameth** {% icon tool %}
>    - *"What do you want to do?"*: `Extract methylation metrics from an alignment file in BAM/CRAM format (extract)`
>    - *"Merge per-Cytosine metrics"*: {% icon param-toggle %} `Yes`
>    - *"Output options"*: `CpG methylation fractions (--fraction)`
>
{: .hands_on}


# Visualization

> <hands-on-title></hands-on-title>
>
> In this step we want to visualize the methylation level around all TSS of our data. When located at gene promoters, DNA methylation is usually a repressive mark.
>
> 1. {% tool [Wig/BedGraph-to-bigWig](wig_to_bigWig) %} with the following parameters:
>    - *"Convert"*: `fraction CpG` (result of **MethylDackel** {% icon tool %})
>
>      > <tip-title>Database edit</tip-title>
>      >
>      > It can happen that you can not select the correct input file. In this case you have to add meta information about the used genome to the file.
>      > * Click on the pencil of the correct history item.
>      > * Change `Database/Build:` to the genome you used.
>      > * In our case the correct genome is `Human Dec. 2013 (GRCh38/hg38) (hg38)`.
>      {: .tip}
>
> 2. Import the BED file with CpG islands from [Zenodo](https://zenodo.org/record/557099) into the history
>
>    ```
>    https://zenodo.org/records/557099/files/CpGIslands.bed
>    ```
>
> 3. {% tool [computeMatrix](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.5.4+galaxy0) %} with the following parameters:
>    - *"Regions to plot"*: `CpGIslands.bed`
>    - *"Sample order matters"*: `No`
>    - *"Score file"*: Output of **Wig/BedGraph-to-bigWig** {% icon tool %}
>    - *"computeMatrix has two main output options"*: `reference-point`
>
> 4. {% tool [plotProfile](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_profile/deeptools_plot_profile/3.5.4+galaxy0) %} with the following parameters:
>    - *"Matrix file from the computeMatrix tool"*: `Matrix` (output of **computeMatrix** {% icon tool %})
>
> The output should look like this:
>
> ![Methylation output](../../images/methylation_output.png)
>
> Lets see how the methylation looks for a few provided files:
>
> 1. Import the BED file with CpG islands from [Zenodo](https://zenodo.org/record/557099) into the history
>
>    ```
>    https://zenodo.org/records/557099/files/NB1_CpG.meth.bedGraph
>    ```
>
> 2. {% tool [Wig/BedGraph-to-bigWig](wig_to_bigWig) %} with the following parameters:
>    - *"Convert"*: `NB1_CpG.meth.bedGraph`
>
>    > <question-title></question-title>
>    >
>    > The execution fails. Do you have an idea why?
>    >
>    > > <solution-title></solution-title>
>    > > A conversion to bigWig would fail right now. If it turned green, the file size should be 0 bytes. Probably dataset info box shows some error message like `hashMustFindVal: '1' not found`. The reason is the source of the reference genome which was used. There is ensembl and UCSC as sources which differ in naming the chromosomes. Ensembl is using just numbers e.g. 1 for chromosome one. UCSC is using chr1 for the same. Be careful with this especially if you have data from different sources. We need to convert this.
>    > {: .solution }
>    {: .question}
>
>    > <comment-title>UCSC - Ensembl convert</comment-title>
>    >
>    >
>    > Download the file containing mapping between Ensembl and UCS chromosome convention of hg38
>    >
>    > ```
>    > https://raw.githubusercontent.com/dpryan79/ChromosomeMappings/master/GRCh38_ensembl2UCSC.txt
>    > ```
>    >
>    > {% tool [Replace column](toolshed.g2.bx.psu.edu/repos/bgruening/replace_column_by_key_value_file/replace_column_with_key_value_file/0.2) %} with the follwing parameters:
>    >    - *"File in which you want to replace some values"*: `NB1_CpG.meth.bedGraph`
>    >    - *"Replace information file"*: `GRCh38_ensembl2UCSC.txt`
>    >    - *"Which column should be replaced?"*: `Column: 1`
>    >    - *"Skip this many starting lines"*: `1`
>    >    - *"Delimited by"*: `Tab`
>    {: .comment}
>
> 4. To save compute time we prepared the converted files for you. Import the following files. Create a collection list and label it `all_coverage_files`. Strip the file extension from the name. For example, rename from `NB1_CpG.meth_ucsc.bedGraph`to `NB1_CpG`.
>
>    ```
>    https://zenodo.org/records/557099/files/NB1_CpG.meth_ucsc.bedGraph
>    https://zenodo.org/records/557099/files/NB2_CpG.meth_ucsc.bedGraph
>    https://zenodo.org/records/557099/files/BT089_CpG.meth_ucsc.bedGraph
>    https://zenodo.org/records/557099/files/BT126_CpG.meth_ucsc.bedGraph
>    https://zenodo.org/records/557099/files/BT198_CpG.meth_ucsc.bedGraph
>    https://zenodo.org/records/557099/files/MCF7_CpG.meth_ucsc.bedgraph
>    ```
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
> 5. Change the datatype to `bedgraph` and set the database to `hg38`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="bedgraph" %}
>
>    {% snippet faqs/galaxy/datasets_change_dbkey.md dbkey="hg38" %}
>
> 6. {% tool [Wig/BedGraph-to-bigWig](wig_to_bigWig) %} with the following parameters:
>    - {% icon param-collection %} *"Convert"*: `all_coverage_files`
>
> 7. {% tool [computeMatrix](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.5.4+galaxy0) %} with the following parameters:
>    - *"Regions to plot"*: `CpGIslands.bed`
>    - *"Sample order matters"*: `No`
>    - *"Score file"*: Output of previous **Wig/BedGraph-to-bigWig** {% icon tool %}
>    - *"computeMatrix has two main output options"*: `reference-point`
>
> 8. {% tool [plotProfile](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_profile/deeptools_plot_profile/3.5.4+galaxy0) %} with the following parameters:
>    - *"Matrix file from the computeMatrix tool"*: `Matrix` (output of previous **computeMatrix** {% icon tool %})
>    - *"Show advanced options"*: `Yes`
>       - *"Make one plot per group of regions"*: {% icon param-toggle %} `Yes`
>
> The output should look like this:
>
> ![Methylation level around TSS](../../images/methyl_level.png)
>
{: .hands_on}

# Metilene

> <hands-on-title>Metilene</hands-on-title>
>
> With metilene it is possible to detect differentially methylated regions (DMRs) which is a necessary prerequisite for characterizing different epigenetic states.
>
>
> 1. Import the following files from [Zenodo](https://zenodo.org/record/557099) into yout history
>
>    ```
>    https://zenodo.org/records/557099/files/NB1_CpG.meth.bedGraph
>    https://zenodo.org/records/557099/files/NB2_CpG.meth.bedGraph
>    https://zenodo.org/records/557099/files/BT198_CpG.meth.bedGraph
>    ```
>
> 2. {% tool [Metilene](toolshed.g2.bx.psu.edu/repos/rnateam/metilene/metilene/0.2.6.1) %} with the following parameters:
>    - *"Input group 1"*: `NB1_CpG.meth.bedGraph` and `NB2_CpG.meth.bedGraph`
>    - *"Input group 2"*: `BT198_CpG.meth.bedGraph`
>    - *"BED file containing regions of interest"*: `CpGIslands.bed`
>
>    > <question-title></question-title>
>    >
>    > Have a look at the produced pdf document. What is the data showing?
>    >
>    > > <solution-title></solution-title>
>    > > It shows the distribution of DMR differences, DMR length in nucleotides and number CpGs, DMR differences vs. q-values, mean methylation group 1 vs. mean methylation group 2 and DMR length in nucleotides vs. length in CpGs
>    > {: .solution }
>    {: .question}
>
>
{: .hands_on}

tools_json:
["Falco", "toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0", "bwameth", "toolshed.g2.bx.psu.edu/repos/iuc/bwameth/bwameth/0.2.7+galaxy0", "MethylDackel", "toolshed.g2.bx.psu.edu/repos/bgruening/pileometh/pileometh/0.5.2+galaxy0", "Wig/BedGraph-to-bigWig", "wig_to_bigWig", "computeMatrix", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.5.4+galaxy0", "plotProfile", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_profile/deeptools_plot_profile/3.5.4+galaxy0", "Replace column", "toolshed.g2.bx.psu.edu/repos/bgruening/replace_column_by_key_value_file/replace_column_with_key_value_file/0.2", "Metilene", "toolshed.g2.bx.psu.edu/repos/rnateam/metilene/metilene/0.2.6.1"]


topic: epigenetics
tutorial: atac-seq
title: ATAC-Seq data analysis
difficulty: intermediate
time_estimation: 3h
requirements_json: []
tags_json: []
content_hash: 51abc81eeeea616e44b9a18d27d4d262
last_modified: 2026-05-27T09:01:48.463949
zenodo_link: https://zenodo.org/record/3862793
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/epigenetics/tutorials/atac-seq/tutorial.html

content:
In many eukaryotic organisms, such as humans, the genome is tightly packed and organized with the help of nucleosomes (chromatin). A nucleosome is a complex formed by eight histone proteins that is wrapped with ~147bp of DNA. When the DNA is being actively transcribed into RNA, the DNA will be opened and loosened from the nucleosome complex. Many factors, such as the chromatin structure, the position of the nucleosomes, and histone modifications, play an important role in the organization and accessibility of the DNA. Consequently, these factors are also important for the activation and inactivation of genes. **A**ssay for **T**ransposase-**A**ccessible **C**hromatin using **seq**uencing ([ATAC-Seq](https://en.wikipedia.org/wiki/ATAC-seq)) is a method to investigate the accessibility of chromatin and thus a method to determine regulatory mechanisms of gene expression. The method can help identify promoter regions and potential enhancers and silencers. A promoter is the DNA region close to the transcription start site (TSS). It contains binding sites for transcription factors that will recruit the RNA polymerase. An enhancer is a DNA region that can be located up to 1 Mb downstream or upstream of the promoter. When transcription factors bind an enhancer and contact a promoter region, the transcription of the gene is increased. In contrast, a silencer decreases or inhibits the gene's expression. ATAC-Seq has become popular for identifying accessible regions of the genome as it's easier, faster and requires less cells than alternative techniques, such as FAIRE-Seq and DNase-Seq.

![ATAC-Seq](../../images/atac-seq/atac-seq.jpeg "Buenrostro et al. 2013 Nat Methods")

With ATAC-Seq, to find accessible (open) chromatin regions, the genome is treated with a hyperactive derivative of the Tn5 transposase. A [transposase](https://en.wikipedia.org/wiki/Transposase) can bind to a [transposable element](https://en.wikipedia.org/wiki/Transposable_element), which is a DNA sequence that can change its position (jump) within a genome (read the two links to get a deeper insight). During ATAC-Seq, the modified Tn5 inserts DNA sequences corresponding to truncated Nextera adapters into open regions of the genome and concurrently, the DNA is sheared by the transposase activity. The read library is then prepared for sequencing, including PCR amplification with full Nextera adapters and purification steps. Paired-end reads are recommended for ATAC-Seq for the reasons described [in their guidelines](https://informatics.fas.harvard.edu/atac-seq-guidelines.html).

In this tutorial we will use data from the study of {% cite Buenrostro2013 %}, the first paper on the ATAC-Seq method. The data is from a human cell line of purified CD4+ T cells, called GM12878. The original dataset had 2 x 200 million reads and would be too big to process in a training session, so we downsampled the original dataset to 200,000 randomly selected reads. We also added about 200,000 reads pairs that will map to chromosome 22 to have a good profile on this chromosome, similar to what you might get with a typical ATAC-Seq sample (2 x 20 million reads in original FASTQ). Furthermore, we want to compare the predicted open chromatin regions to the known binding sites of CTCF, a DNA-binding protein implicated in 3D structure: [CTCF](https://en.wikipedia.org/wiki/CTCF). CTCF is known to bind to thousands of sites in the genome and thus it can be used as a positive control for assessing if the ATAC-Seq experiment is good quality. Good ATAC-Seq data would have accessible regions both within and outside of TSS, for example, at some CTCF binding sites. For that reason, we will download binding sites of CTCF identified by ChIP in the same cell line from ENCODE (ENCSR000AKB, dataset ENCFF933NTR).

## When working with real data

When you use your own data we suggest you to use [this workflow](https://usegalaxy.eu/u/ldelisle/w/atac-seq-gtm-with-control) which includes the same steps but is compatible with replicates. If you do not have any control data you can import and edit this workflow, removing all steps with the controls. Controls for the ATAC-Seq procedure are not commonly performed, as discussed [in the guidelines](https://informatics.fas.harvard.edu/atac-seq-guidelines.html), but could be ATAC-Seq of purified DNA.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


{% snippet faqs/galaxy/analysis_results_may_vary.md %}

# Preprocessing

## Get Data

We first need to download the sequenced reads (FASTQs) as well as other annotation files. Then, to increase the number of reads that will map to the reference genome (here human genome version 38, GRCh38/hg38), we need to preprocess the reads.


> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo](https://doi.org/10.5281/zenodo.3862792) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/record/3862793/files/ENCFF933NTR.bed.gz
>    https://zenodo.org/record/3862793/files/SRR891268_chr22_enriched_R1.fastq.gz
>    https://zenodo.org/record/3862793/files/SRR891268_chr22_enriched_R2.fastq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Add a tag called `#SRR891268_R1` to the R1 file and a tag called `#SRR891268_R2` to the R2 file.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
> 4. Check that the datatype of the 2 FASTQ files is `fastqsanger.gz` and the peak file (ENCFF933NTR.bed.gz) is `encodepeak`. If they are not then change the datatype as described below.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Create a paired collection named `Paired Reads`
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
{: .hands_on}

> <comment-title>FASTQ format</comment-title>
> If you are not familiar with FASTQ format, see the [Quality Control tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %})
{: .comment}
>
> <comment-title>BED / encode narrowPeak format</comment-title>
> If you are not familiar with BED format or encode narrowPeak format, see the [BED Format](https://genome.ucsc.edu/FAQ/FAQformat.html)
{: .comment}

We will visualise regions later in the analysis and obtain the gene information now. We will get information for chromosome 22 genes (names of transcripts and genomic positions) using the UCSC tool.

> <hands-on-title>Obtain Annotation for hg38 genes</hands-on-title>
>
> 1. {% tool UCSC Main table browser %} with the following parameters:
>    - *"clade"*: `Mammal`
>    - *"genome"*: `Human`
>    - *"assembly"*: `Dec. 2013 (GRCh38/hg38)`
>    - *"group"*: `Genes and Gene Prediction`
>    - *"track"*: `All GENCODE V37`
>    - *"table"*: `Basic`
>    - *"region"*: `position` `chr22`
>    - *"output format"*: `all fields from selected table`
>    - *"Send output to"*: `Galaxy`
> 2. Click **get output**
> 3. Click **Send query to Galaxy**
>
>    This table contains all the information but is not in a BED format. To transform it into BED format we will cut out the required columns and rearrange:
>
> 4. {% tool [Cut columns from a table](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c3,c5,c6,c13,c12,c4`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: `UCSC Main on Human: wgEncodeGencodeBasicV37 (chr22:1-50,818,468)`
>
> 5. Check the contents of your file, is this as you expect it to be?
>
>    > <question-title>Expected output</question-title>
>    >
>    > Our goal here was to convert the data to BED format.
>    >
>    > 1. Which columns do you expect in your file? (Tip: read about [BED format](https://genome.ucsc.edu/FAQ/FAQformat.html#format1))
>    > 2. Does your file look like a valid BED format?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. We expect at least 3 columns, `chromosome - start - end`, and possibly more optional columns
>    > > 2. Your file should look something like this:
>    > >    ```
>    > >    chr22	10736170	10736283	U2	0	-
>    > >    chr22	11066417	11068174	CU104787.1	0	+
>    > >    chr22	11249808	11249959	5_8S_rRNA	0	-
>    > >    chr22	15273854	15273961	U6	0	+
>    > >    [..]
>    > >    ```
>    > >
>    > > - **Troubleshooting:** Is your second column the `Strand` column?
>    > >    - Make sure you used the correct **Cut** {% icon tool %} (the one that matches the tool name mentioned in the previous step *exactly*)
>    > >    - There is another tool with `(cut)` behind the title, we do NOT want to use this tool in this step.
>    > >
>    > > - **Tip:** Always check your output files to make sure they match your expectations!
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
>
> 6. **Rename** {% icon galaxy-pencil %} the dataset as `chr22 genes`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 7. **Change** {% icon galaxy-pencil %} its datatype to BED
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="bed" %}
>
> 8. Click on the {% icon galaxy-eye %} (eye) icon of the file. It should have now column names and they should match the content.
>
{: .hands_on}

>
> <comment-title>Gene file</comment-title>
> The chr22 genes BED we produced only contains the start, the end, the name, and the strand of each transcript. It does not contain exon information.
> To be able to have the exon information, you could use a GTF file which can be downloaded from the [gencode website](ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_37/gencode.v37.annotation.gtf.gz) but this file would include the information for the whole genome and would slow the analysis.
{: .comment}

## Quality Control

The first step is to check the quality of the reads and the presence of the Nextera adapters. When we perform ATAC-Seq, we can get DNA fragments of about 40 bp if two adjacent Tn5 transposases cut the DNA {% cite Adey2010 %}. This can be smaller than the sequencing length so we expect to have Nextera adapters at the end of those reads. We can assess the reads with **FastQC**.

> <hands-on-title>Task description</hands-on-title>
>
> 1. Run {% tool [Flatten collection](__FLATTEN__) %} with the following parameters:
>    - *"Input collection"*: `Paired Reads`
>   
> 2. Rename the flatten collection: `Flat Collection`
>   
> 3. Run {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy1) %} with the following parameters:
>    - *"Raw read data from your current history"*: `Flat Collection` (Flattened paired end read dataset collection)
>
> 4. Inspect the web page output of **FastQC** {% icon tool %} for the `SRR891268_R1` sample. Check what adapters are found at the end of the reads.
>
> > <question-title></question-title>
> >
> > 1. How many reads are in the FASTQ?
> > 2. Which sections have a warning?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. There are 285247 reads.
> > > 2. The 3 steps below have warnings:
> > >
> > >    1. **Per base sequence content**
> > >
> > >       It is well known that the Tn5 has a strong sequence bias at the insertion site. You can read more about it in {% cite Green2012 %}.
> > >
> > >    2. **Sequence Duplication Levels**
> > >
> > >       The read library quite often has PCR duplicates that are introduced
> > >       simply by the PCR itself. We will remove these duplicates later on.
> > >
> > >    3. **Overrepresented sequences**
> > >
> > >       One sequence is over represented: 
> > >       you have 306 reads which are exactly the sequence of the Nextera adapter.
> > >       They correspond to adapters amplified head-to-head.
> > >       306 is really low (only 0.1% of reads).
> > >
> > {: .solution}
> >
>    {: .question}
{: .hands_on}

> <comment-title>FastQC Results</comment-title>
> This is what you should expect from the **Adapter Content** section:
> ![FastQC screenshot of the Adapter Content section](../../images/atac-seq/Screenshot_fastqcBeforecutadapt.png "FastQC screenshot on the Adapter Content section")
{: .comment}

The FastQC web page **Adapter Content** section shows the presence of Nextera Transposase Sequence in the reads. We will remove the adapters with Cutadapt.

## Trimming Reads

To trim the adapters we provide the Nextera adapter sequences to **Cutadapt**. These adapters are shown in the image below.

![Nextera library with the sequence of adapters](../../images/atac-seq/nexteraLibraryPicture.svg.png "Nextera library with the sequence of adapters")

The forward and reverse adapters are slightly different. We will also trim low quality bases at the ends of the reads (quality less than 20). We will only keep reads that are at least 20 bases long. We remove short reads (< 20bp) as they are not useful, they will either be thrown out by the mapping or may interfere with our results at the end.


> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [Cutadapt](toolshed.g2.bx.psu.edu/repos/lparsons/cutadapt/cutadapt/5.1+galaxy0) %} with the following parameters:
>    - *"Single-end or Paired-end reads?"*: `Paired-end Collection`
>        - {% icon param-collection %} *"Paired Collection"*: `Paired Reads`
>        - In *"Read 1 Adapters"*:
>            - In *"3' (End) Adapters"*:
>                - {% icon param-repeat %} *"Insert 3' (End) Adapters"*
>                    - *"Source"*: `Enter custom sequence`
>                        - *"Enter custom 3' adapter name (Optional if Multiple output is 'No')"*: `Nextera R1`
>                        - *"Enter custom 3' adapter sequence"*: `CTGTCTCTTATACACATCTCCGAGCCCACGAGAC`
>        - In *"Read 2 Adapters"*:
>            - In *"3' (End) Adapters"*:
>                - {% icon param-repeat %} *"Insert 3' (End) Adapters"*
>                    - *"Source"*: `Enter custom sequence`
>                        - *"Enter custom 3' adapter name (Optional)"*: `Nextera R2`
>                        - *"Enter custom 3' adapter sequence"*: `CTGTCTCTTATACACATCTGACGCTGCCGACGA`
>    - In *"Other Read Trimming Options"*:
>        - *"Quality cutoff(s) (R1)"*: `20` (will be applied to both forward and reverse reads)
>    - In *"Read Filtering Options"*:
>        - *"Minimum length (R1)"*: `20` (will be applied to both forward and reverse reads)
>    - In *"Additional outputs to generate"*:
>        - Select `Report: Cutadapt's per-adapter statistics. You can use this file with MultiQC. `
>
> 2. Rename the Cutadapt output `Reads` collection: `Adapter Trimmed Reads`
> 
> 3. Click on the {% icon galaxy-eye %} (eye) icon of the report and read the first lines.
{: .hands_on}

> <comment-title>Cutadapt Results</comment-title>
> You should get similar output to this from Cutadapt:
> ![Summary of cutadapt](../../images/atac-seq/Screenshot_cutadaptSummary.png "Summary of cutadapt")
{: .comment}


> <question-title></question-title>
>
> 1. What percentage of reads contain adapters?
> 2. What percentage of reads are still longer than 20bp after the trimming?
>
> > <solution-title></solution-title>
> >
> > 1. ~14%
> > 2. ~99%
> >
> {: .solution}
>
{: .question}

> <hands-on-title>Check Adapter Removal with FastQC</hands-on-title>
>
> 1. Run {% tool [Flatten collection](__FLATTEN__) %} with the following parameters:
>    - *"Input collection"*: `Adapter Trimmed Reads`
>   
> 2. Rename the flatten collection: `Flat Trimmed Reads`
>   
> 3. Run {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy1) %} with the following parameters:
>    - *"Raw read data from your current history"*: `Flat Trimmed Reads`
>
> 4. Click on the {% icon galaxy-eye %} (eye) icon of the report and read the first lines.
{: .hands_on}

> <comment-title>FastQC Results</comment-title>
> Now, you should see under **Overrepresented sequences** that there is no more overrepresented sequences and under **Adapter Content** that the Nextera adapters are no longer present.
> ![FastQC screenshot on the adapter content section after cutadapt](../../images/atac-seq/Screenshot_fastqcAftercutadapt.png "FastQC screenshot on the adapter content section after cutadapt")
> However, you may have noticed that you have a new section with warning: **Sequence Length Distribution**. This is expected as you trimmed part of the reads.
{: .comment}

# Mapping

## Mapping Reads to Reference Genome

Next we map the trimmed reads to the human reference genome. Here we will use **Bowtie2**. We will extend the maximum fragment length (distance between read pairs) from 500 to 1000 because we know some valid read pairs are from this fragment length. We will use the `--very-sensitive` parameter to have more chance to get the best match even if it takes a bit longer to run. We will run the **end-to-end** mode because we trimmed the adapters so we expect the whole read to map, no clipping of ends is needed. Regarding the genome to choose. The hg38 version of the human genome contains [alternate loci](https://www.ncbi.nlm.nih.gov/grc/help/definitions/#ALTERNATE). This means that some region of the genome are present both in the canonical chromosome and on its alternate loci. The reads that map to these regions would map twice. To be able to filter reads falling into repetitive regions but keep reads falling into regions present in alternate loci, we will map on the Canonical version of hg38 (only the chromosome with numbers, chrX, chrY, and chrM).

> <comment-title>Dovetailing</comment-title>
> We will allow dovetailing of read pairs with Bowtie2. This is because adapters are removed by Cutadapt only when at least 3 bases match the adapter sequence, so it is possible that after trimming a read can contain 1-2 bases of adapter and go beyond it's mate start site. For example, if the first mate in the read pair is: `GCTATGAAGAATAGGGCGAAGGGGCCTGCGGCGTATTCGATGTTGAAGCT` and the second mate is `CTTCAACATCGAATACGCCGCAGGCCCCTTCGCCCTATTCTTCATAGCCT`, where both contain 2 bases of adapter sequence, they will not be trimmed by Cutadapt and will map this way:
> ```
<--------------------Mate 1-----------------------
AGCTTCAACATCGAATACGCCGCAGGCCCCTTCGCCCTATTCTTCATAGC
  CTTCAACATCGAATACGCCGCAGGCCCCTTCGCCCTATTCTTCATAGCCT
  ----------------------Mate 2--------------------->
```
> This is what we call dovetailing and we want to consider this pair as a valid concordant alignment.
{: .comment}


> <hands-on-title>Mapping reads to reference genome</hands-on-title>
>
> 1. {% tool [Bowtie2](toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.4+galaxy0) %} with the following parameters:
>    - *"Is this single or paired library"*: `Paired-end`
>        - {% icon param-collection %} *"FASTQ Paired Dataset"*: `Adapter Trimmed Reads`
>        - *"Do you want to set paired-end options?"*: `Yes`
>            - *"Set the maximum fragment length for valid paired-end alignments"*: `1000`
>            - *"Allow mate dovetailing"*: `Yes`
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>        - *"Select reference genome"*: `Human (Homo sapiens): hg38 Canonical`
>    - *"Set read groups information?"*: `Do not set`
>    - *"Select analysis mode"*: `1: Default setting only`
>        - *"Do you want to use presets?"*: `Very sensitive end-to-end (--very-sensitive)`
>    - *"Do you want to tweak SAM/BAM Options?"*: `No`
>    - *"Save the bowtie2 mapping statistics to the history"*: `Yes`
>
> 2. Click on the {% icon galaxy-eye %} (eye) icon of the mapping stats.
{: .hands_on}

> <comment-title>Bowtie2 Results</comment-title>
> You should get similar results to this from Bowtie2:
> ![Mapping statistics of bowtie2](../../images/atac-seq/Screenshot_bowtie2MappingStats.png "Mapping statistics of bowtie2")
{: .comment}

> <question-title></question-title>
>
> What percentage of read pairs mapped concordantly?
>
> > <solution-title></solution-title>
> >
> > 54.8+42.87=97.67%
> >
> {: .solution}
>
{: .question}

> <comment-title>On the number of uniquely mapped.</comment-title>
>
> You might be surprised by the number of uniquely mapped compared to the number of multi-mapped reads (reads mapping to more than one location in the genome).
> One of the reasons is that we have used the parameter `--very-sensitive`. Bowtie2 considers a read as multi-mapped even if the second hit has a much lower quality than the first one.
> Another reason is that we have reads that map to the mitochondrial genome. The mitochondrial genome has a lot of regions with similar sequence.
>
{: .comment}

# Filtering Mapped Reads

## Filter Uninformative Reads

We apply some filters to the reads after the mapping. ATAC-Seq datasets can have a lot of reads that map to the mitchondrial genome because it is nucleosome-free and thus very accessible to Tn5 insertion. The mitchondrial genome is uninteresting for ATAC-Seq so we remove these reads. We also remove reads with low mapping quality and reads that are not properly paired.


> <hands-on-title>Filtering of uninformative reads</hands-on-title>
>
> 1. {% tool [Filter BAM datasets on a variety of attributes](toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.4.1) %} with the following parameters:
>    - {% icon param-file %} *"BAM dataset(s) to filter"*: Select the output of  **Bowtie2** {% icon tool %} *"alignments"*
>    - In *"Condition"*:
>        - {% icon param-repeat %} *"Insert Condition"*
>            - In *"Filter"*:
>                - {% icon param-repeat %} *"Insert Filter"*
>                    - *"Select BAM property to filter on"*: `mapQuality`
>                        - *"Filter on read mapping quality (phred scale)"*: `>=30`
>                - {% icon param-repeat %} *"Insert Filter"*
>                    - *"Select BAM property to filter on"*: `isProperPair`
>                        - *"Select properly paired reads"*: `Yes`
>                - {% icon param-repeat %} *"Insert Filter"*
>                    - *"Select BAM property to filter on"*: `reference`
>                        - *"Filter on the reference name for the read"*: `!chrM`
>    - *"Would you like to set rules?"*: `No`
>
>
> 2. Click on the input and the output BAM files of the filtering step. Check the size of the files.
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Based on the file size, what proportion of alignments was removed (approximately)?
> 2. Which parameter should be modified if you are interested in repetitive regions?
>
> > <solution-title></solution-title>
> >
> > 1. The original BAM file is 28.1 MB, the filtered one is 15.2 MB. Approximately half of the alignments were removed.
> >
> > 2. You should modify the mapQuality criteria and decrease the threshold.
> >
> {: .solution}
>
{: .question}

High numbers of mitochondrial reads can be a problem in ATAC-Seq. Some ATAC-Seq samples have been reported to be 80% mitochondrial reads and so wet-lab methods have been developed to deal with this issue {% cite Corces2017 %} and {% cite Litzenburger2017 %}. It can be a useful QC to assess the number of mitochondrial reads.
However, it does not predict the quality of the rest of the data. It is just that sequencing reads have been wasted.

> <tip-title>Getting the number of mitochondrial reads</tip-title>
>
> To get the number of reads that mapped to the mitochondrial genome (chrM) you can run {% tool [Samtools idxstats](toolshed.g2.bx.psu.edu/repos/devteam/samtools_idxstats/samtools_idxstats/2.0.3) %} on the output of  **Bowtie2** {% icon tool %} *"alignments"*.
> The columns of the output are: chromosome name, chromosome length, number of reads mapping to the chromosome, number of unaligned mate whose mate is mapping to the chromosome.
> The first 2 lines of the result would be (after using {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/1.1.1) %}):
>
> ![Samtools idxstats result](../../images/atac-seq/Screenshot_samtoolsIdxStatsChrM.png "Samtools idxstats result")
>
> There are 220 000 reads which map to chrM and 165 000 which map to chr22.
{: .tip}

## Filter Duplicate Reads

Because of the PCR amplification, there might be read duplicates (different reads mapping to exactly the same genomic region) from overamplification of some regions. As the Tn5 insertion is random within an accessible region, we do not expect to see fragments with the same coordinates. We consider such fragments to be PCR duplicates. We will remove them with **Picard MarkDuplicates**.

> <hands-on-title>Remove duplicates</hands-on-title>
>
> 1. {% tool [MarkDuplicates](toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_MarkDuplicates/2.18.2.2) %} with the following parameters:
>    - {% icon param-file %} *"Select SAM/BAM dataset or dataset collection"*: Select the output of  **Filter** {% icon tool %} *"BAM"*
>    - *"If true do not write duplicates to the output file instead of writing them with appropriate flags set"*: `Yes`
>
>    > <comment-title>Defaults of MarkDuplicates</comment-title>
>    >
>    > By default, the tool will only "Mark" the duplicates. This means that it will change the Flag of the duplicated reads to enable them to be filtered afterwards. We use the parameter *"If true do not write duplicates to the output file instead of writing them with appropriate flags set"* to directly remove the duplicates.
>    {: .comment}
>
> 2. Click on the {% icon galaxy-eye %} (eye) icon of the MarkDuplicate metrics.
{: .hands_on}

> <comment-title>MarkDuplicates Results</comment-title>
> You should get similar output to this from MarkDuplicates:
> ![Metrics of MarkDuplicates](../../images/atac-seq/Screenshot_picardRemoveDup.png "Metrics of MarkDuplicates")
{: .comment}

> <tip-title>Formatting the MarkDuplicate metrics for readability</tip-title>
>
> 1. {% tool [Select lines that match an expression](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: Select the output of  **MarkDuplicates** {% icon tool %}
>    - *"that*: `Matching`
>    - *"the pattern*: `(Library|LIBRARY)`
>
> 2. Check that the datatype is tabular. If not, change it.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 3. {% tool  [Transpose rows/columns in a tabular file](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.1.0) %}:
>    - {% icon param-file %} *"Select lines from"*: Select the output of **Select** {% icon tool %}
>
> ![Metrics of MarkDuplicates](../../images/atac-seq/Screenshot_picardRemoveDupAfterTranspose.png "Metrics of MarkDuplicates")
>
{: .tip}

> <question-title></question-title>
>
> 1. How many pairs were in the input?
> 2. How many pairs are duplicates?
>
> > <solution-title></solution-title>
> >
> > 1. 135813
> > 2. 3584
> >
> {: .solution}
>
{: .question}

Once again, if you have a high number of replicates it does not mean that your data are not good, it just means that you sequenced too much compared to the diversity of the library you generated. Consequently, libraries with a high portion of duplicates should not be resequenced as this would not increase the amount of data.

## Check Insert Sizes

We will check the insert sizes with **Paired-end histogram** of insert size frequency. The insert size is the distance between the R1 and R2 read pairs. This tells us the size of the DNA fragment the read pairs came from. The fragment length distribution of a sample gives a very good indication of the quality of the ATAC-Seq.

> <hands-on-title>Plot the distribution of fragment sizes.</hands-on-title>
>
> 1. {% tool [Paired-end histogram](toolshed.g2.bx.psu.edu/repos/iuc/pe_histogram/pe_histogram/1.0.1) %} with the following parameters:
>    - {% icon param-file %} *"BAM file"*: Select the output of  **MarkDuplicates** {% icon tool %} *"BAM output"*
>    - *"Lower bp limit (optional)"*: `0`
>    - *"Upper bp limit (optional)"*: `1000`
>
> 2. Click on the {% icon galaxy-eye %} (eye) icon of the lower one of the 2 outputs (the png file).
{: .hands_on}

> <comment-title>CollectInsertSizeMetrics Results</comment-title>
> This is what you get from CollectInsertSizeMetrics:
> ![Fragment size distribution](../../images/atac-seq/Screenshot_sizeDistribution.png "Fragment size distribution")
{: .comment}

> <question-title></question-title>
>
> Could you guess what the peaks at approximately 50bp, 200bp, 400bp and 600bp correspond to?
>
> > <solution-title></solution-title>
> >
> > The first peak (50bp) corresponds to where the Tn5 transposase inserted into nucleosome-free regions. The second peak (a bit less than 200bp) corresponds to where Tn5 inserted around a single nucleosome. The third one (around 400bp) is where Tn5 inserted around two adjacent nucleosomes and the fourth one (around 600bp) is where Tn5 inserted around three adjacent nucleosomes.
> >
> {: .solution}
>
{: .question}

This fragment size distribution is a good indication if your experiment worked or not.
In absence of chromatin (without nucleosome), this is the profile you would get:

![Fragment size distribution of a purified DNA](../../images/atac-seq/Screenshot_sizeDistribution_Naked.png "Fragment size distribution of a purified DNA")

Here are examples of Fragment size distributions of ATAC-Seq which were very noisy:

![Fragment size distribution of a failed ATAC-Seq](../../images/atac-seq/Screenshot_sizeDistribution_Failed.png "Fragment size distribution of a failed ATAC-Seq")

![Fragment size distribution of another failed ATAC-Seq](../../images/atac-seq/Screenshot_sizeDistribution_Failed2.png "Fragment size distribution of another very noisy ATAC-Seq")

A final example of a Fragment size distribution of a very good ATAC-Seq, even if we cannot see the third nucleosome "peak".
![Fragment size distribution of a good ATAC-Seq](../../images/atac-seq/Screenshot_sizeDistribution_Good.png "Fragment size distribution of a good ATAC-Seq")


# Peak calling

## Call Peaks

We have now finished the data preprocessing. Next, in order to find regions corresponding to potential open chromatin regions, we want to identify regions where reads have piled up (peaks) greater than the background read coverage. The tools which are currently used are [Genrich](https://github.com/jsh58/Genrich) and [MACS2](https://github.com/taoliu/MACS). MACS2 is more widely used. Genrich has a mode dedicated to ATAC-Seq but is still not published and the more reads you have, the less peaks you get (see the issue [Genrich#33](https://github.com/jsh58/Genrich/issues/33)). That's why we will not use Genrich in this tutorial.

At this step, two approaches exists:

- The first one is to select only paired whose fragment length is below 100bp corresponding to nucleosome-free regions and to use a peak calling like you would do for a ChIP-seq, joining signal between mates. The disadvantages of this approach is that you can only use it if you have paired-end data and you will miss small open regions where only one Tn5 bound.
- The second one chosen here is to use all reads to be more exhaustive. In this approach, it is very important to re-center the signal of each reads on the 5' extremity (read start site) as this is where Tn5 cuts. Indeed, you want your peaks around the nucleosomes and not directly on the nucleosome:
![Scheme of ATAC-Seq reads relative to nucleosomes](../../images/atac-seq/schemeWithLegend.jpg "Scheme of ATAC-Seq reads relative to nucleosomes")

> <comment-title>on Tn5 insertion</comment-title>
>
> When Tn5 cuts an accessible chromatin locus it inserts adapters separated by 9bp ({% cite Kia2017 %}):
> ![Nextera Library Construction](../../images/atac-seq/NexteraLibraryConstruction.jpg "Nextera Library Construction")
>
> This means in order to have the read start site reflecting the centre of where Tn5 bound, the reads on the positive strand should be shifted 4 bp to the right and reads on the negative strands should be shifted 5 bp to the left as in [Buenrostro et al. 2013](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3959825). **Genrich** can apply these shifts when ATAC-seq mode is selected. In most cases, we do not have 9bp resolution so we don't take it into account but if you are interested in the footprint, this is important.
{: .comment}

If we only assess the coverage of the 5' extremity of the reads, the data would be too sparse and it would be impossible to call peaks. Thus, we will extend the start sites of the reads by 200bp (100bp in each direction) to assess coverage.

### Using MACS2

We convert the BAM file to BED format because when we set the extension size in MACS2, it will only consider one read of the pair while here we would like to use the information from both.

> <hands-on-title>Convert the BAM to BED</hands-on-title>
>
> 1. {% tool [bedtools BAM to BED converter](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_bamtobed/2.30.0) %} with the following parameters:
>    - {% icon param-file %} *"Convert the following BAM file to BED"*: Select the output of **MarkDuplicates** {% icon tool %}
>
{: .hands_on}

We call peaks with MACS2. In order to get the coverage centered on the 5' extended 100bp each side we will use `--shift -100` and `--extend 200`:
![MACS2 options to get 100bp each side](../../images/atac-seq/macs2Options.jpg "MACS2 options to get 100bp each side")


> <hands-on-title>Call peaks with MACS2</hands-on-title>
>
> 1. {% tool [MACS2 callpeak](toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.1.1.20160309.6) %} with the following parameters:
>    - *"Are you pooling Treatment Files?"*: `No`
>        - {% icon param-file %} Select the output of **bedtools BAM to BED** converter {% icon tool %}
>    - *"Do you have a Control File?"*: `No`
>    - *"Format of Input Files"*: `Single-end BED`
>    - *"Effective genome size"*: `H. sapiens (2.7e9)`
>    - *"Build Model"*: `Do not build the shifting model (--nomodel)`
>        - *"Set extension size"*: `200`
>        - *"Set shift size"*: `-100`. It needs to be - half the extension size to be centered on the 5'.
>    - *"Additional Outputs"*:
>        - Check `Peaks as tabular file (compatible with MultiQC)`
>        - Check `Peak summits`
>        - Check `Scores in bedGraph files`
>    - In *"Advanced Options"*:
>        - *"Composite broad regions"*: `No broad regions`
>            - *"Use a more sophisticated signal processing approach to find subpeak summits in each enriched peak region"*: `Yes`
>        - *"How many duplicate tags at the exact same location are allowed?"*: `all`
>
>    > <comment-title>Why keeping all duplicates is important</comment-title>
>    >
>    > We previously removed duplicates using **MarkDuplicates** {% icon tool %} using paired-end information. If two pairs had identical R1 but different R2, we knew it was not a PCR duplicate. Because we converted the BAM to BED we lost the pair information. If we keep the default (removing duplicates) one of the 2 identical R1 would be filtered out as duplicate.
>    {: .comment}
>
{: .hands_on}

# Visualisation of Coverage

## Prepare the Datasets

### Extract CTCF peaks on chr22 in intergenic regions
As our training dataset is focused on chromosome 22 we will only use the CTCF peaks from chr22. We expect to have ATAC-seq coverage at TSS but only good ATAC-seq have coverage on intergenic CTCF. Indeed, the CTCF protein is able to position nucleosomes and creates a region depleted of nucleosome of around 120bp {% cite fu_insulator_2008 %}. This is smaller than the 200bp nucleosome-free region around TSS and also probably not present in all cells. Thus it is more difficult to get enrichment. In order to get the list of intergenic CTCF peaks of chr22, we will first select the peaks on chr22 and then exclude the one which overlap with genes.

> <hands-on-title>Select CTCF peaks from chr22 in intergenic regions:</hands-on-title>
>
> 1. {% tool [Filter data on any column using simple expressions](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: Select the first dataset: `ENCFF933NTR.bed.gz`
>    - *"With following condition"*: `c1=='chr22'`
>
> 1. {% tool [bedtools Intersect intervals find overlapping intervals in various ways](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.30.0) %} with the following parameters:
>    - {% icon param-file %} *"File A to intersect with B"*: Select the output of **Filter** data on any column using simple expressions {% icon tool %}
>    - *"Combined or separate output files"*: `One output file per 'input B' file`
>        - {% icon param-file %} *"File B to intersect with A"*:  Select the dataset `chr22 genes`
>    - *"What should be written to the output file?"*: `Write the original entry in A for each overlap (-wa)`
>    - *"Required overlap"*: `Default: 1bp`
>    - *"Report only those alignments that **do not** overlap with file(s) B"*: `Yes`
>
> 3. Rename the datasets `intergenic CTCF peaks chr22`.
{: .hands_on}


### Convert bedgraph from **MACS2** to bigwig
The bedgraph format is easily readable for human but it can be very large and visualising a specific region is quite slow. We will change it to bigwig format which is a binary format, so we can visualise any region of the genome very quickly.

> <tip-title>Speed-up the bedgraph to bigwig conversion</tip-title>
>
> In this tutorial we focus on chr22, thus we could restrict our bedgraph to chr22 before doing the conversion. This will both speed-up the conversion and decrease the amount of memory needed.
>
> To do so, we will use {% tool [Filter data on any column using simple expressions](Filter1) %} with the following parameters:
> - {% icon param-file %} *"Filter"*: Select the output of **MACS2** {% icon tool %} (Bedgraph Treatment).
> - *"With following condition"*: `c1=='chr22'`
>
> This will decrease by half the size of the file.
> In the next step, choose the output of **Filter** instead of the output of **MACS2**.
>
{: .tip}

> <hands-on-title>Convert bedgraphs to bigWig.</hands-on-title>
>
> 1. {% tool [Wig/BedGraph-to-bigWig](wig_to_bigWig) %} with the following parameters:
>    - {% icon param-file %} *"Convert"*: Select the output of **MACS2** {% icon tool %} (Bedgraph Treatment).
>    - *"Converter settings to use"*: `Default`
>
> 2. Rename the datasets `MACS2 bigwig`.
{: .hands_on}


## Create heatmap of coverage at TSS with deepTools

You might be interested in checking the coverage on specific regions. For this, you can compute a heatmap. We will use the **deepTools plotHeatmap**. As an example, we will here make a heatmap centered on the transcription start sites (TSS) and another one centered on intergenic CTCF peaks. First, on the TSS:

### Generate computeMatrix

The input of **plotHeatmap** is a matrix in a hdf5 format. To generate it we use the tool **computeMatrix** that will evaluate the coverage at each locus we are interested in.

> <hands-on-title>Generate the matrix</hands-on-title>
>
> 1. {% tool [computeMatrix](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.3.2.0.0) %} with the following parameters:
>    - In *"Select regions"*:
>        - {% icon param-repeat %} *"Insert Select regions"*
>            - {% icon param-file %} *"Regions to plot"*: Select the dataset `chr22 genes`
>    - *"Sample order matters"*: `No`
>        - {% icon param-file %} *"Score file"*: Select the output of **Wig/BedGraph-to-bigWig** {% icon tool %} that should be named `MACS2 bigwig`.
>    - *"computeMatrix has two main output options"*: `reference-point`
>    - *"The reference point for the plotting"*: `beginning of region (e.g. TSS)`
>    - *"Show advanced output settings"*: `no`
>    - *"Show advanced options"*: `yes`
>        - *"Convert missing values to 0?"*: `Yes`
>
{: .hands_on}


### Plot with **plotHeatmap**

We will now generate a heatmap. Each line will be a transcript. The coverage will be summarized with a color code from red (no coverage) to blue (maximum coverage). All TSS will be aligned in the middle of the figure and only the 2 kb around the TSS will be displayed. Another plot, on top of the heatmap, will show the mean signal at the TSS. There will be one heatmap per bigwig.

> <hands-on-title>Generate the heatmap</hands-on-title>
>
> 1. {% tool [plotHeatmap](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_heatmap/deeptools_plot_heatmap/3.3.2.0.1) %} with the following parameters:
>    - {% icon param-file %} *"Matrix file from the computeMatrix tool"*: Select the output of **computeMatrix** {% icon tool %}.
>    - *"Show advanced output settings"*: `no`
>    - *"Show advanced options"*: `no`
{: .hands_on}

> <comment-title>plotHeatmap Results</comment-title>
> This is what you get from plotHeatmap:
> ![plotHeatmap output](../../images/atac-seq/plotHeatmapOutput.png "plotHeatmap output")
{: .comment}

> <question-title></question-title>
>
> 1. Is the coverage symmetric?
> 2. What is the mean value in genes?
>
> > <solution-title></solution-title>
> >
> > 1. No, it is higher on the left which is expected as usually the promoter of active genes is accessible.
> > 2. Around 5.5.
> >
> {: .solution}
>
{: .question}

Now we will repeat the procedure for CTCF peaks of chr22 in intergenic regions:

> <hands-on-title>Generate the matrix</hands-on-title>
>
> 1. {% tool [computeMatrix](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.3.2.0.0) %} with the following parameters:
>    - In *"Select regions"*:
>        - {% icon param-repeat %} *"Insert Select regions"*
>            - {% icon param-file %} *"Regions to plot"*: Select the dataset `intergenic CTCF peaks chr22`
>    - *"Sample order matters"*: `No`
>        - {% icon param-file %} *"Score file"*: Select the output of **Wig/BedGraph-to-bigWig** {% icon tool %} that should be named `MACS2 bigwig`.
>    - *"Would you like custom sample labels?"*: `No, use sample names in the history`
>    - *"computeMatrix has two main output options"*: `reference-point`
>        - *"The reference point for the plotting"*: `center of region`
>    - *"Show advanced output settings"*: `no`
>    - *"Show advanced options"*: `yes`
>        - *"Convert missing values to 0?"*: `Yes`
>
> 1. {% tool [plotHeatmap](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_heatmap/deeptools_plot_heatmap/3.3.2.0.1) %} with the following parameters:
>    - {% icon param-file %} *"Matrix file from the computeMatrix tool"*:  Select the output of **computeMatrix** {% icon tool %}.
>    - *"Show advanced output settings"*: `no`
>    - *"Show advanced options"*: `yes`
>        - In *"Colormap to use for each sample"*:
>            - {% icon param-repeat %} *"Insert Colormap to use for each sample"*
>                - *"Color map to use for the heatmap"*: `Blues` # Or what you want
>        - *"The x-axis label"*: `distance from peak center (bp)`
>        - *"The y-axis label for the top panel"*: `CTCF peaks`
>        - *"Reference point label"*: `peak center`
>        - *"Labels for the regions plotted in the heatmap"*: `CTCF_peaks`
>        - *"Did you compute the matrix with more than one groups of regions?"*: `Yes, I used multiple groups of regions`
>
{: .hands_on}

> <comment-title>plotHeatmap Results</comment-title>
> This is what you get from plotHeatmap, this is much more symetric:
> ![plotHeatmap output on CTCF](../../images/atac-seq/plotHeatmapOutput_CTCF.png "plotHeatmap output on CTCF")
{: .comment}


## Visualise Regions with pyGenomeTracks

In order to visualise a specific region (e.g. the gene *RAC2*), we can either use a genome browser like **IGV** or **UCSC browser**, or use **pyGenomeTracks** to make publishable figures. We will use **pyGenomeTracks**.

> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [pyGenomeTracks](toolshed.g2.bx.psu.edu/repos/iuc/pygenometracks/pygenomeTracks/3.6) %} with the following parameters:
>    - *"Region of the genome to limit the operation"*: `chr22:37,193,000-37,252,000`
>    - In *"Include tracks in your plot"*:
>        - {% icon param-repeat %} *"Insert Include tracks in your plot"*
>            - *"Choose style of the track"*: `Bigwig track`
>                - *"Plot title"*: `Coverage from MACS2 (extended +/-100bp)`
>                - {% icon param-file %} *"Track file(s) bigwig format"*: Select the output of **Wig/BedGraph-to-bigWig** {% icon tool %} called `MACS2 bigwig`.
>                - *"Color of track"*: Select the color of your choice
>                - *"Minimum value"*: `0`
>                - *"height"*: `5`
>                - *"Show visualization of data range"*: `Yes`
>        - {% icon param-repeat %} *"Insert Include tracks in your plot"*
>            - *"Choose style of the track"*: `NarrowPeak track`
>                - *"Plot title"*: `Peaks from MACS2 (extended +/-100bp)`
>                - {% icon param-file %} *"Track file(s) encodepeak or bed format"*: Select the output of **MACS2** {% icon tool %} (narrow Peaks).
>                - *"Color of track"*: Select the color of your choice
>                - *"display to use"*: `box: Draw a box`
>                - *"Plot labels (name, p-val, q-val)"*: `No`
>        - {% icon param-repeat %} *"Insert Include tracks in your plot"*
>            - *"Choose style of the track"*: `Gene track / Bed track`
>                - *"Plot title"*: `Genes`
>                - {% icon param-file %} *"Track file(s) bed or gtf format"*: `chr22 genes`
>                - *"Color of track"*: Select the color of your choice
>                - *"height"*: `5`
>                - *"Plot labels"*: `yes`
>                    - *"Put all labels inside the plotted region"*: `Yes`
>                    - *"Allow to put labels in the right margin"*: `Yes`
>        - {% icon param-repeat %} *"Insert Include tracks in your plot"*
>            - *"Choose style of the track"*: `NarrowPeak track`
>                - *"Plot title"*: `CTCF peaks`
>                - {% icon param-file %} *"Track file(s) encodepeak or bed format"*: Select the first dataset: `ENCFF933NTR.bed.gz`
>                - *"Color of track"*: Select the color of your choice
>                - *"display to use"*: `box: Draw a box`
>                - *"Plot labels (name, p-val, q-val)"*: `No`
>        - {% icon param-repeat %} *"Insert Include tracks in your plot"*
>            - *"Choose style of the track"*: `X-axis`
>
> 2. Click on the {% icon galaxy-eye %} (eye) icon of the output.
>
{: .hands_on}

> <comment-title>pyGenomeTracks Results</comment-title>
> You should get similar to results to this from pyGenomeTracks:
> ![pyGenomeTracks output](../../images/atac-seq/pyGenomeTracksOutput.png "pyGenomeTracks output")
{: .comment}

> <question-title></question-title>
> In the ATAC-Seq sample in this selected region we see four peaks detected by MACS2.
>
> 1. How many TSS are accessible in the sample in the displayed region?
> 2. How many CTCF binding loci are accessible?
> 3. Can you spot peaks with no TSS and no CTCF peak?
>
> > <solution-title></solution-title>
> >
> > 1. In total, we can see 3 TSS for 6 transcripts for 2 genes. The TSS of RAC2 corresponds to an ATAC-Seq peak whereas there is no significant coverage on both TSS of SSTR3.
> >
> > 2. Only the first peak on the left overlaps with a CTCF binding site.
> >
> > 3. Amongst the 4 peaks in this region, the 2 in the middle do not correspond to CTCF peaks or TSS.
> >
> {: .solution}
>
{: .question}

As CTCF creates accessible regions, a region containing a peak with no corresponding CTCF peak or TSS could be a putative enhancer. In the pyGenomeTracks plot we see a region like this located in the intron of a gene and another one between genes. However, it is impossible to guess from the position which would be the gene controlled by this region. And of course, more analyses are needed to assess if it is a real enhancer, for example, histone ChIP-seq, 3D structure, transgenic assay, etc.


# Conclusion

In this training you have learned the general principles of ATAC-Seq data analysis. ATAC-Seq
is a method to investigate the chromatin accessibility and the genome is treated with
a transposase (enzyme) called Tn5. It marks open chromatin regions by cutting and
inserting adapters for sequencing. The training material gave you an insight into how to quality control the data. You should look for low quality bases, adapter contamination, correct insert size and PCR duplicates (duplication level). We showed you how to remove adapters and PCR duplicates, if **FastQC**, shows a warning in these areas. We mapped the reads
with **Bowtie2**, filtered our reads for properly paired, good quality and reads that do not
map to the mitochondrial genome. We found open chromatin regions with **MACS2**, a tool to find regions of genomic enrichment (peaks). We investigated the read coverage around TSS with the help of **computeMatrix** and **plotHeatmap**. Last but not least, we visualised the peaks and other informative tracks, such as CTCF binding regions and hg38 genes, with the help of **pyGenomeTracks**. At the end, we found open chromatin regions that did not overlap with CTCF sites or TSS, which could be potential putative enhancer regions detected by the ATAC-Seq experiment.

![ATAC workflow](../../images/atac-seq/ATACWF.svg "ATAC workflow")

tools_json:
["Cut columns from a table", "Cut1", "Flatten collection", "__FLATTEN__", "FastQC", "toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy1", "Cutadapt", "toolshed.g2.bx.psu.edu/repos/lparsons/cutadapt/cutadapt/5.1+galaxy0", "Bowtie2", "toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.4+galaxy0", "Filter BAM datasets on a variety of attributes", "toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.4.1", "Samtools idxstats", "toolshed.g2.bx.psu.edu/repos/devteam/samtools_idxstats/samtools_idxstats/2.0.3", "Sort", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/1.1.1", "MarkDuplicates", "toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_MarkDuplicates/2.18.2.2", "Select lines that match an expression", "Grep1", "Transpose rows/columns in a tabular file", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.1.0", "Paired-end histogram", "toolshed.g2.bx.psu.edu/repos/iuc/pe_histogram/pe_histogram/1.0.1", "bedtools BAM to BED converter", "toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_bamtobed/2.30.0", "MACS2 callpeak", "toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.1.1.20160309.6", "Filter data on any column using simple expressions", "Filter1", "bedtools Intersect intervals find overlapping intervals in various ways", "toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.30.0", "Wig/BedGraph-to-bigWig", "wig_to_bigWig", "computeMatrix", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.3.2.0.0", "plotHeatmap", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_heatmap/deeptools_plot_heatmap/3.3.2.0.1", "pyGenomeTracks", "toolshed.g2.bx.psu.edu/repos/iuc/pygenometracks/pygenomeTracks/3.6"]


topic: epigenetics
tutorial: hicexplorer
title: Hi-C analysis of Drosophila melanogaster cells using HiCExplorer
difficulty: intermediate
time_estimation: 2h
requirements_json: []
tags_json: []
content_hash: 4b176b7b0c649777a04ee0f429be8796
last_modified: 2026-05-27T09:01:48.466114
zenodo_link: https://doi.org/10.5281/zenodo.16416373
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/epigenetics/tutorials/hicexplorer/tutorial.html

content:
In this HiCExplorer tutorial we will generate and plot a Hi-C contact matrix.
For this the following steps are necessary to be performed:

1. Map the Hi-C reads to the reference genome
2. Creation of a Hi-C matrix
3. Plotting the Hi-C matrix
4. Correction of Hi-C matrix
5. TAD Calling
6. A/B compartments computation
7. pyGenomeTracks visualization
8. Loop detection

After a corrected Hi-C matrix is created other tools can be used to visualise it, call TADS or compare it with other matrices.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data upload

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import from [Zenodo](https://doi.org/10.5281/zenodo.16416373).
>
>    ```
>    https://zenodo.org/records/16416373/files/HiC_S2_1p_10min_lowU_R1.fastq.gz
>    https://zenodo.org/records/16416373/files/HiC_S2_1p_10min_lowU_R2.fastq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. **Rename** {% icon galaxy-pencil %} the data set to something meaningful
>    - e.g. `HiC_S2_1p_10min_lowU_R1` and `HiC_S2_1p_10min_lowU_R2`.
>    - By default, when data is imported via its link, Galaxy names it with its URL.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> > <comment-title>Get data from public sources</comment-title>
> > HiCExplorer needs as input the forward and reverse strand of a pair end read which are mapped independently. A usual start point for a typical analysis is the given GSE number of a publication, e.g. GSE63525 for Rao 2014. To get the actual data, go to [NCBI](https://www.ncbi.nlm.nih.gov/geo/) and search for the [GSE number](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE63525). In the section 'Samples' the GSM numbers of all samples are given. Select the correct one for you, and go to the [European Nucleotide Archive](https://www.ebi.ac.uk/ena) and enter the GSM number. Select a matching result e.g. [SRX764936](https://www.ebi.ac.uk/ena/data/view/SRX764936) and download the data given in the row 'FASTQ files (FTP)' the forward and reverse strand.
> > It is important to have the forward and reverse strand individual as a FASTQ file and to map it individually, HiCExplorer can not work with interleaved files.
> {: .comment}
>
{: .hands_on}

# Reads mapping

Mates have to be mapped individually to avoid mapper specific heuristics designed for standard paired-end libraries.

We have used the HiCExplorer successfully with `bwa`, `bowtie2` and `hisat2`. In this tutorial we will be using `Bowtie2` tool. It is important to remember to:
- use local mapping, in contrast to end-to-end. A fraction of Hi-C reads are chimeric and will not map end-to-end thus, local mapping is important to increase the number of mapped reads
- tune the aligner parameters to penalize deletions and insertions. This is important to avoid aligned reads with gaps if they happen to be chimeric.
- If bowtie2 or hisat2 are used, `--reorder` option and as a file format `bam_native` needs to be used. Regular `bam` files are sorted by Galaxy and can not be used as an input for HiCExplorer.

> <hands-on-title>Mapping reads</hands-on-title>
>
> 1. {% tool [`Bowtie2`](toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.3+galaxy1) %}: Run `Bowtie2` on both strands `HiC_S2_1p_10min_lowU_R1` and `HiC_S2_1p_10min_lowU_R2` with:
>    - *"Is this single or paired library"*: `Single-end`
>    - *Set multiple data sets*
>    - *"FASTQ file"*: `HiC_S2_1p_10min_lowU_R1`and `HiC_S2_1p_10min_lowU_R2`
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in index`
>    - *"Select a reference genome"*: `dm3`
>    - *"Do you want to tweak SAM/BAM Options?"*: `Yes`
>       - *"Reorder output to reflect order of the input file"*: `Yes`
>
> 2. **Rename** {% icon galaxy-pencil %} the output of the tool according to the corresponding files: `R1.bam` and `R2.bam`
>
{: .hands_on}

# Creating a Hi-C matrix

Once the reads have been mapped the Hi-C matrix can be built.

For this step we will use [`hicBuildMatrix`](http://hicexplorer.readthedocs.io/en/latest/content/tools/hicBuildMatrix.html#hicbuildmatrix) tool, which builds the matrix of read counts over the bins in the genome, considering the sites around the given restriction site.

For versions `3.6` and later, `hicBuildMatrix` requires an input BED file that specifies the locations of all restriction cuts. We must first identify restriction enzyme sites use the tool `hicFindRestSite`, which requires a `fasta` file containing the genome of organism we are studying. We will import the `dm3_genome.fasta` genome file from Zenodo.

> <hands-on-title>Import from Zenodo</hands-on-title>
>
> 1. Import the reference genome
>    ```
>    https://zenodo.org/records/16416373/files/dm3_genome.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}


> <hands-on-title>Find Restriction Sites and Build Matrix</hands-on-title>
>
> 1. {% tool [`hicFindRestSite`](toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicfindrestrictionsites/hicexplorer_hicfindrestrictionsites/3.7.6+galaxy1) %}:
>    - *"Fasta file for the organism genome."*: `dm3_genome.fasta`
>    - *"Restriction enzyme sequence"*: `GATC`
>
>       > <comment-title></comment-title>
>       >
>       > Our data uses the DpnII restriction enzyme, which has the *Restriction enzyme sequence*: `GATC`. The *Restriction enzyme sequence* will be specific to the restriction enzyme that your data uses (e.g. for HindIII this will be `AAGCTT`).
>       {: .comment}
>
> 2. **Rename** {% icon galaxy-pencil %} the output to `Restriction Sites`.
>
> 3. {% tool [`hicBuildMatrix`](toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicbuildmatrix/hicexplorer_hicbuildmatrix/3.7.6+galaxy1) %}: Run `hicBuildMatrix` on the `R1.bam` and `R2.bam` mapping output.
>    - *"1: Sam/Bam files to process (forward/reverse)"*: `R1.bam`
>    - *"2: Sam/Bam files to process (forward/reverse)"*: `R2.bam`
>    - *"BED file with all restriction cut places"*: `Restriction sites`
>    - *"Sequence of the restriction site"*: `GATC`
>    - *"Dangling sequence"*: `GATC`
>    - *"Bin size in bp"*: `10000`
>
>       > <comment-title></comment-title>
>       >
>       > *hicBuildMatrix* creates two files, a bam file containing only the valid Hi-C read pairs and a matrix containing the Hi-C contacts at the given resolution. The bam file is useful to check the quality of the Hi-C library on the genome browser. A good Hi-C library should contain piles of reads near the restriction fragment sites. In the QC folder a html file is saved with plots containing useful information for the quality control of the Hi-C sample like the number of valid pairs, duplicated pairs, self-ligations etc. Usually, only 25%-40% of the reads are valid and used to build the Hi-C matrix mostly because of the reads that are on repetitive regions that need to be discarded.
>       {: .comment}
>
>       > <comment-title></comment-title>
>       >
>       > Normally 25% of the total reads are selected. The output matrices have counts for the genomic regions. The extension of output matrix files is .h5.
>       > A quality report is created in e.g. `hicMatrix/R1_10kb_QC`, have a look at the report hicQC.html.
>       {: .comment}
>
> 4. **Rename** {% icon galaxy-pencil %} the output to `10kb contact matrix`.
>
{: .hands_on}


> <comment-title></comment-title>
> If you do not have access to the genome for your organism (e.g. if you plan to use your Hi-C data to assist with genome assembly) then you can instead select `hicBuildMatrix` version 3.4.3.0.
>
>   > <hands-on-title>Build Hi-C Matrix</hands-on-title>
>   >
>   > 1. {% tool [`hicBuildMatrix`](toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicbuildmatrix/hicexplorer_hicbuildmatrix/3.4.3.0) %}: Run `hicBuildMatrix` on the `R1.bam` and `R2.bam` mapping output.
>   >    - *"1: Sam/Bam files to process"*: `R1.bam`
>   >    - *"2: Sam/Bam files to process"*: `R2.bam`
>   >    - *"Choose to use a restriction cut file or a bin size"*: `Bin size`
>   >    - *"Bin size in bp"*: `10000`
>   >    - *"Sequence of the restriction site"*: `GATC`
>   {: .hands_on}
>
{: .comment}

# Plotting the Hi-C matrix

A 10 kb bin matrix is too large to plot, it's better to reduce the resolution. We usually run out of memory for a 1 kb or a 10 kb bin matrix and the time to plot it is very long (minutes instead of seconds). In order to reduce the resolution we use the tool [`hicMergeMatrixBins`](http://hicexplorer.readthedocs.io/en/latest/content/tools/hicMergeMatrixBins.html#hicmergematrixbins).

[`hicMergeMatrixBins`](http://hicexplorer.readthedocs.io/en/latest/content/tools/hicMergeMatrixBins.html#hicmergematrixbins) merges the bins into larger bins of given number (specified by –numBins). We will merge 100 bins in the original (uncorrected) matrix and then correct it. The new bin size is going to be 10.000 bp * 100 = 1.000.000 bp = 1 Mb

> <hands-on-title>Merge Matrix Bins</hands-on-title>
>
> 1. {% tool [`hicMergeMatrixBins`](toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicmergematrixbins/hicexplorer_hicmergematrixbins/3.7.6+galaxy1) %}:
>    - *"Matrix to compute on"*: `10kb contact matrix`
>    - *"Number of bins to merge"*: `100`
>
> 2. **Rename** {% icon galaxy-pencil %} the output to `1MB contact matrix`.
>
> 3. {% tool [`hicPlotMatrix`](toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicplotmatrix/hicexplorer_hicplotmatrix/3.4.3.0) %}:
>    - *"Matrix to compute on"*: `1MB contact matrix`
>    - *"Plot title"*: `Hi-C matrix for dm3`
>    - *"Remove masked bins from the matrix"*: `Yes`
>    - *"Plot the log1p of the matrix values"*: `Yes`
>    - *"Chromosomes to include (and order to plot in)"*: `chr2L`
>    - *"+ Insert Chromosomes to include (and order to plot in):"*: `chr2R`
>    - *"+ Insert Chromosomes to include (and order to plot in):"*: `chr3L`
>    - *"+ Insert Chromosomes to include (and order to plot in):"*: `chr3R`
>    - *"+ Insert Chromosomes to include (and order to plot in):"*: `chrX`
>
>    > <tip-title>log1p</tip-title>
>    >
>    > Because of the large differences in counts found in the matrix, it is better to plot the counts using the `–log1p` option.
>    {: .tip}
>
{: .hands_on}

The resulting plot of the 1 Mb contact matrix should look like:
![TAD plot](../../images/plotMatrix.png)

# Correcting the Hi-C matrix

[`hicCorrectMatrix`](http://hicexplorer.readthedocs.io/en/latest/content/tools/hicCorrectMatrix.html#hiccorrectmatrix) corrects the matrix counts in an iterative manner. For correcting the matrix, it’s important to remove the unassembled scaffolds (e.g. NT_) and keep only chromosomes, as scaffolds create problems with matrix correction. Therefore we use the chromosome names (chr2R, chr2L, chr3R, chr3L, chrX) here.

Matrix correction works in two steps: first a histogram containing the sum of contact per bin (row sum) is produced. This plot needs to be inspected to decide the best threshold for removing bins with lower number of reads. The second steps removes the low scoring bins and does the correction.

> <hands-on-title>Matrix diagnostic</hands-on-title>
>
> 1. {% tool [`hicCorrectMatrix`](toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hiccorrectmatrix/hicexplorer_hiccorrectmatrix/3.7.6+galaxy1) %}:
>    - *"Matrix to compute on"*: `10kb contact matrix`
>    - *"Range restriction (in bp)"*: `Diagnostic plot`
>    - *"Chromosomes to include (and order to plot in)"*: `chr2L`
>    - *"+ Insert Chromosomes to include (and order to plot in):"*: `chr2R`
>    - *"+ Insert Chromosomes to include (and order to plot in):"*: `chr3L`
>    - *"+ Insert Chromosomes to include (and order to plot in):"*: `chr3R`
>    - *"+ Insert Chromosomes to include (and order to plot in):"*: `chrX`
>
{: .hands_on}

The output of the program prints a threshold suggestion that is usually accurate but is better to revise the histogram plot. The threshold is visualised in the plot as a blue vertical line.

![Diagnostic plot](../../images/hic_diagnostic_plot.png)

In our case the distribution describes the counts per bin of a genomic distance. To remove all bins with a z-score threshold less / more than X means to remove all bins which have less / more counts than X of mean of their specific distribution in units of the standard deviation. Looking at the distribution, we can select the value of -1.6 (lower end) and 1.8 (upper end) to remove. This is given by the –filterThreshold option in hicCorrectMatrix set to 'correct matrix' mode.

> <hands-on-title>Matrix correction</hands-on-title>
>
> 1. {% tool [`hicCorrectMatrix`](toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hiccorrectmatrix/hicexplorer_hiccorrectmatrix/3.7.6+galaxy1) %}:
>    - *"Matrix to compute on"*: `10kb contact matrix`
>    - *"Range restriction (in bp)"*: `Correct matrix`
>    - *"Normalize each chromosome separately"*: `Yes`
>    - *"Remove bins of low coverage"*: `-1.6`
>    - *"Remove bins of large coverage"*: `1.8`
>    - *"Chromosomes to include (and order to plot in)"*: `chr2L`
>    - *"+ Insert Chromosomes to include (and order to plot in):"*: `chr2R`
>    - *"+ Insert Chromosomes to include (and order to plot in):"*: `chr3L`
>    - *"+ Insert Chromosomes to include (and order to plot in):"*: `chr3R`
>    - *"+ Insert Chromosomes to include (and order to plot in):"*: `chrX`
>
> 2. **Rename** {% icon galaxy-pencil %} the corrected matrix to `10kb corrected contact matrix`.
>
{: .hands_on}

It can happen that the correction stops with:

```
ERROR:iterative correction:*Error* matrix correction produced extremely large values.
This is often caused by bins of low counts. Use a more stringent filtering of bins.
```

This can be solved by a more stringent z-score values for the filter threshold or by a look at the plotted matrix. For example, chromosomes with 0 reads in its bins can be excluded from the correction by not defining it for the set of chromosomes that should be corrected (parameter 'Include chromosomes').

## Plotting the corrected Hi-C matrix

We can now plot chromosome 2L with the corrected matrix.

> <hands-on-title>Plotting the corrected Hi-C matrix</hands-on-title>
>
> 1. {% tool [`hicPlotMatrix`](toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicplotmatrix/hicexplorer_hicplotmatrix/3.4.3.0) %}:
>    - *"Matrix to compute on"*: `10kb corrected contact matrix`
>    - *"Plot title"*: `Hi-C matrix for dm3`
>    - *"Plot per chromosome"*: `No`
>    - *"Plot only this region"*: `chr2L`
>    - *"Plot the log1p of the matrix values"*: `True`
>
{: .hands_on}


## Load new data

The steps so far would have led to long run times if real data would have been used. We therefore prepared a new matrix for you, `corrected contact matrix dm3 large`. Please import it into your history from Zenodo.


> <hands-on-title>Import from Zenodo</hands-on-title>
>
> 1. Import the following file into your history:
>    ```
>    https://zenodo.org/records/16416373/files/corrected_contact_matrix_dm3_large.h5
>    ```
>
{: .hands_on}

# TAD calling

“The partitioning of chromosomes into topologically associating domains (TADs) is an emerging concept that is reshaping our understanding of gene regulation in the context of physical organization of the genome” ({% cite Ramrez2017 %}).

TAD calling works in two steps: First HiCExplorer computes a TAD-separation score based on a z-score matrix for all bins. Then those bins having a local minimum of the TAD-separation score are evaluated with respect to the surrounding bins to assign a p-value. Then a cutoff is applied to select the bins more likely to be TAD boundaries.

[hicFindTADs](http://hicexplorer.readthedocs.io/en/latest/content/tools/hicFindTADs.html#hicfindtads) tries to identify sensible parameters but those can be change to identify more stringent set of boundaries.

> <hands-on-title>Finding TADs</hands-on-title>
>
> 1. {% tool [`hicFindTADs`](toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicfindtads/hicexplorer_hicfindtads/3.7.6+galaxy1) %}:
>    - *"Matrix to compute on"*: `corrected_contact_matrix_dm3_large.h5`
>    - *"Minimum window length (in bp) to be considered to the left and to the right of each Hi-C bin."*: `30000`
>    - *"Maximum window length (in bp) to be considered to the left and to the right of each Hi-C bin."*: `100000`
>    - *"Step size when moving from minDepth to maxDepth"*: `10000`
>    - *"Multiple Testing Corrections"*: `False discovery rate`
>    - *"q-value"*: `0.05`
>    - *"Minimum threshold of the difference between the TAD-separation score of a putative boundary and the mean of the TAD-sep. score of surrounding bins."*: `0.001`
>
> 2. **Rename** {% icon galaxy-pencil %} the TAD boundary positions to `Boundary positions`.
> 3. **Rename** {% icon galaxy-pencil %} the multi-scale TAD scores matrix to `Matrix with multi-scale TAD scores`.
> 4. **Rename** {% icon galaxy-pencil %} the TAD domains to `TAD domains`.
> 5. **Rename** {% icon galaxy-pencil %} the boundary information to `Boundary information plus score`.
> 6. **Rename** {% icon galaxy-pencil %} the TAD information in bm file to `TAD information in bm file`.
>
{: .hands_on}

As an output we get the boundaries, domains and scores separated files. We will use in the plot later only the TAD-score file.

# A/B compartments computation

> <hands-on-title>Computing A / B compartments</hands-on-title>
>
> 1. {% tool [`hicPCA`](toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicpca/hicexplorer_hicpca/3.7.6+galaxy1) %}:
>    - *"Matrix to compute on"*: `corrected_contact_matrix_dm3_large.h5 `
>    - *"Output file format"*: `bigwig`
>    - *"Return internally used Pearson matrix"*: `Yes`
>
> 2. **Rename** {% icon galaxy-pencil %} the Pearson matrix to `Pearson matrix`.
>
{: .hands_on}

> <hands-on-title>Plotting the pearson matrix and PCA track</hands-on-title>
>
> 1. {% tool [`hicPlotMatrix`](toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicplotmatrix/hicexplorer_hicplotmatrix/3.4.3.0) %}:
>    - *"Matrix to compute on"*: `Pearson matrix`
>    - *"Plot title"*: `Pearson matrix and PC1`
>    - *"Chromosomes to include"*: `chr2L`
>    - *"Color map to use for the heatmap"*: `gist_heat`
>    - *"Plot the log1p of the matrix values (log(Hi-C contacts+1))"*: `Yes`
>    - *"Datatype of eigenvector file"*: `bigwig`
>    - *"Eigenvector file"*: `pca1`
>
> 2. {% tool [`hicPlotMatrix`](toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicplotmatrix/hicexplorer_hicplotmatrix/3.4.3.0) %}:
>    - *"Matrix to compute on"*: `Pearson matrix`
>    - *"Plot title"*: `Pearson matrix and PC2`
>    - *"Chromosomes to include"*: `chr2L`
>    - *"Color map to use for the heatmap"*: `gist_heat`
>    - *"Plot the log1p of the matrix values (log(Hi-C contacts+1))"*: `Yes`
>    - *"Datatype of eigenvector file"*: `bigwig`
>    - *"Eigenvector file"*: `pca2`
>
{: .hands_on}


![Pearson PC1](../../images/pearson_pc1.png)


![Pearson PC2](../../images/pearson_pc2.png)

The first principal component correlates with the chromosome arms, while the second component correlates with A/B compartments.

# Integrating Hi-C and other data

We can plot the TADs for a given chromosomal region. For this we will use [`pyGenomeTracks`](http://hicexplorer.readthedocs.io/en/latest/content/tools/hicPlotTADs.html). For the next step we need additional data tracks. Please import `dm3_genes.bed`, `H3K27me3.bw`, `H3K36me3.bw` and `H4K16ac.bw` to your history from Zenodo.

> <hands-on-title>Import from Zenodo</hands-on-title>
>
> 1. Import the following files into Galaxy:
>    ```
>    https://zenodo.org/records/16416373/files/dm3_genes.bed
>    https://zenodo.org/records/16416373/files/H3K27me3.bw
>    https://zenodo.org/records/16416373/files/H3K36me3.bw
>    https://zenodo.org/records/16416373/files/H4K16ac.bw
>    ```
>
{: .hands_on}

> <hands-on-title>Update `dm3_genes.bed` database</hands-on-title>
>
> If the database for `dm3_genes.bed` is not "dm3" (i.e. `database "?"` or a different database is listed), then the database must be updated to "dm3".
>
> {% snippet faqs/galaxy/datasets_change_dbkey.md %}
>
{: .hands_on}


> <hands-on-title>Plotting TADs</hands-on-title>
>
> 1. {% tool [`pyGenomeTracks `](toolshed.g2.bx.psu.edu/repos/iuc/pygenometracks/pygenomeTracks/3.8+galaxy2) %}:
>    - *"Region of the genome to plot"*: `chr2L:14500000-16500000`
>    - *"Choose style of the track"*: `TAD visualization (triangle)`
>         - *"Plot title"*: `HiC dm3 chr2L:14500000-16500000`
>         - *"Matricies to plot"*: `corrected_contact_matrix_dm3_large.h5`
>         - *"Depth"*: `750000`
>         - *"Height"*: `4`
>         - *"Boundaries file"*: `TAD domains`
>
>    - *"+Insert Include tracks in your plot"*
>        - *"Choose style of the track"*: `Bedgraph matrix track/TAD score`
>        - *"Plot title"*: `TAD separation score`
>        - *"Track file(s) bedgraph format"*: `TAD information in bm file`
>        - *"Height"*: `4`
>        - *"type of plotting"*: `lines: each column in the bedgraph will be a line and a mean line will be added`
>
>    - *"+Insert Include tracks in your plot"*
>        - *"Choose style of the track"*: `Bigwig track`
>        - *"Plot title"*: `PC1`
>        - *"Track file bigwig format"*: `pca1`
>        - *"Height"*: `1.5`
>        - *"Color of track"* to a color of your choice
>
>    - *"+Insert Include tracks in your plot"*
>        - *"Choose style of the track"*: `Bigwig track`
>        - *"Plot title"*: `PC2`
>        - *"Track file bigwig format"*: `pca2`
>        - *"Height"*: `1.5`
>        - *"Color of track"* to a color of your choice
>
>    - *"+Insert Include tracks in your plot"*
>        - *"Choose style of the track"*: `Bigwig track`
>        - *"Plot title"*: `H3K36me3`
>        - *"Track file(s) bigwig format"*: `H3K36me3.bw`
>        - *"Height"*: `1.5`
>        - *"Color of track"* to a color of your choice
>
>    - *"+Insert Include tracks in your plot"*
>        - *"Choose style of the track"*: `Bigwig track`
>        - *"Plot title"*: `H3K27me3`
>        - *"Track file(s) bigwig format"*: `H3K27me3.bw`
>        - *"Height"*: `1.5`
>        - *"Color of track"* to a color of your choice
>
>    - *"+Insert Include tracks in your plot"*
>        - *"Choose style of the track"*: `Bigwig track`
>        - *"Plot title"*: `H4K16ac`
>        - *"Track file(s) bigwig format"*: `H4K16ac.bw`
>        - *"Height"*: `1.5`
>        - *"Color of track"* to a color of your choice
>
>    - *"+Insert Include tracks in your plot"*
>        - *"Choose style of the track"*: `Gene track / Bed track`
>        - *"Plot title"*: `dm3 genes`
>        - *"Track file(s) bed or gtf format"*: `dm3_genes.bed`
>        - *"Height"*: `3`
>        - *"Configure other bed parameters"*:
>        	- *"Maximum number of gene rows"*: `15`
>        - *"Color of track"* to a color of your choice
>
>    - *"+Insert Include tracks in your plot"*
>        - *"Choose style of the track"*: `Vlines track`
>        - *"Track file bed format"*: `TAD domains`
>
{: .hands_on}

The resulting image should look like this one:
![TAD plot](../../images/plotTADs.png)

# Loop detection

In Hi-C data, the term `loop` refers to a 3D structure which represents enhancer-promoter, gene, architectural or polycomb-mediated interactions. These interactions have the characteristics to be enriched in a single region compared to the local background. These loops are also called long-range interactions with an expected maximum distance of 2 MB (see {% cite Rao2014 %}).

![Loops visualization](../../images/loops_bonev_cavalli.png)


To compute loops, we will use a published Hi-C sequencing dataset from the human cell line GM12878, mapped to hg19 and of 25 kb resolution ({% cite wolff2025 %}). The original Hi-C data is from {% cite Rao2014 %} (GSE63525). We can import this dataset from Zenodo.

We use a new file because to detect loop structures the read coverage is required to be in the millions per chromosome; this was not the case for the previous used drosophila dataset.


> <hands-on-title>Import via URL</hands-on-title>
>
> 1. Import the following file to Galaxy
>    ```
>    https://zenodo.org/records/16416373/files/GM12878_25kb_cooler_coarsen.cool
>    ```
{: .hands_on}



> <hands-on-title>Matrix information</hands-on-title>
>
> 1. {% tool [`hicInfo`](toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicinfo/hicexplorer_hicinfo/3.7.6+galaxy1) %}:
>    - *"Select"* `Multiple datasets`
>    - *"Matrix to compute on"*: `corrected_contact_matrix_dm3_large.h5 ` and `GM12878_25kb_cooler_coarsen.cool`
>
{: .hands_on}

We can view the result of `hicInfo` and see that the new imported file has 900 million non-zero elements, while the drosophila Hi-C interaction matrix has around 12 million non-zero elements.

We next run `hicDetectLoops` to find loops. In this tutorial we will only look for loops in chromosome 1 to reduce the required computational resources and computing time.

> <hands-on-title>Computing loops</hands-on-title>
>
> 1. {% tool [`hicDetectLoops`](toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicdetectloops/hicexplorer_hicdetectloops/3.7.6+galaxy1) %}:
>    - *"Matrix to compute on"*: `GM12878_25kb_cooler_coarsen.cool`
>    - *"Peak width"*: `6`
>    - *"Window size"*: `10`
>    - *"P-value preselection"*: `0.02`
>    - *"P-value"*: `0.02`
>    - *"Chromosomes to include"*: `1`
>
{: .hands_on}


The detection of the loops is based on a pre-selection of interactions, a p-value given a continuous negative binomial distribution over all interactions of a relative distance is computed. All interactions are filtered with a threshold (`p-value preselection`) to retrieve loop candidates. In a second step, the selected peak candidate is compared against its background using a Wilcoxon rank-sum test.

As an output we get a loop file containing the positions of both anchor points of the loop and the p-value of the used statistical test.

![Loops computed_file](../../images/loops_25kb_res_result.png)

> <hands-on-title>Plotting detected loops</hands-on-title>
>
> 1. {% tool [`hicPlotMatrix`](toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicplotmatrix/hicexplorer_hicplotmatrix/3.4.3.0) %}:
>    - *"Matrix to compute on"*: `GM12878_25kb_cooler_coarsen.cool`
>    - *"Plot title"*: `Loops`
>    - *"Plot only this region"*: `1:18000000-22000000`
>    - *"Plot the log1p of the matrix values"*: `Yes`
>    - *"Plot Loops"*: `Yes`
>       - *"Add detected loops"*: `Computed loops`
>    - *"DPI for image"*: `300`
>
{: .hands_on}

![Loops result_plot](../../images/loops_25kb_res_plot.png)

# Conclusion

In this tutorial we used HiCExplorer to analyze drosophila melanogaster cells. We mapped chimeric reads and created a contact matrix, to reduce noise this contact matrix was normalized. We showed how to visualise a contact matrix and how we can investigate topological associating domains and relate them to additional data like gene tracks. Moreover, we used a human Hi-C interaction matrix to compute loop structures.


To improve your learned skills, there is an additional [`hicexplorer` tutorial](http://hicexplorer.readthedocs.io/en/latest/content/example_usage.html) based on mouse stem cells.

tools_json:
["`Bowtie2`", "toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.3+galaxy1", "`hicFindRestSite`", "toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicfindrestrictionsites/hicexplorer_hicfindrestrictionsites/3.7.6+galaxy1", "`hicBuildMatrix`", "toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicbuildmatrix/hicexplorer_hicbuildmatrix/3.7.6+galaxy1", "toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicbuildmatrix/hicexplorer_hicbuildmatrix/3.4.3.0", "`hicMergeMatrixBins`", "toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicmergematrixbins/hicexplorer_hicmergematrixbins/3.7.6+galaxy1", "`hicPlotMatrix`", "toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicplotmatrix/hicexplorer_hicplotmatrix/3.4.3.0", "`hicCorrectMatrix`", "toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hiccorrectmatrix/hicexplorer_hiccorrectmatrix/3.7.6+galaxy1", "`hicFindTADs`", "toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicfindtads/hicexplorer_hicfindtads/3.7.6+galaxy1", "`hicPCA`", "toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicpca/hicexplorer_hicpca/3.7.6+galaxy1", "`pyGenomeTracks `", "toolshed.g2.bx.psu.edu/repos/iuc/pygenometracks/pygenomeTracks/3.8+galaxy2", "`hicInfo`", "toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicinfo/hicexplorer_hicinfo/3.7.6+galaxy1", "`hicDetectLoops`", "toolshed.g2.bx.psu.edu/repos/bgruening/hicexplorer_hicdetectloops/hicexplorer_hicdetectloops/3.7.6+galaxy1"]


topic: epigenetics
tutorial: tal1-binding-site-identification
title: Identification of the binding sites of the T-cell acute lymphocytic leukemia protein 1 (TAL1)
difficulty: intermediate
time_estimation: 3h
requirements_json: []
tags_json: ["ChIP-seq"]
content_hash: 2ad9f6e741ba47290b8d419a04973d64
last_modified: 2026-05-27T09:01:48.466787
zenodo_link: https://doi.org/10.5281/zenodo.197100
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/epigenetics/tutorials/tal1-binding-site-identification/tutorial.html

content:
This tutorial uses ChIP-seq datasets from a study published by {% cite Wu_2014 %}. The goal of this study was to investigate "the dynamics of occupancy and the role in gene regulation of the transcription factor TAL1, a critical regulator of hematopoiesis, at multiple stages of hematopoietic differentiation."

To this end, ChIP-seq experiments were performed in multiple mouse cell types including G1E - a GATA-null immortalized cell line derived from targeted disruption of GATA-1 in mouse embryonic stem cells - and megakaryocytes.

This dataset (GEO Accession: [GSE51338](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE51338)) consists of biological replicate TAL1 ChIP-seq and input control experiments.
Input control experiments are used to identify and remove sampling bias, for example open/accessible chromatin or GC bias.

Because of the long processing time for the large original files, we have downsampled the original raw data files to include only reads that align to chromosome 19 and a subset of interesting genomic loci identified by {% cite Wu_2014 %}.

**Table 1**: Metadata for ChIP-seq experiments in this tutorial. SE: single-end.

| Cellular state | Datatype | ChIP Ab | Replicate | SRA Accession | Library type | Read length | Stranded? | Data size (MB) |
| ---            | ---      | :-:     | :-:       | ---           | :-:          | :-:         | :-:       | ---            |
| G1E            | ChIP-seq | input   | 1         | SRR507859     | SE           | 36          | No        | 35.8           |
| G1E            | ChIP-seq | input   | 2         | SRR507860     | SE           | 55          | No        | 427.1          |
| G1E            | ChIP-seq | TAL1    | 1         | SRR492444     | SE           | 36          | No        | 32.3           |
| G1E            | ChIP-seq | TAL1    | 2         | SRR492445     | SE           | 41          | No        | 62.7           |
| Megakaryocyte  | ChIP-seq | input   | 1         | SRR492453     | SE           | 41          | No        | 57.2           |
| Megakaryocyte  | ChIP-seq | input   | 2         | SRR492454     | SE           | 55          | No        | 403.8          |
| Megakaryocyte  | ChIP-seq | TAL1    | 1         | SRR549006     | SE           | 55          | No        | 340.3          |
| Megakaryocyte  | ChIP-seq | TAL1    | 2         | SRR549007     | SE           | 48          | No        | 356.9          |

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Quality control

As for any NGS data analysis, ChIP-seq data must be quality controlled before being aligned to a reference genome. For more detailed information on NGS quality control, check [out the tutorials]({% link topics/sequence-analysis/index.md %}).

> <hands-on-title>Performing quality control</hands-on-title>
>
> 1. Create and name a new history for this tutorial.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the ChIP-seq raw data (\*.fastqsanger) from [Zenodo](https://doi.org/10.5281/zenodo.197100).
>
>    ```
>    https://zenodo.org/record/197100/files/G1E_input_R1_downsampled_SRR507859.fastqsanger
>    https://zenodo.org/record/197100/files/G1E_input_R2_downsampled_SRR507860.fastqsanger
>    https://zenodo.org/record/197100/files/G1E_Tal1_R1_downsampled_SRR492444.fastqsanger
>    https://zenodo.org/record/197100/files/G1E_Tal1_R2_downsampled_SRR492445.fastqsanger
>    https://zenodo.org/record/197100/files/Megakaryocyte_input_R1_downsampled_SRR492453.fastqsanger
>    https://zenodo.org/record/197100/files/Megakaryocyte_input_R2_downsampled_SRR492454.fastqsanger
>    https://zenodo.org/record/197100/files/Megakaryocyte_Tal1_R1_downsampled_SRR549006.fastqsanger
>    https://zenodo.org/record/197100/files/Megakaryocytes_Tal1_R2_downsampled_SRR549007.fastqsanger
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Examine the data in a FASTQ file by clicking on the {% icon galaxy-eye %} (eye) icon.
>
>    > <question-title></question-title>
>    >
>    > 1. What are four key features of a FASTQ file?
>    > 2. What is the main difference between a FASTQ and a FASTA file?
>    >
>    > > <solution-title></solution-title>
>    > > 1. A FASTQ file contains a sequence identifier and additional information, the raw sequence, information about the sequence again with optional information, and quality information about the sequence.
>    > > 2. A FASTA file contains only the description of the sequence and the sequence itself. A FASTA file does not contain any quality information.
>    > {: .solution }
>    {: .question}
>
> 4. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72+galaxy1) %}: Run FastQC on each FASTQ file to assess the quality of the raw data. An explanation of the results can be found on the [FastQC web page](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/).
>    - {% icon param-files %} *"Short read data from your current history"*: The uploaded fastqsanger files.
>
>    ![fastqbefore](../../images/tal1/fastqc_before.png "Sequence quality per base generated by FastQC before end trimming.")
>
>    > <question-title></question-title>
>    >
>    > 1. What does the y-axis represent in Figure 3?
>    > 2. Why is the quality score decreasing across the length of the reads?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The phred-score. This score gives the probability of an incorrect base *e.g.* a score of 20 means that it is likely by 1% that one base is incorrect. See [the wikipedia page on Phred](https://en.wikipedia.org/wiki/Phred_quality_score) for more information.
>    > > 2. This is an unsolved technical issue of the sequencing machines. The longer the sequences are the more likely are errors. See [this article](https://www.ecseq.com/support/ngs/why-does-the-sequence-quality-decrease-over-the-read-in-illumina) for more information.
>    > {: .solution }
>    {: .question}
{: .hands_on}

# Trimming and clipping reads

It is often necessary to trim a sequenced read to remove bases sequenced with high uncertainty (*i.e.* low-quality bases). In addition, artificial adaptor sequences used in library preparation protocols need to be removed before attempting to align the reads to a reference genome.

> <hands-on-title>Trimming and clipping reads</hands-on-title>
>
> 1. {% tool [Trimmomatic](toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.38.0) %}: Run Trimmomatic to trim low-quality reads.
>    - *"Single-end or paired-end reads?"*: `Single-end`
>    - {% icon param-files %} *"Input FASTQ file"*: Select all of the FASTQ files
>    - *"Perform initial ILLUMINACLIP?"*: `No`
>    - *"Select Trimmomatic operation to perform"*: `Sliding window trimming (SLIDINGWINDOW)`
>    - *"Number of bases to average across"*: `4`
>    - *"Average quality required"*: `20`
>
>    > <tip-title>Changing datatypes</tip-title>
>    >
>    > If the FASTQ files cannot be selected, check whether their format is FASTQ with Sanger-scaled quality values (*fastqsanger*). If not, you can edit the data type by clicking on the pencil symbol next to a file in the history, clicking the "Datatype" tab, and choosing *fastqsanger* as the "New Type".
>    {: .tip}
>
> 2. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72+galaxy1) %}: Rerun FastQC on each trimmed/clipped FASTQ file to determine whether low-quality and adaptor sequences were correctly removed.
>    - {% icon param-files %} *"Short read data from your current history"*: The output of Trimmomatic.
>
>    > <question-title></question-title>
>    >
>    > 1. How did the range of read lengths change after trimming/clipping?
>    >
>    > > <solution-title></solution-title>
>    > > 1. Before trimming, all the reads were the same length, which reflected the number of rounds of nucleotide incorporation in the sequencing experiment. After trimming, read lengths span a range of values reflecting different lengths of the actual DNA fragments captured during the ChIP experiement.
>    > {: .solution }
>    {: .question}
>
>    ![fastqafter](../../images/tal1/fastqc_after.png "Sequence quality per base generated by FastQC after end trimming.")
{: .hands_on}

# Aligning reads to a reference genome

To determine where DNA fragments originated from in the genome, the sequenced reads must be aligned to a reference genome. This is equivalent to solving a jigsaw puzzle, but unfortunately, not all pieces are unique. In principle, you could do a BLAST analysis to figure out where the sequenced pieces fit best in the known genome. Aligning millions of short sequences this way, however, can take a couple of weeks.
Nowadays, there are many read alignment programs for sequenced DNA, BWA being one of them. You can read more about the BWA algorithm and tool in {% cite Li_2009 %}.

> <hands-on-title>Aligning reads to a reference genome</hands-on-title>
>
> 1. {% tool [BWA](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa/0.7.17.4) %}: Run BWA to map the trimmed/clipped reads to the mouse genome.
>    - *"Will you select a reference genome..."*: `Use a built-in genome index`
>    - *"Using reference genome"*: `Mouse (mus musculus) mm10`
>    - *"Select input type"*: `Single fastq`
>    - {% icon param-files %} *"Select fastq dataset"*: Select all of the trimmed FASTQ files
>
> 2. Rename files to reflect the origin and contents.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. Inspect a file produced by BWA.
>
>    > <question-title></question-title>
>    >
>    > 1. What datatype is the BWA output file?
>    > 2. How many reads were mapped from each file?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The output is a BAM file.
>    > > 2. Check the number of lines for each file in your history. This gives you a rough estimate.
>    > {: .solution }
>    {: .question}
>
> 4. {% tool [Samtools idxstats](toolshed.g2.bx.psu.edu/repos/devteam/samtools_idxstats/samtools_idxstats/2.0.3) %}: Run idxstats to get statistics of the BWA alignments.
>     - {% icon param-files %} *"BAM file"*: Select all of the mapped BAM files
>
> 5. Examine the output ({% icon galaxy-eye %})
>
>    > <question-title></question-title>
>    >
>    > 1. What does each column in the output represent (**Tip**: look at the Tool Form)?
>    > 2. How many reads were mapped to chromosome 19 in each experiment?
>    > 3. If the mouse genome has 21 pairs of chromosomes, what are the other reference chromosomes (*e.g.* chr1_GL456210_random)?
>    >
>    > > <solution-title></solution-title>
>    > > 1. Column 1: Reference sequence identifier
>    > >    Column 2: Reference sequence length
>    > >    Column 3: Number of mapped reads
>    > >    Column 4: Number of placed but unmapped reads (typically unmapped partners of mapped reads)
>    > > 2. This information can be seen in column 3, *e.g.* for Megakaryocyte_Tal1_R1 2143352 reads are mapped. Your answer might be slightly different if different references or tool versions are used.
>    > > 3. Some of these other reference sequences are parts of chromosomes but it is unclear where exactly, *e.g.* chr1_GL456210_random is a part of chromosome 1. There are entries like `chrUn` that are not associated with a chromosome but it is believed that they are part of the genome.
>    > {: .solution }
>    {: .question}
{: .hands_on}

# Assessing correlation between samples

To assess the similarity between the replicates sequencing datasets, it is a common technique to calculate the correlation of read counts for the different samples.

We expect that the replicate samples will cluster more closely to each other than to other samples. We will be use tools from the package deepTools for the next few steps. More information on deepTools can be found [in deepTools' documentation](https://deeptools.readthedocs.io/en/latest/content/list_of_tools.html).

> <hands-on-title>Assessing correlation between samples</hands-on-title>
>
> multiBamSummary splits the reference genome into bins of equal size and counts the number of reads in each bin from each sample. We set a small bin size here because we are working with a subset of reads that align to only a fraction of the genome.
>
> 1. {% tool [multiBamSummary](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_multi_bam_summary/deeptools_multi_bam_summary/3.3.2.0.0) %}: Run multiBamSummary to get read coverage of the alignments.
>     - *"Sample order matters"*: `No`
>     - {% icon param-files %} *"Bam files"*: Select all of the aligned BAM files
>     - *"Bin size in bp"*: 1000
>
> 2. {% tool [plotCorrelation](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_correlation/deeptools_plot_correlation/3.3.2.0.0) %}: Run plotCorrelation to visualize the results.
>     - *"Matrix file from the multiBamSummary tool"*: Select the multiBamSummary output file
>     - *"Correlation method"*: `Pearson`
>     - *"Plotting type"*: `Heatmap`
>     - *"Plot the correlation value"*: `Yes`
>     - *"Skip zeros"*: `Yes`
>     - *"Remove regions with very large counts"*: `Yes`
>
>     Feel free to play around with these parameter settings!
>
>     > <question-title></question-title>
>     >
>     > 1. Why do we want to skip zeros in **plotCorrelation**?
>     > 2. What happens if the Spearman's correlation method is used instead of the Pearson method?
>     > 3. What does the output of making a Scatterplot instead of a Heatmap look like?
>     >
>     > > <solution-title></solution-title>
>     > > 1. Large areas of zeros would lead to a correlation of these areas. The information we would get out of this computation would be meaningless.
>     > > 2. The clusters are different, *e.g.* Megakaryocyte_input_R2 and G1E_input_R2 are clustered together. [ More information about Pearson and Spearman's correlation. ](http://support.minitab.com/en-us/minitab-express/1/help-and-how-to/modeling-statistics/regression/supporting-topics/basics/a-comparison-of-the-pearson-and-spearman-correlation-methods/)
>     > > 3. Try making a Scatterplot to see for yourself!
>     > {: .solution }
>     {: .question}
>
>     ![heatmap](../../images/tal1/plotCorrelation_heatmap_pearson_1kb.png "Heatmap of correlation matrix generated by plotCorrelation.")
{: .hands_on}

Additional information on how to interpret plotCorrelation plots can be found [in deepTools' documentation](https://deeptools.readthedocs.io/en/latest/content/tools/plotCorrelation.html#background).

# Assessing IP strength

We will now evaluate the quality of the immunoprecipitation step in the ChIP-seq protocol.

> <hands-on-title>Assessing IP strength</hands-on-title>
>
> 1. {% tool [plotFingerprint](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_fingerprint/deeptools_plot_fingerprint/3.3.2.0.0) %}: Run plotFingerprint to assess ChIP signal strength.
>    - {% icon param-files %} *"Bam files"*: Select all of the aligned BAM files for the G1E cell type
>    - *"Show advanced options"*: `yes`
>    - *"Bin size in bases"*: `100`
>    - *"Skip zeros"*: `Yes`
>
> 2. Rerun plotFingerprint for the Megakaryocyte cell type.
>
> 3. View the output images.
>
>    ![fingerprintimage](../../images/tal1/plotFingerprint_graph_v2.png "Strength of immunoprecipitation of TAL1.")
>
>    > <question-title></question-title>
>    >
>    > 1. What does this graph in Figure 10 represent?
>    > 2. How do (or should) input datasets differ from IP datasets?
>    > 3. What do you think about the quality of the IP for this experiment?
>    > 4. How does the quality of the IP for megakaryocytes compare to G1E cells?
>    >
>    > > <solution-title></solution-title>
>    > > 1. It shows us how good the ChIP Signal compared to the control signal is. An ideal control (input) with perfect uniform distribution of reads along the genome (*i.e.* without enrichment in open chromatin etc.) and infinite sequencing coverage should generate a straight diagonal line. A very specific and strong ChIP enrichment will be indicated by a prominent and steep rise of the cumulative sum towards the highest rank.
>    > > 2. We expect that the control (input) signal is more or less uniform distributed over the genome (*e.g.* like the green line in the image above). The IP dataset should look more like the red line but it would be better if the values for IP start to increase at around 0.8 on the x-axis.
>    > > 3. The enrichment did not work as it should. Compare the blue line with the red one! For your future experiments: You can never have enough replicates!
>    > > 4. The quality of megakaryocytes is better then G1E.
>    > {: .solution }
>    {: .question}
{: .hands_on}

Additional information on how to interpret plotFingerprint plots can be found [in deepTools' documentation](https://deeptools.readthedocs.io/en/latest/content/tools/plotFingerprint.html#background).

# Determining TAL1 binding sites

Now that BWA has aligned the reads to the genome, we will use the tool MACS2 to identify regions of TAL1 occupancy, which are called "peaks". Peaks are determined from pileups of sequenced reads across the genome that correspond to where TAL1 binds.

MACS2 will perform two tasks:
1. Identify regions of TAL1 occupancy (peaks).
2. Generate bedGraph files for visual inspection of the data on a genome browser.

More information about MACS2 can be found in {% cite Zhang_2008 %}.

> <hands-on-title>Determining TAL1 binding sites</hands-on-title>
>
> 1. {% tool [MACS2 callpeak](toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.1.1.20160309.6) %}: Run MACS2 callpeak with the aligned read files from the previous step as Treatment (TAL1) and Control (input).
>    - *"Are you pooling Treatment Files?"*: `Yes`
>    - {% icon param-files %} *"ChIP-Seq Treatment File"*: Select all of the replicate ChIP-Seq treatment aligned BAM files for one cell type
>    - *"Do you have a Control File?"*: `Yes`
>    - *"Are you pooling Control Files?"*: `Yes`
>    - {% icon param-files %} *"ChIP-Seq Control File"*: Select replicate ChIP-Seq control aligned BAM files for the same cell type
>    - *"Format of Input Files"*: `Single-end BAM`
>    - *"Effective genome size"*: `M. musculus`
>    - *"Additional Outputs"*: Select `Peaks as tabular file (compatible wih MultiQC)`, `Peak summits`, `Scores in bedGraph files (--bdg)`
>
> 2. Rename files to reflect the origin and contents.
>
> 3. Repeat for the other cell type.
{: .hands_on}

# Inspection of peaks and aligned data

It is critical to visualize NGS data on a genome browser after alignment to evaluate the "goodness" of the analysis. Evaluation criteria will differ for various NGS experiment types, but for ChIP-seq data we want to ensure reads from a Treatment/IP sample are enriched at peaks and do not localize non-specifically (like the control/input condition).

MACS2 generates bedGraph and BED files that we will use to visualize read abundance and peaks, respectively, at regions MACS2 determines to be TAL1 peaks using Galaxy's in-house genome browser, Trackster.

## Inspection of peaks and aligned data with Trackster

We will import a gene annotation file so we can visualize aligned reads and TAL1 peaks relative to gene features and positions.

> <hands-on-title>Inspecting peaks and aligned data with Trackster</hands-on-title>
>
> 1. Import gene annotations file from Zenodo [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.197100.svg)](https://doi.org/10.5281/zenodo.197100)
>
> 2. Click "Visualize" on the page header and select "Create Visualization"
>
>    ![vizbutton](../../images/tal1/create-visualization.png "Trackster can be accessed from the Visualize button at the top of the screen.")
>
> 3. Set up Trackster
>    - Select Trackster
>    - *"Select a dataset to visualize"*: Select the imported gene annotation file (**Tip**: if this file doesn't appear as an option, go back to the history and edit the attribute Database/Build to be `mm10`)
>    - Click "Create Visualization"
>
> 4. Configure the visualization
>    - Select "View in new visualization"
>    - *"Browser name"*: Enter a name for your visualization
>    - *"Reference genome build (dbkey):"*: `mm10`
>    - Click "Create"
>
>    ![tracksterdb](../../images/tal1/configure-visualization.png "Assign session name and reference genome.")
>
> 5. Add tracks to Trackster
>    - Click the "Add tracks" (plus sign) button at the top right
>    - Using the search bar, search for and add the following tracks from MACS2 callpeak output to your view
>       - G1E Treatment bedGraph
>       - G1E Control bedGraph
>       - G1E narrow peaks
>       - Megakaryocytes Treatment bedGraph
>       - Megakaryocytes Control bedGraph
>       - Megakaryocytes narrow peaks
>    - Rename the tracks, if desired
>    - Play around with the track configurations, for example the color or the display mode
>
>    ![tracksterselect](../../images/tal1/add-tracks.png "Select data from the history to view in Trackster.")
>
> 6. Navigate to the *Runx1* locus (`chr16:92501466-92926074`) to inspect the aligned reads and TAL1 peaks.
>
>    > <question-title></question-title>
>    >
>    > 1. What do you see at the *Runx1* locus in Trackster?
>    >
>    > > <solution-title></solution-title>
>    > > 1. Directly upstream of the shorter *Runx1* gene models is a cluster of 3 TAL1 peaks that only appear in the G1E cell type, but not in Megakaryocytes. Further upstream, there are some shared TAL1 peaks in both cell types.
>    > {: .solution }
>    {: .question}
>
>    ![runx1](../../images/tal1/Trackster_Runx1_locus.png "The Runx1 locus with TAL1 peaks identified.")
{: .hands_on}

## Inspection of peaks and aligned data with IGV
We show here an alternative to Trackster, [IGV](http://software.broadinstitute.org/software/igv/).

> <hands-on-title>Inspecting peaks with IGV</hands-on-title>
>
> 1. Open IGV on your local computer.
> 2. Click on each narrow peaks result file from the MACS2 computations on "display with IGV" --> "local Mouse mm10"
> 3. For more information about IGV see [the IGV Tutorial]({% link topics/introduction/tutorials/igv-introduction/tutorial.md %})
{: .hands_on}

# Identifying unique and common TAL1 peaks between stages

We have processed ChIP-seq data from two stages of hematopoiesis and have lists of TAL1-occupied sites (peaks) in both cellular states. The next analysis step is to identify TAL1 peaks that are *shared* between the two cellular states and peaks that are *specific* to either cellular state.

> <hands-on-title>Identifying unique and common TAL1 peaks between states</hands-on-title>
>
> 1. {% tool [bedtools Intersect intervals](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.29.0) %}: Run bedtools Intersect intervals to find peaks that exist both in G1E and megakaryocytes.
>    - {% icon param-file %} *"File A to intersect with B"*: Select the TAL1 G1E narrow peaks BED file
>    - {% icon param-file %} *"File B to intersect with A"*: Select the TAL1 Megakaryocytes narrow peaks BED file
>    - Running this tool with the default settings will return overlapping peaks of both files.
>
> 2. {% tool [bedtools Intersect intervals](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.29.0) %}: Run bedtools Intersect intervals to find peaks that exist only in G1E.
>    - {% icon param-file %} *"File A to intersect with B"*: Select the TAL1 G1E narrow peaks BED file
>    - {% icon param-file %} *"File B to intersect with A"*: Select the TAL1 Megakaryocytes narrow peaks BED file
>    - *"Report only those alignments that \*\*do not\*\* overlap the BED file"*: `Yes`
>
> 3. {% tool [bedtools Intersect intervals](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.29.0) %}: Run bedtools Intersect intervals to find peaks that exist only in megakaryocytes.
>    - {% icon param-file %} *"File A to intersect with B"*: Select the TAL1 Megakaryocytes narrow peaks BED file
>    - {% icon param-file %} *"File B to intersect with A"*: Select the TAL1 G1E narrow peaks BED file
>    - *"Report only those alignments that \*\*do not\*\* overlap the BED file"*: `Yes`
>
> 4. Rename files to reflect the origin and contents.
>
>    > <question-title></question-title>
>    >
>    > 1. How many TAL1 peaks are common to both G1E cells and megakaryocytes?
>    > 2. How many are unique to G1E cells?
>    > 3. How many are unique to megakaryocytes?
>    >
>    > > <solution-title></solution-title>
>    > > 1. 1 peak (answer may vary depending on references and tool versions used)
>    > > 2. 407 peaks (answer may vary depending on references and tool versions used)
>    > > 3. 139 peaks (answer may vary depending on references and tool versions used)
>    > {: .solution }
>    {: .question}
{: .hands_on}

# Generating Input normalized coverage files

We will generate Input normalized coverage (bigWig) files for the ChIP samples, using the bamCompare tool from deepTools2. bamCompare provides multiple options to compare the two files (*e.g.* log2ratio, subtraction). We will use log2 ratio of the ChIP samples over Input.

> <hands-on-title>Generating input-normalized bigwigs</hands-on-title>
>
> 1. {% tool [bamCompare](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_compare/deeptools_bam_compare/3.3.2.0.0) %}: Run bamCompare to get the log2 read ratios between treatment and control samples.
>    - *"First BAM/CRAM file (*e.g.* treated sample)"*: Select the Megakaryocyte TAL1 aligned BAM file for replicate 1 (R1)
>    - *"Second BAM/CRAM file (*e.g.* control sample)"*: Select the Megakaryocyte input aligned BAM file for replicate 1 (R1)
>    - *"How to compare the two files"*: `Compute log2 of the number of reads`
>
> 2. Repeat this step for all treatment and control samples:
>     - Megakaryocyte TAL1 aligned BAM R2 and Megakaryocyte input aligned BAM R2
>     - G1E TAL1 aligned BAM R1 and G1E input aligned BAM R1
>     - G1E TAL1 aligned BAM R2 and G1E input aligned BAM R2
>
> 3. Rename files to reflect the origin and contents.
{: .hands_on}

# Plot the signal on the peaks between samples

Plotting your region of interest will involve using two tools from the deepTools suite:
+ computeMatrix: Computes the signal on given regions, using the bigwig coverage files from different samples.
+ plotHeatmap: Plots heatMap of the signals using the computeMatrix output.

Optionally, you can use plotProfile to create a profile plot using to computeMatrix output.

> <hands-on-title>Calculating signal matrix on MACS2 output</hands-on-title>
>
> 1. {% tool [computeMatrix](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.3.2.0.0) %}: Run computeMatrix to prepare data for plotting a heatmap of TAL1 peaks.
>    - {% icon param-file %} *Select Regions* > *"Regions to plot"*: Select the MACS2 narrow peaks files for G1E cells (TAL1 over Input)
>    - {% icon param-file %} *"Score file"*: Select the bigWig files for the G1E cells (log2 ratios from bamCompare)
>    - *"computeMatrix has two main output options"*: `reference-point`
>    - *"The Reference point for plotting"*: `center of region`
>    - *"Distance upstream of the start site of the regions defined in the region file"*: `5000`
>    - *"Distance downstream of the end site of the given regions"*: `5000`
>    - *"Show advanced options"*: `Yes`
>    - *"Convert missing values ot zero"*: `Yes`
>    - *"Skip zeros"*: `Yes`
>
> 2. Repeat for Megakaryoctes.
{: .hands_on}

> <hands-on-title>Plotting a heatmap of TAL1 peaks</hands-on-title>
>
> 1. {% tool [plotHeatmap](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_heatmap/deeptools_plot_heatmap/3.3.2.0.1) %}: Run plotHeatmap to create a heatmap for score distributions across TAL1 peak genomic regions in each cell type.
>    - *"Matrix file from the computeMatrix tool"*: Select the computeMatrix output for G1E cells
>    - *"Show advanced options"*: `Yes`
>    - *"Labels for the samples (each bigwig) plotted"*: Enter sample labels in the order you added them in computeMatrix, separated by spaces.
>
> 2. Repeat for Megakaryocytes.
>
> The outputs should look similar to this:
>
> ![hm](../../images/tal1/TAL1-plotheatmap.png)
{: .hands_on}

# Additional optional analyses

## Assessing GC bias

A common problem of PCR-based protocols is the observation that GC-rich regions tend to be amplified more readily than GC-poor regions.

We will now check whether the samples have more reads from regions of the genome with high GC.

> <hands-on-title>Assessing GC bias</hands-on-title>
>
> 1. {% tool [computeGCbias](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_gc_bias/deeptools_compute_gc_bias/3.3.2.0.0) %}: Run computeGCbias to determine the GC bias of the sequenced reads.
>    - {% icon param-file %} *"Bam file"*: Select an aligned BAM file
>    - *"Reference genome"*: `locally cached`
>    - *"Using reference genome"*: `mm10`
>    - *"Effective genome size"*: `user specified`
>    - *"Effective genome size"*: `10000000`
>    - *"Fragment length used for the sequencing"*: `50`
>
>    > <question-title></question-title>
>    >
>    > 1. Why would we worry more about checking for GC bias in an input file?
>    > 2. Does this dataset have a GC bias?
>    >
>    > > <solution-title></solution-title>
>    > > 1. In an input ChIP-seq file, the expectation is that DNA fragments are uniformly sampled from the genome. This is in contrast to an IP ChIP-seq file where it is expected that certain genomic regions contain more reads (*i.e.* regions that are bound by the protein that is immunopurified). Therefore, non-uniformity of reads in the input sample could be a result of GC-bias, whereby more GC-rich fragments are preferentially amplified during PCR.
>    > > 2. To answer this question, run the computeGCbias tool as described above and check out the results. What do YOU think? For more examples and information on how to interpret the results, check out the tool usage documentation [in deepTools' documentation](https://deeptools.readthedocs.io/en/latest/content/tools/computeGCBias.html#background).
>    > {: .solution }
>    {: .question}
>
> 2. {% tool [correctGCbias](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_correct_gc_bias/deeptools_correct_gc_bias/3.3.2.0.0) %}: Run correctGCbias to generate GC-corrected BAM/CRAM files.
>
>    > <question-title></question-title>
>    >
>    > 1. What does the tool correctGCbias do?
>    > 2. What is the output of this tool?
>    > 3. What are some caveats to be aware of if using the output of this tool in downstream analyses?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The correctGCbias tool removes reads from regions with higher coverage than expected (typically corresponding to GC-rich regions) and adds reads to regions with lower coverage than expected (typically corresponding to AT-rich regions).
>    > > 2. The output of this tool is a GC-corrected file in BAM, bigWig, or bedGraph format.
>    > > 3. The GC-corrected output file likely contains duplicated reads in low-coverage regions where reads were added to match the expected read density. Therefore, it is necessary to avoid filtering or removing duplicate reads in any downstream analyses.
>    > {: .solution }
>    {: .question}
{: .hands_on}

Additional information on how to interpret computeGCbias plots can be found [in deepTools' documentation](https://deeptools.readthedocs.io/en/latest/content/tools/computeGCBias.html).


# Conclusion


In this exercise you imported raw Illumina sequencing data, evaluated the quality before and after you trimmed reads with low confidence scores, aligned the trimmed reads, identified TAL1 peaks relative to the negative control (background), and visualized the aligned reads and TAL1 peaks relative to gene structures and positions. Additional, you assessed the "goodness" of the experiments by looking at metrics such as GC bias and IP enrichment.

tools_json:
["FastQC", "toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72+galaxy1", "Trimmomatic", "toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.38.0", "BWA", "toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa/0.7.17.4", "Samtools idxstats", "toolshed.g2.bx.psu.edu/repos/devteam/samtools_idxstats/samtools_idxstats/2.0.3", "multiBamSummary", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_multi_bam_summary/deeptools_multi_bam_summary/3.3.2.0.0", "plotCorrelation", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_correlation/deeptools_plot_correlation/3.3.2.0.0", "plotFingerprint", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_fingerprint/deeptools_plot_fingerprint/3.3.2.0.0", "MACS2 callpeak", "toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.1.1.20160309.6", "bedtools Intersect intervals", "toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.29.0", "bamCompare", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_compare/deeptools_bam_compare/3.3.2.0.0", "computeMatrix", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.3.2.0.0", "plotHeatmap", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_heatmap/deeptools_plot_heatmap/3.3.2.0.1", "computeGCbias", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_gc_bias/deeptools_compute_gc_bias/3.3.2.0.0", "correctGCbias", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_correct_gc_bias/deeptools_correct_gc_bias/3.3.2.0.0"]


topic: epigenetics
tutorial: formation_of_super-structures_on_xi
title: Formation of the Super-Structures on the Inactive X
difficulty: intermediate
time_estimation: 3h
requirements_json: []
tags_json: []
content_hash: e27a61e40e301705a0c7bcb7e05ee4e8
last_modified: 2026-05-27T09:01:48.465431
zenodo_link: https://zenodo.org/record/1324070
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/epigenetics/tutorials/formation_of_super-structures_on_xi/tutorial.html

content:
Within a cell nucleus, the DNA is tightly-packed and the chromatin is spatially distributed with different levels and scales of organizations.

At the smallest scale, DNA is packaged into units called nucleosomes, made of eight histone proteins.

On a larger scale than nucleosomes, DNA is forming loops. DNA elements that would be otherwise separated by large distances can interact. The corresponding self-interacting (or self-associating) domains are found in many organisms: they are called Topologically Associating Domains (TADs) in mammalian cells. Mammalian chromosomes are also partitioned into two spatial compartments, labeled "A" and "B", where regions in compartment A tend to interact preferentially with A compartment-associated regions than B compartment-associated ones. Similarly, regions in compartment B tend to associate with other B compartment-associated regions.

> <comment-title>Tip: Learn more about chromosome conformation</comment-title>
> To learn more about chromosome conformation and TADs, you can follow our [HiC tutorial]({{site.baseurl}}/topics/epigenetics/tutorials/hicexplorer/tutorial.html)
{: .comment}

In mammals, the X chromosome inactivation (XCI) balances the dosage of X-linked genes between females and males. The genes on the inactive X (Xi) chromosome are not expressed.

Binding certain molecules to histone proteins can modify the chromatin structure and may result in changes in transcription level. For example, the H3K4me3 results from adding 3 methyl-group to the 4th lysine in the histone 3 protein. This modification is known to activate the transcription on nearby genes by opening the chromatin. The H3K27me3 on the other hand is inactivating the transcription of the nearby genes:

![Fadloun et al, 2013](../../images/formation_of_super-structures_on_xi/histone_modifications.jpg "Source: Fadloun et al, 2013")

More H3K27me3 and less H3K4me3 on the Xi could explain a lower expression of the genes there.

It has been also observed that the Xi reconfigures uniquely into a specific chromosomal conformation. cohesins, condensins and CCCTC-binding factor (CTCF) play key roles in chromosomal architectures and TAD formation which are other potential cause of the repression of the expression of the genes on Xi.

The structural-maintenance-of-chromosomes hinge domain containing 1 (SMCHD1) has been found enriched on the Xi. It may be the potential actor in the shape of Xi and the change in gene expression there.

{% cite Wang2018 %} investigates the mechanism by which the SMCHD1 gene shapes the Xi and represses the expression of the genes on Xi in mouse.

Their idea was to identify the differences which could be observed between the Xi and activated X chromosome, on both wild-type and SMCHD1 gene knockdown samples to study the SMCHD1 effect.
In different experiments, they targeted histones with H3K27me3 or H3K4me3 and CTCF using ChIP-seq experiments.

![ChIP-seq workflow](../../images/formation_of_super-structures_on_xi/Chromatin_immunoprecipitation_sequencing.jpg "Source: http://e.biohackers.net/ChIP-seq")

They obtained sequences corresponding to a portion of DNA linked to histones with H3K27me3, H3K4me3 or CTCF. Using this information, they could identify if there are differences in the H3K27me3, H3K4me3 and CTCF between the X (active or inactive) chromosomes and the potentially influenced genes.

In the upcoming tutorial, we will only use the wild type data from {% cite Wang2018 %} and analyze the ChIP-seq data step by step:

- CTCF with 2 replicates: `wt_CTCF_rep1` and `wt_CTCF_rep2`
- H3K4me3 with 2 replicates: `wt_H3K4me3_rep1` and `wt_H3K4me3_rep2`
- H3K27me3 with 2 replicates: `wt_H3K27me3_rep1` and `wt_H3K27me3_rep2`
- 'input' with 2 replicates: `wt_input_rep1` and `wt_input_rep2`

    In 'input' samples, the same treatment as the ChIP-seq samples was done except for the immunoprecipitation step.
    They are also used along with the 'ChIP-seq' samples to identify the potential sequencing bias and help for differential analysis.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Step 1: Quality control and treatment of the sequences

The first step of any ChIP-Seq data analysis is quality control of the raw sequencing data.

To save time, we will do it only on the data of one sample `wt_H3K4me3_rep1` which has been already down-sampled. keep in mind that with real data this should be done on each and every sample.

> <hands-on-title>Import the data</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import `wt_H3K4me3_read1.fastq.gz` and `wt_H3K4me3_read2.fastq.gz` from [Zenodo](https://zenodo.org/record/1324070) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/1324070/files/wt_H3K4me3_read1.fastq.gz
>    https://zenodo.org/record/1324070/files/wt_H3K4me3_read2.fastq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Add a tag called `#wt_H3K4me3_read1` to the read1 file and a tag called `#wt_H3K4me3_read2` to the read2 file.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
> 4. Inspect the first file by clicking on the {% icon galaxy-eye %} (eye) icon (**View data**)
>
> 5. Create a paired collection named `Paired Reads`
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
{: .hands_on}

{% include topics/sequence-analysis/tutorials/quality-control/fastq_question.md %}

During sequencing, errors are introduced, such as incorrect nucleotides being called. These are due to the technical limitations of each sequencing platform. Sequencing errors might bias the analysis and can lead to a misinterpretation of the data.

Sequence quality control is therefore an essential first step in your analysis. We use here similar tools as described in ["Quality control" tutorial]({{site.baseurl}}/topics/sequence-analysis): [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) and [Trim Galore](https://www.bioinformatics.babraham.ac.uk/projects/trim_galore/).

> <hands-on-title>Quality control</hands-on-title>
> 
> 1. Run {% tool [Flatten collection](__FLATTEN__) %} with the following parameters:
>    - *"Input collection"*: `Paired Reads`
>   
> 2. Rename the flatten collection: `Flat Collection`
>   
> 3. Run {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy1) %} with the following parameters:
>    - {% icon param-collection %} *"Raw read data from your current history"*: `Flat Collection` (Flattened paired end read dataset collection)
>
> 4. Inspect the generated HTML files
>
>    > <question-title></question-title>
>    >
>    > 1. How is the quality of the reads in `wt_H3K4me3_read1`?
>    > 2. And in `wt_H3K4me3_read2`?
>    > 3. What should we do if the quality of the reads is not good?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The reads in `wt_H3K4me3_read1` are of good quality:
>    > >     - There is 50,000 sequences, all of 51 bp
>    > >     - The "Per base sequence quality" is not decreasing too much at the end of the sequences
>    > >
>    > >        ![Per base sequence quality for read1](../../images/formation_of_super-structures_on_xi/read1_per_base_sequence_quality.png "Per base sequence quality")
>    > >
>    > >     - The mean quality score over the reads is quite high
>    > >
>    > >        ![Per sequence quality scores for read1](../../images/formation_of_super-structures_on_xi/read1_per_sequence_quality_scores.png "Per sequence quality scores")
>    > >
>    > >     - Homogeneous percentage of the bases
>    > >
>    > >        ![Per base sequence content for read1](../../images/formation_of_super-structures_on_xi/read1_per_base_sequence_content.png "Per base sequence content")
>    > >
>    > >     - No N in the reads
>    > >
>    > >        ![Per base N content for read1](../../images/formation_of_super-structures_on_xi/read1_per_base_n_content.png "Per base N content")
>    > >
>    > >     - No duplicated sequences
>    > >
>    > >        ![Sequence Duplication Levels for read1](../../images/formation_of_super-structures_on_xi/read1_sequence_duplication_levels.png "Sequence Duplication Levels")
>    > >
>    > >     - No more known adapters
>    > >
>    > >        ![Adapter Content for read1](../../images/formation_of_super-structures_on_xi/read1_adapter_content.png "Adapter Content")
>    > >
>    > > 2. The reads in `wt_H3K4me3_reverse` are a bit worse:
>    > >
>    > >     - The "Per base sequence quality" is decreasing more at the end of the sequences, but it stays correct
>    > >
>    > >        ![Per base sequence quality for read2](../../images/formation_of_super-structures_on_xi/read1_per_base_sequence_quality.png "Per base sequence quality")
>    > >
>    > >     - The sequence `NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN` is over represented
>    > >
>    > >     Despite these small things, the overall quality of the reads is really good.
>    > >
>    > > 3. If the quality of the reads is not good, we should:
>    > >    1. Check what is wrong and think about it: it may come from the type of sequencing or what we sequenced (high quantity of overrepresented sequences in transcriptomics data, biaised percentage of bases in HiC data)
>    > >    2. Ask the sequencing facility about it
>    > >    3. Perform some quality treatment (in a reasonable way to not loose too much information) with some trimming or removal of bad reads
>    > >
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

It is often necessary to trim sequenced read, for example, to get rid of bases that were sequenced with high uncertainty (= low quality bases) at the read ends.

{% include topics/sequence-analysis/tutorials/quality-control/paired_end_question.md forward="wt_H3K4me3_read1" reverse="wt_H3K4me3_read2" %}

> <hands-on-title>Trimming low quality bases</hands-on-title>
>
> 1. Run {% tool [Trim Galore!](toolshed.g2.bx.psu.edu/repos/bgruening/trim_galore/trim_galore/0.6.7+galaxy0) %} with the following parameters:
>    - *"Is this library paired- or single-end?"*: `Paired Collection`
>       - {% icon param-collection %} *"Select a paired collection"*: `Paired Reads` (Input collection)
>
>       > <tip-title>Not selectable files?</tip-title>
>       >
>       > If your FASTQ files cannot be selected, you might check whether their format is FASTQ with Sanger-scaled quality values (`fastqsanger`). You can edit the data type by clicking on the `pencil` symbol.
>       {: .tip}
>
>    - *"Trim Galore! advanced settings"*: `Full parameter list`
>       - *"Trim low-quality ends from reads in addition to adapter removal"*: `15`
>       - *"Overlap with adapter sequence required to trim a sequence"*: `3`
>       - *"Generate a report file"*: `Yes`
>
> 2. Rename the Trim Galore! output `reads` collection: `Trimmed Reads`
>
> 3. Inspect the generated txt file (`report file`)
>
>    > <question-title></question-title>
>    >
>    > 1. How many basepairs has been removed from the forwards reads because of bad quality?
>    > 2. And from the reverse reads?
>    > 3. How many sequence pairs have been removed because at least one read was shorter than the length cutoff?
>    >
>    > > <solution-title></solution-title>
>    > > 1. 32,198 bp (1.3%) (first `Quality-trimmed:`)
>    > > 2. 116,414 bp (4.6%) (second `Quality-trimmed:`). It is not a surprise: we saw that the quality was dropping more at the end of the sequences for the reverse reads thant for the forward reads.
>    > > 3. 1569 (3.14%) sequences (last line of the file)
>    > {: .solution }
>    {: .question}
{: .hands_on}

# Step 2: Mapping of the reads

With ChiP sequencing, we obtain sequences corresponding to a portion of DNA linked to the histone mark of interest, H3K4me3 in this case. As H3K4me3 opens the chromatime, nearby genes are going to be more transcribed. It would be interesting to know if there is a difference in the quantity of DNA impacted by H3K4me3 and the impacted genes between active and inactive X chromosome.

{% include topics/sequence-analysis/tutorials/mapping/mapping_explanation.md
    to_identify="binding sites"
    mapper="Bowtie2"
    mapper_link="http://bowtie-bio.sourceforge.net/bowtie2/index.shtml"
    answer_3="Wang et al. (2018) did ChIP-seq on mouse. So we should use the mouse reference genome. We will use mm10 (the latest build)"
%}

## Running Bowtie2

> <hands-on-title>Mapping</hands-on-title>
>
> 1. {% tool [Bowtie2](toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.4+galaxy0) %} with the following parameters:
>    - *"Is this single or paired library"*: `Paired-end`
>        - {% icon param-collection %} *"FASTQ Paired Dataset"*: `Trimmed Reads`
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>        - *"Select reference genome"*: `Mouse (Mus musculus): mm10`
>    - *"Save the bowtie2 mapping statistics to the history"*: `Yes`
>
> 2. Inspect the mapping stats
>
>    > <question-title></question-title>
>    >
>    > How many reads where mapped? Uniquely or several times?
>    >
>    > > <solution-title></solution-title>
>    > > The overall alignment rate is 98.57%. This score is quite high. If you have less than 70-80%, you should investigate the cause: contamination, etc.
>    > >
>    > > 41514 (85.72%) reads have been aligned concordantly exactly 1 time and 5190 (10.72%) aligned concordantly >1 times. The latter ones correspond to multiple mapped reads. Allowing for multiple  mapped reads increases the number of usable reads and the sensitivity of peak detection;
>    > > however, the number of false positives may also increase.
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

The output of Bowtie2 is a BAM file.

## Inspection of a BAM file

{% include topics/sequence-analysis/tutorials/mapping/bam_explanation.md mapper="Bowtie2" %}

## Visualization using a Genome Browser

{% include topics/sequence-analysis/tutorials/mapping/igv.md tool="Bowtie2" region_to_zoom="chr2:91,053,413-91,055,345" %}

### Visualization using JBrowse2

Besides running IGV locally on our computer, we can use Jbrowse2 to look at the genomes and visualize different tracks such as Chip-Seq data using the Galaxy. To do so:

> <hands-on-title>Genome Visualization via Jbrowse2</hands-on-title>
>
> 1. {% tool [BAM BED GFF coverage bigWigs](toolshed.g2.bx.psu.edu/repos/iuc/bbgbigwig/bbgtobigwig/0.1) %} with the following parameters:
>    - *"bam/bed/gff to convert"*: `Output of the bowtie2`
>    - *"Source Genome Build"*: `Mouse (Mus Musculus): mm10 Full`
> 2. {% tool [JBrowse2](toolshed.g2.bx.psu.edu/repos/fubar/jbrowse2/jbrowse2/2.17.0+galaxy0) %} with the following parameters:
>    - *"Select a built in reference genome or custom genome"*: `Mouse (Mus Musculus): mm10 Full`
>    - *"Insert Track Group"*
>        - *"Insert Annotation Track"*
>        - *"Track Type"*: `BigWig Track`
>        - *"BigWig Track Data"*: `The output from step 1 (The BigWig File made from the BAM file)`
>
{: .hands_on}

This takes some time to finish. Again, you can explore the genome. For example, you can look at the following coordinate in the genome: `chr2:91,053,413-91,055,345`. For a comprehensive tutorial on Jbrowse please look at this tutorial: https://training.galaxyproject.org/training-material/topics/visualisation/tutorials/jbrowse/tutorial.html.

# Step 3: ChIP-seq Quality Control

We already checked the quality of the raw sequencing reads in the first step.
Now we would like to test the quality of the ChIP-seq preparation, to know if our ChIP-seq samples are more enriched than the control (input) samples.

## Correlation between samples

To assess the similarity between the replicates of the ChIP and the input, respectively, it is a common technique to calculate the correlation of read counts on different regions for all different samples.
We expect that the replicates of the ChIP-seq experiments should be clustered more closely to each other than the replicates of the input sample. That is, because the input samples should not have enriched regions included - remember the immuno-precipitation step was skiped during the sample preparation.

To compute the correlation between the samples we are going to to use the QC modules of [deepTools](http://deeptools.readthedocs.io/), a software package for the QC, processing and analysis of NGS data. Before computing the correlation a time consuming step is required, which is to compute the read coverage (number of unique reads mapped at a given nucleotide) over a large number of regions from each of the inputed BAM files. For this we will use the tool **multiBamSummary** {% icon tool %}. Then, we use **plotCorrelation** {% icon tool %} from deepTools to compute and visualize the sample correlation. This is a fast process that allows to quickly try different color combinations and outputs.

Since in this tutorial we are interested in assessing H3K4me3, H3K27me3 and CTCF ChIP samples, the previous steps (quality control and mapping) needs to be run on all the replicates of ChIP samples as well as the input samples. To save time, we have already done that and you can now work directly on the BAM files of the provided 8 samples. For simplicity, the files include only the ChrX.

> <hands-on-title>Correlation between samples</hands-on-title>
>
> 1. Create a new history
> 2. Import the 8 BAM files from [Zenodo](https://zenodo.org/record/1324070) or from the data library into the history
>
>    ```
>    https://zenodo.org/record/1324070/files/wt_CTCF_rep1.bam
>    https://zenodo.org/record/1324070/files/wt_CTCF_rep2.bam
>    https://zenodo.org/record/1324070/files/wt_H3K4me3_rep1.bam
>    https://zenodo.org/record/1324070/files/wt_H3K4me3_rep2.bam
>    https://zenodo.org/record/1324070/files/wt_H3K27me3_rep1.bam
>    https://zenodo.org/record/1324070/files/wt_H3K27me3_rep2.bam
>    https://zenodo.org/record/1324070/files/wt_input_rep1.bam
>    https://zenodo.org/record/1324070/files/wt_input_rep2.bam
>    ```
>
> 3. Rename the files
> 3. {% tool [multiBamSummary](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_multi_bam_summary/deeptools_multi_bam_summary/3.5.1.0.0) %} with the following parameters:
>    - *"Sample order matters"*: `No`
>       - {% icon param-files %} *"BAM/CRAM file"*: the 8 imported BAM files
>    - *"Choose computation mode"*: `Bins`
>       - *"Bin size in bp"*: `1000`
>
>           This corresponds to the length of the fragments that were sequenced; it is not the read length!
>
>       - *"Distance between bins"*: `500`
>
>           It reduces the computation time for the tutorial
>
>    - *"Region of the genome to limit the operation to"*: `chrX`
>
>    Using these parameters, the tool will take bins of 1000 bp separated by 500 bp on the chromosome X. For each bin the overlapping reads in each sample will be computed and stored into a matrix.
>
> 4. {% tool [plotCorrelation](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_correlation/deeptools_plot_correlation/3.5.1.0.0) %} with the following parameters:
>    - {% icon param-files %} *"Matrix file from the multiBamSummary tool"*: `correlation matrix`(output of **multiBamSummary** {% icon tool %})
>    - *"Correlation method"*: `Pearson`
>
>    Feel free to try different parameters for the configuration of the plot (colors, title, etc)
>
{: .hands_on}

> <question-title></question-title>
>
> ![Output for plotCorrelation with the correlation scores between the 8 samples](../../images/formation_of_super-structures_on_xi/plotCorrelation.png "Correlation scores between the 8 samples")
>
> How are your samples clustered? Does that correspond to your expectations?
>
> > <solution-title></solution-title>
> > As one could expect, the input replicates cluster together and the ChIP replicates cluster together. It confirms that the immuno-precipitation step worked on our ChIP replicates.
> >
> > Moreover, for each sample, there is a high correlation between the two replicates which confirms the validity of the experiments.
> >
> {: .solution }
{: .question}

## IP strength estimation

To evaluate the quality of the immuno-precipitation(IP) step, we can compute the IP strength. It determines how well the signal in the ChIP-seq sample can be differentiated from the background distribution of reads in the control sample ('input'). After all, around 90% of all DNA fragments in a ChIP experiment will represent the genomic background.

To do that we take the data from the `rep1` of the `wt_H3K4me3` ChIP-seq sample and compare it with its corresponding input sample, using **plotFingerprint** {% icon tool %} of deepTools.

Similar to **multiBamSummary** {% icon tool %}, **plotFingerprint** {% icon tool %} randomly samples genome regions of a specified length (bins) and sums the per-base coverage in the indexed BAM files that overlap with those regions. These coverage values are then sorted according to their rank and the cumulative sum of read counts is plotted.

> <hands-on-title>IP strength estimation</hands-on-title>
>
> 1. {% tool [plotFingerprint](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_fingerprint/deeptools_plot_fingerprint/3.5.1.0.0) %} with the following parameters:
>    - *"Sample order matters"*: `No`
>       - {% icon param-files %} *"BAM/CRAM file"*: `wt_input_rep1` and `wt_H3K4me3_rep1`
>    - *"Region of the genome to limit the operation to"*: `chrX`
>    - *"Show advanced options"*: `Yes`
>       - *"Number of samples"*: `10000`
{: .hands_on}

The plotFingerprint tool generates a fingerprint plot. You need to intepret it to know the IP strength.

An ideal 'input' with perfect uniform distribution of reads along the genome (i.e. without enrichments in open chromatin) and infinite sequencing coverage should generate a straight diagonal line. A very specific and strong ChIP enrichment will be indicated by a prominent and steep rise of the cumulative sum towards the highest rank. This means that a big chunk of reads from the ChIP sample is located in few bins which corresponds to high, narrow enrichments typically seen for transcription factors.

![A guide to interpret a fingerprint plot](../../images/estrogen-receptor-binding-site-identification/QC_fingerprint.png "How to interpret a fingerprint plot? Image extracted from the deepTools documentation")

> <question-title></question-title>
>
> ![Output for plotFingerprint with the fingerprint plot to estimate the IP strength](../../images/formation_of_super-structures_on_xi/plotFingerPrint.png "Fingerprint plot for the first replicates to estimate the IP strength")
>
> What do you think about the quality of the IP for this experiment?
>
> > <solution-title></solution-title>
> > There is clear distinction between H3K4me3 and the input.
> >
> > A small percentage of the genome contain a very large fraction of the reads (>70%, point of change in the blue curve)
> >
> > The curves start to rise around 0.25. It means that almost 25% of the chromosome X is not sequenced at all.
> {: .solution }
{: .question}

> <hands-on-title>(Optional) IP strength estimation (other samples)</hands-on-title>
>
> Run the same analysis on the other ChIP-seq data along with their corresponding input and compare the output
{: .hands_on}

# Step 4: Normalization

One of the goals in ChIP-seq data analysis is finding regions on the genome which are enriched for the ChIP data of interest (regions with significantly higher read coverage for the ChIP data comparing to its corresponding input). In the following exercise we would like to know where the H3K4me3 sites are. For this we need to extract which parts of the genome have been enriched (i.e. more reads mapped) within the samples that underwent immunoprecipitation. However, to reach a reliable comparison the data needs to be normalized to remove any technical bias. For the normalization we have two steps:

1. Normalization by sequencing depth
2. Normalization by input file

To learn how to do the normalization, we will take the `wt_H3K4me3_rep1` sample as ChIP data and `wt_input_rep1` as input.

## Generation of coverage files normalized by sequencing depth

> <hands-on-title>Estimation of the sequencing depth</hands-on-title>
>
> 1. {% tool [Samtools idxstats](toolshed.g2.bx.psu.edu/repos/devteam/samtools_idxstats/samtools_idxstats/2.0.4) %} with the following parameters:
>    - {% icon param-files %} *"BAM file"*: `wt_H3K4me3_rep1.bam` and `wt_input_rep1.bam`
>
> > <question-title></question-title>
> >
> > 1. What is the output of **Samtools idxstats** {% icon tool %}?
> > 2. How many reads has been mapped on chrX for the input and for the ChIP-seq samples?
> > 3. Why are the number of reads different? And what could be the impact of this difference?
> >
> > > <solution-title></solution-title>
> > > 1. This tool generates a table with 4 columns:  reference sequence identifier, reference sequence length, number of mapped reads and number of placed but unmapped reads. Here it estimates how many reads mapped to which chromosome. Furthermore, it tells the chromosome lengths and naming convention (with or without 'chr' in the beginning)
> > > 2. 1,204,821 for ChIP-seq samples and 1,893,595 for the input
> > > 3. The number of reads can be different because of  different sequencing depth. It can bias the interpretation of the number of reads mapped to a specific genome region and the identification of the H3K4me3 sites. Specially here, as the number of reads for the input is higher than the ChIP data less regions could be identified having a significantly higher read coverage for the ChIP data comparing to the corresponding input.
> > {: .solution }
> {: .question}
{: .hands_on}

The different samples have usually a different sequencing depth, i.e. a different number of reads. These differences can bias the interpretation of the number of reads mapped to a specific genome region. We first need to make the samples comparable by normalizing the coverage by the sequencing depth.

We are using **bamCoverage** {% icon tool %}. Given a BAM file, this tool generates coverages by first calculating all the number of reads (either extended to match the fragment length or not) that overlap each bin in the genome and then normalizing with various options. It produces a coverage file where for each bin the number of overlapping reads (possibly normalized) is noted.

> <hands-on-title>Coverage file normalization</hands-on-title>
>
> 1. {% tool [bamCoverage](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_coverage/deeptools_bam_coverage/3.5.1.0.0) %} with the following parameters:
>    - {% icon param-files %} *"BAM file"*: `wt_H3K4me3_rep1.bam` and `wt_input_rep1.bam`
>    - *"Bin size in bases"*: `25`
>    - *"Scaling/Normalization method"*: `Normalize coverage to 1x`
>       - *"Effective genome size"*: `GRCm38/mm10 (2308125349)`
>    - *"Coverage file format"*: `bedgraph`
>    - *"Region of the genome to limit the operation to"*: `chrX`
>
>    > <question-title></question-title>
>    >
>    > 1. What is a `bedgraph` file?
>    > 2. Which regions have the highest coverage in ChIP data and in the input?
>    >
>    > > <solution-title></solution-title>
>    > > 1. It is a tabular file with 4 columns: chrom, chromStart, chromEnd and a data value (coverage)
>    > > 2. We can run **Sort** {% icon tool %} on the 4th column in descending order to get the regions with the highest (normalized) coverage. For `wt_H3K4me3_rep1`, the regions between 152,233,600	and 152,233,625	 are the most covered. For `wt_input_rep1`, between 143,483,000 and 143,483,100.
>    > {: .solution }
>    {: .question}
>
> 2. {% tool [bamCoverage](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_coverage/deeptools_bam_coverage/3.5.1.0.0) %} with the same parameters but
>    - *"Coverage file format"*: `bigWig`
>
>    > <question-title></question-title>
>    >
>    > What is a `bigWig` file?
>    >
>    > > <solution-title></solution-title>
>    > > A `bigWig` file is a compressed `bedgraph` file. Similar in relation as BAM to SAM, but this time just for coverage data. This means bigWig and bedgraph files are much smaller than BAM or SAM files (no sequence or quality information).
>    > {: .solution }
>    {: .question}
>
> 3. Use **IGV** {% icon tool %} to inspect both signal coverages (input and ChIP samples). Alternatively, you can use **JBrowse2** as mentioned above to load multiple tracks and look at genome intervals.
>
{: .hands_on}

> <question-title></question-title>
>
> If you zoom to `chrX:151,385,260-152,426,526`, what do you observe?
>
> ![Output of bamCoverage](../../images/formation_of_super-structures_on_xi/bamcoverage_igv.png "bamCoverage for wt_H3K4me3_rep1 and wt_input_rep1 on chrX:151,385,260-152,426,526")
>
> > <solution-title></solution-title>
> > The track with the coverage for the input (`wt_input_rep1`) seems quite homogeneous. On the other hand, for `wt_H3K4me3_rep1`, we can observe some clear peaks at the beginning of the genes.
> {: .solution }
{: .question}

## Generation of input-normalized coverage files

To extract only the information induced by the immunoprecipitation, we normalize the coverage file for the sample that underwent immunoprecipitation by the coverage file for the input sample. Here we use the tool **bamCompare** {% icon tool %} which compare 2 BAM files while caring for sequencing depth normalization.

> <hands-on-title>Generation of input-normalized coverage files</hands-on-title>
>
> 1. {% tool [bamCompare](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_compare/deeptools_bam_compare/3.5.1.0.0) %} with the following parameters:
>    - {% icon param-file %} *"First BAM file (e.g. treated sample)"*: `wt_H3K4me3_rep1.bam`
>    - {% icon param-file %} *"Second BAM file (e.g. control sample)"*: `wt_input_rep1.bam`
>    - *"Bin size in bases"*: `50`
>    - *"How to compare the two files"*: `Compute log2 of the number of reads ratio`
>    - *"Coverage file format"*: `bedgraph`
>    - *"Region of the genome to limit the operation to"*: `chrX`
>
>    > <question-title></question-title>
>    >
>    > 1. What does a positive value or a negative value mean in the 4th column?
>    > 2. Which regions have the highest coverage in the ChIP data? and the lowest?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The 4th column contains the log2 of the number of reads ratio between the ChIP-seq sample and the input sample. A positive value means that the coverage on this portion of genome is higher in the ChIP-seq sample than in the input sample
>    > > 2. The highest: 152,233,800-152,233,850 (consistent with the most covered regions in `wt_H3K4me3_rep1` given by **bamCoverage** {% icon tool %}). The lowest: 169,916,600-169,916,650
>    > {: .solution }
>    {: .question}
>
> 2. {% tool [bamCompare](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_compare/deeptools_bam_compare/3.5.1.0.0) %} with the same parameters but:
>    - *"Coverage file format"*: `bigWig`
>
> 3. Use **IGV** {% icon tool %} to inspect the log2 ratio. Or, as an alternative, you can use **JBrowse2** from the tools on Galaxy.
>
{: .hands_on}

> <question-title></question-title>
>
> How could you interpret the new track if you zoom to `chrX:151,385,260-152,426,526`?
>
> ![Output of bamCoverage](../../images/formation_of_super-structures_on_xi/bamcompare_igv.png "bamCoverage for wt_H3K4me3_rep1 and wt_input_rep1 on chrX:151,385,260-152,426,526")
>
> > <solution-title></solution-title>
> > The new track is the difference between the first track (`wt_H3K4me3_rep1`) and the second track (`wt_input_rep1`)
> {: .solution }
{: .question}

# Step 5: Detecting enriched regions (peak calling)

We could see in the ChIP data some enriched regions (peaks). We now would like to call these regions to obtain their coordinates, using **MACS2 callpeak** {% icon tool %}

> <hands-on-title>Peak calling</hands-on-title>
>
> 1. {% tool [MACS2 callpeak](toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.2.7.1+galaxy0) %} with the following parameters:
>    - *"Are you pooling Treatment Files?"*: `No`
>       - {% icon param-file %} *"ChIP-Seq Treatment File"*: `wt_H3K4me3_rep1.bam`
>    - *"Do you have a Control File?"*: `Yes`
>       - *"Are you pooling Treatment Files?"*: `No`
>           - {% icon param-file %} *"ChIP-Seq Treatment File"*: `wt_input_rep1.bam`
>    - *"Format of Input Files"*: `Paired-end BAM`
>    - *"Effective genome size"*: `M.musculus (1.87e9)`
>    - *"Outputs"*: `Summary page (html)`
>
>    > <comment-title></comment-title>
>    > The advanced options may be adjusted, depending of the samples.
>    > If your ChIP-seq experiment targets regions of broad enrichment, *e.g.* non-punctuate histone modifications, select calling of broad regions.
>    > If your sample has a low duplication rate (*e.g.* below 10%), you might keep all duplicate reads (tags). Otherwise, you might use the 'auto' option to estimate the maximal allowed number of duplicated reads per genomic location.
>    {: .comment}
>
> 2. Inspect the {% icon param-file %} `(narrow Peaks)` file (output of **MACS2 callpeak** {% icon tool %})
>
>    > <question-title></question-title>
>    >
>    > Which type of files were generated? What do they include?
>    >
>    > > <solution-title></solution-title>
>    > > **MACS2 callpeak** {% icon tool %} has generated a bed file with the coordinates of the identified peaks: chromosome, start, stop, name, integer score, strand, fold-change, -log10pvalue, -log10qvalue and relative summit position to peak start, as well as a html report which contains links to additional bed and xls files.
>    > {: .solution }
>    {: .question}
>
> 3. **IGV** {% icon tool %} (or **JBrowse2**) to inspect with the signal coverage and log2 ratio tracks
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many peaks have been identified in `chrX:151,385,260-152,426,526` based on IGV?
>
>    ![Output of MACS2](../../images/formation_of_super-structures_on_xi/macs2_igv.png "Peaks for wt_H3K4me3_rep1 and wt_input_rep1 on chrX:151,385,260-152,426,526")
>
> 2. What are the fold change of the peaks identified in `chrX:151,385,260-152,426,526`? Hint: using the BED file
> 3. How many peaks have been identified on the full chromosome X? How many peaks have a fold change > 50?
>
> > <solution-title></solution-title>
> > 1. We can see 11 peaks (track below the genes).
> > 2. Using {% tool [Filter](Filter1) %} with `c2>151385260 and c3<152426526`, we found that the 11 peaks with fold changes between 3.81927 and 162.06572
> > 4. On the 656 peaks on the full chromosome (number of lines of the original BED file) there are 252 peaks with FC>50 (using **Filter** {% icon tool %} with `c7>50`)
> {: .solution }
{: .question}

The called peak regions can be filtered by, *e.g.* fold change, FDR and region length for further downstream analysis.

# Step 6: Plot the signal between samples

So far, we have normalized the data and identified peaks. Now, we would like to visualize scores associated with certain genomic regions, for example ChIP enrichment values around the TSS of genes. Moreover, we would like to compare the enrichment of several ChIP samples (e.g. CTCF and H3K4me3) on the regions of interest.

Since we already generated the required files for the H3K4me3 sample, let's make them only for the CTCF sample:

> <hands-on-title>Prepare the peaks and data for CTCF</hands-on-title>
>
> 1. {% tool [bamCompare](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_compare/deeptools_bam_compare/3.5.1.0.0) %} with the following parameters:
>    - {% icon param-file %} *"First BAM file (e.g. treated sample)"*: `wt_CTCF_rep1.bam`
>    - {% icon param-file %} *"Second BAM file (e.g. control sample)"*: `wt_input_rep1.bam`
>    - *"Bin size in bases"*: `50`
>    - *"How to compare the two files"*: `Compute log2 of the number of reads ratio`
>    - *"Coverage file format"*: `bigwig`
>    - *"Region of the genome to limit the operation to"*: `chrX`
> 2. Rename the output of **bamCompare** {% icon tool %} with the name of the sample
> 3. {% tool [MACS2 callpeak](toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.2.7.1+galaxy0) %} with the following parameters
>    - *"Are you pooling Treatment Files?"*: `No`
>       - {% icon param-file %} *"ChIP-Seq Treatment File"*: `wt_CTCF_rep1.bam`
>    - *"Do you have a Control File?"*: `Yes`
>       - *"Are you pooling Treatment Files?"*: `No`
>           - {% icon param-file %} *"ChIP-Seq Treatment File"*: `wt_input_rep1.bam`
>    - *"Format of Input Files"*: `Paired-end BAM`
>    - *"Effective genome size"*: `M.musculus (1.87e9)`
>
{: .hands_on}

We can now concatenate the MACS2 outputs with the location of the peaks (concatenate the files and merge the overlapping regions) to obtain one BED file corresponding to the coordinates of the interesting regions to plot.

> <hands-on-title>Prepare the peak coordinates</hands-on-title>
>
> 1. {% tool [Concatenate multiple datasets or collections](cat1) %} with the following parameters:
>    - {% icon param-file %} *"Concatenate Dataset"*: output of **MACS2 callpeak** {% icon tool %} for `wt_CTCF_rep1`
>    - Click on *"Insert Dataset"*:
>       - For *"Dataset"*: select the output of **MACS2 callpeak** {% icon tool %} for `wt_H3K4me3_rep1`
> 2. {% tool [bedtools SortBED](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_sortbed/2.30.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Sort the following BED/bedGraph/GFF/VCF/EncodePeak file"*: output of **Concatenate** {% icon tool %}
> 3. {% tool [bedtools MergeBED](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_mergebed/2.30.0) %} with the following parameters:
>    - {% icon param-file %} *"Sort the following BAM/BED/bedGraph/GFF/VCF/EncodePeak file"*: output of **SortBED** {% icon tool %}
>
{: .hands_on}

To plot the the peaks score on the region generated above (MergeBED output) two tools from the [deepTools](http://deeptools.readthedocs.io/) package are used:
- **computeMatrix** {% icon tool %}: it computes the signal on given regions, using the `bigwig` coverage files from different samples.
- **plotHeatmap** {% icon tool %}: it plots heatMap of the signals using the **computeMatrix** {% icon tool %} output.

Optionally, we can also use **plotProfile** {% icon tool %} to create a profile plot from **computeMatrix** {% icon tool %} output.

> <hands-on-title>Plot the heatmap</hands-on-title>
>
> 1. {% tool [computeMatrix](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.5.1.0.0)%} with the following parameters:
>    - *"Select regions"*:
>       - {% icon param-file %} *"Regions to plot"*: output of **MergeBED** {% icon tool %}
>    - *"Sample order matters"*: `No`
>       - {% icon param-files %} *"Score file"*: the 2 `bigwig` files generated by **bamCompare** {% icon tool %} and renamed
>    - *"computeMatrix has two main output options"*: `reference-point`
>       - *"The reference point for the plotting"*: `center of region`
>       - *"Distance upstream of the start site of the regions defined in the region file"*: `3000`
>       - *"Distance downstream of the end site of the given regions"*: `3000`
> 2. {% tool [plotHeatmap](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_heatmap/deeptools_plot_heatmap/3.5.1.0.1) %} with the following parameters
>    - {% icon param-file %} *"Matrix file from the computeMatrix tool"*: `Matrix` (output of **computeMatrix** {% icon tool %})
>    - *"Show advanced options"*: `yes`
>       - *"Reference point label"*: select the right label
>       - *"Did you compute the matrix with more than one groups of regions?"*: `No, I used only one group`
>           - *"Clustering algorithm"*: `Kmeans clustering`
>           - *"Number of clusters to compute"*: `2`
{: .hands_on}

It should generate an heatmap similar to:

![Output of plotHeatmap for 2 samples](../../images/formation_of_super-structures_on_xi/peak_heatmap_1.png "Scores around the peaks for wt_H3K4me3_rep1 and wt_input_rep1")

When we look at this graph, it seems that less but larger peaks are found for `H3K4me3_rep1` and that only few peaks are shared.

> <question-title></question-title>
>
> 1. How many peaks have been found for `CTCF_rep1` and for `H3K4me3_rep1`?
> 2. What are the mean width of the peaks for `CTCF_rep1` and for `H3K4me3_rep1`?
> 3. How many peaks are specific to `CTCF_rep1` or `H3K4me3_rep1`?
>
> > <solution-title></solution-title>
> > 1. 656 peaks for `H3K4me3_rep1` and 2,688 for `CTCF_rep1` (number of lines in the **MACS2 callpeak** {% icon tool %} BED file)
> > 2. 1630.77 bp for `H3K4me3_rep1` and 404.55 for `CTCF_rep1` (**Compute** {% icon tool %} with `c3-c2` and then **Datamash** {% icon tool %} with `Mean` on `Column:11`)
> > 3. 443 peaks (over 656) are specific to `H3K4me3_rep1` and 2,464 (over 2,688) to `CTCF_rep1` (**Intersect intervals** {% icon tool %}). Around 220 peaks are then overlapping.
> {: .solution }
{: .question}

So far, we have only analyzed 2 samples, but we can do the same for all the 6 samples:

> <hands-on-title>(Optional) Plot the heatmap for all the samples</hands-on-title>
>
> 1. {% tool [bamCompare](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_compare/deeptools_bam_compare/3.5.1.0.0) %} for each combination input - ChIP data:
>     1. `wt_CTCF_rep1` - `wt_input_rep1` (already done)
>     2. `wt_H3K4me3_rep1` - `wt_input_rep1` (already done)
>     3. `wt_H3K27me3_rep1` - `wt_input_rep1`
>     4. `wt_CTCF_rep2` - `wt_input_rep2`
>     5. `wt_H3K4me3_rep2` - `wt_input_rep2`
>     6. `wt_H3K27me3_rep2` - `wt_input_rep2`
> 2. Rename the outputs of **bamCompare** {% icon tool %} with the name of the ChIP data
> 3. {% tool [MACS2 callpeak](toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.2.7.1+galaxy0) %} for each combination input - ChIP data
> 4. {% tool [Concatenate multiple datasets or collections](cat1) %} with the following parameters:
>    - {% icon param-file %} *"Concatenate Dataset"*: one output of **MACS2 callpeak** {% icon tool %}
>    - Click on *"Insert Dataset"*:
>       - In *"Select"*: one other output of **MACS2 callpeak** {% icon tool %}
>    - Redo for the 6 outputs of **MACS2 callpeak** {% icon tool %}
> 5. {% tool [bedtools SortBED](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_sortbed/2.30.0+galaxy2) %} with the following parameters
>    - {% icon param-file %} *"Sort the following bed,bedgraph,gff,vcf file"*: output of **Concatenate** {% icon tool %}
> 6. {% tool [bedtools MergeBED](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_mergebed/2.30.0) %} with the following parameters
>    - {% icon param-file %} *"Sort the following bed,bedgraph,gff,vcf file"*: output of **SortBED** {% icon tool %}
> 7. {% tool [computeMatrix](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.5.1.0.0)%} with the same parameters but:
>    - *"Select regions"*:
>       - {% icon param-file %} *"Regions to plot"*: output of **MergeBED** {% icon tool %}
>    - *"Sample order matters"*: `No`
>       - {% icon param-files %} *"Score file"*: the 6 `bigwig` files generated by **bamCompare** {% icon tool %} and renamed
>    - *"computeMatrix has two main output options"*: `reference-point`
>       - *"The reference point for the plotting"*: `center of region`
>       - *"Distance upstream of the start site of the regions defined in the region file"*: `3000`
>       - *"Distance downstream of the end site of the given regions"*: `3000`
> 8. {% tool [plotHeatmap](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_heatmap/deeptools_plot_heatmap/3.5.1.0.1) %} with the following parameters
>    - {% icon param-file %} *"Matrix file from the computeMatrix tool"*: `Matrix` (output of **computeMatrix** {% icon tool %})
>    - *"Show advanced options"*: `yes`
>       - *"Reference point label"*: select the right label
>       - *"Did you compute the matrix with more than one groups of regions?"*: `No, I used only one group`
>           - *"Clustering algorithm"*: `Kmeans clustering`
>           - *"Number of clusters to compute"*: `6`
{: .hands_on}

> <question-title></question-title>
>
> 1. How many peaks are found for the different samples?
> 2. How are the peaks?
>
>    ![Output of plotHeatmap for all samples](../../images/formation_of_super-structures_on_xi/peak_heatmap_all.png "Scores around the peaks for all samples")
>
> 3. How could be interpreted the peaks and read coverage in the `chrX:151,385,260-152,426,526` region?
>
>    ![Coverage and peaks for the replicate 1](../../images/formation_of_super-structures_on_xi/rep1_igv.png "Coverage and peaks for the replicate 1")
>
> > <solution-title></solution-title>
> > 1. Found peaks (number of lines in **MACS2 callpeak** {% icon tool %} outputs):
> >
> >     Target | Rep 1 | Rep 2
> >     --- | --- | ---
> >     CTCF | 2,672 | 2,061
> >     H3K4me3 | 657 | 718
> >     H3K27me3 | 220 | 73
> >
> >    The tendencies are similar for both replicates: more peaks for CTCF, less for H3K4me3 and only few for H3K27me3.
> >
> > 2. As observed with the 2 samples, the peaks for H3K4me3 are wider than for CTCF. We also observe that the peaks found with one replicate are found with the other replicate.
> > 3. The H3K4me3 sample has clear and large regions in which the read coverage are enriched. H3K4me3 is one of the least abundant histone modifications. It is highly enriched at active promoters near transcription start sites (TSS) and positively correlated with transcription.
> >
> >    For H3K27me3, the coverage is more homogeneous. A gene is a broad domain of H3K27me3 enrichment across its body of genes corresponds to a gene with a transcription inhibited by H3K27me3. We can also identified some "bivalent" genes: gene with a peak around the TSS for H3K27me3 (e.g. region_208 for gene Gpr173) but also H3K4me3. We also observe some H3K27me3-depleted regions sharply demarcated, with boundaries coinciding with gene borders (e.g. Kdm5c). This is a chromatin signature reminiscent of genes that escape XCI.
> >
> >    To reproduce, run **bamCoverage** {% icon tool %}, **IGV** {% icon tool %} and **MACS2 callpeak** {% icon tool %} outputs.
> {: .solution }
{: .question}

# Conclusion


Along this tutorial, we learn how to extract peaks and coverage information from raw data of ChIP experiments:

![Workflow to extract peaks and coverage information from raw data of ChIP experiments](../../images/formation_of_super-structures_on_xi/tutorial-scheme.png "Workflow to extract peaks and coverage information from raw data of ChIP experiments")

This information can be then related to the biology to answer the original question. We tried then to relate the observed differences of peak and read coverage between H3K27me3 and H3K4me3 to the known biology. We could go even further in the analysis to reproduce the results of the original paper (e.g. by looking at the bivalent genes, identifying the differences between Xa and Xi).

tools_json:
["Flatten collection", "__FLATTEN__", "FastQC", "toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy1", "Trim Galore!", "toolshed.g2.bx.psu.edu/repos/bgruening/trim_galore/trim_galore/0.6.7+galaxy0", "Bowtie2", "toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.4+galaxy0", "BAM BED GFF coverage bigWigs", "toolshed.g2.bx.psu.edu/repos/iuc/bbgbigwig/bbgtobigwig/0.1", "JBrowse2", "toolshed.g2.bx.psu.edu/repos/fubar/jbrowse2/jbrowse2/2.17.0+galaxy0", "multiBamSummary", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_multi_bam_summary/deeptools_multi_bam_summary/3.5.1.0.0", "plotCorrelation", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_correlation/deeptools_plot_correlation/3.5.1.0.0", "plotFingerprint", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_fingerprint/deeptools_plot_fingerprint/3.5.1.0.0", "Samtools idxstats", "toolshed.g2.bx.psu.edu/repos/devteam/samtools_idxstats/samtools_idxstats/2.0.4", "bamCoverage", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_coverage/deeptools_bam_coverage/3.5.1.0.0", "bamCompare", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_bam_compare/deeptools_bam_compare/3.5.1.0.0", "MACS2 callpeak", "toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.2.7.1+galaxy0", "Filter", "Filter1", "Concatenate multiple datasets or collections", "cat1", "bedtools SortBED", "toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_sortbed/2.30.0+galaxy2", "bedtools MergeBED", "toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_mergebed/2.30.0", "computeMatrix", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.5.1.0.0", "plotHeatmap", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_heatmap/deeptools_plot_heatmap/3.5.1.0.1"]


topic: epigenetics
tutorial: cut_and_run
title: CUT&RUN data analysis
difficulty: intermediate
time_estimation: 3h
requirements_json: []
tags_json: []
content_hash: 802607e67d97dc26ea8536870a837a89
last_modified: 2026-05-27T09:01:48.464558
zenodo_link: https://zenodo.org/record/6823059
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/epigenetics/tutorials/cut_and_run/tutorial.html

content:
In many organism, {TF} play an important tole in the regulation of the gene expression. In human, we have up to 2,800 proteins and more than 1,600 are
TF ([list of transcription factors](https://en.wikipedia.org/wiki/List_of_human_transcription_factors)), although the number might change over timer.
Investigating the role of TFs, such as [GATA1](https://en.wikipedia.org/wiki/GATA1), is a very important task to understand the regulatory mechanisms in the cell and thus ascertain
the source of a disease, such as [myelofibrosis](https://en.wikipedia.org/wiki/Primary_myelofibrosis) a type of blood cancer.

![CUT_and_RUN](../../images/cut_and_run/cut_and_run.png "Skene and Henikoff 2017 eLIFE")

{CUT&RUN} {% cite Skene2017 %} became the new and advanced method to analyse DNA-associated proteins. CUT&RUN uses an antibody just as [ChIP-Seq](https://en.wikipedia.org/wiki/ChIP_sequencing) to select the {POI}.
The big difference, CUT&RUN couples the usage of an antibody with the usage of a pA-MNase, which you can see in **Figure 1**. In the revised version of the protocol {% cite Meers2019 %},
the pA-MNase has been replaced by a {pA(G)-MNase} to expand this technology to mouse primary antibodies. The [enzyme](https://en.wikipedia.org/wiki/Micrococcal_nuclease) fused to the protein A/G is an
endo-exonuclease that cleaves and shortens DNA. The fusion will ensure that the MNase comes in close proximity with the POI and thus cleaves DNA *in-situ*, close to POI binding sites. CUT&RUN allows to
fragment the DNA in unfixed cells and thus allows to study protein-DNA interactions in a more natural state. The added pA(G)-MNase thus creates shorted fragments that lead to a higher resolution for the mapping
in comparison to your standard ChIP-Seq protocol. CUT&RUN follows four fundamental steps: (1) permeabilize the nuclei or cells coated on beads, (2) incubate with the selective antibody of the POI, (3)
add the pA(G)-MNase and activate the enzyme for a small period of time, (4) release CUT&RUN fragments and collect the DNA from the supernatant. Afterward, the DNA can be purified, PCR amplified and
prepared for sequencing. You can find the CUT&RUN protocol at [protocols.io](https://www.protocols.io/view/cut-amp-run-targeted-in-situ-genome-wide-profiling-14egnr4ql5dy/v3).

In this tutorial, we will use data from the study of {% cite Zhu2019 %}. The article introduces a CUT&RUN pipeline that we will **not** completely follow. It is important to note at this point that
a CUT&RUN data analysis is more similar to an ATAC-Seq experiment than a standard ChIP-Seq. We will analyze the two biological replicates from a CUT&RUN experiment for the aforementioned TF GATA1 in humans.
We downsampled the data to speed up the run times in this tutorial. Our results will be compared to identified binding sites of GATA1 of a ChIP-Seq experiment {% cite Canver2017 %}.

## When working with real data
The workflow for this training material can be found [at the European Galaxy instance](https://usegalaxy.eu/u/heylf/w/copy-of-cutandrunlong). When you use your data we suggest using [this workflow](https://usegalaxy.eu/u/heylf/w/cutandrunlong)
which includes additional steps for your data analysis. Both workflows do not support peak calling with controls as CUT&RUN has a low background.
It is often recommended to use a positive or negative control as a comparison. Spike-in controls can be done for CUT&RUN but need then additional steps in the provided workflows to consider them.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet faqs/galaxy/analysis_results_may_vary.md %}

# Step 1: Preprocessing

## Get Data

We first need to download the sequenced reads (FASTQs) as well as other annotation files. Then, to increase the number of reads that will map to the reference genome (here human genome version 38, GRCh38/hg38), we need to preprocess the reads.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo](https://doi.org/10.5281/zenodo.6823059) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/record/6823059/files/chip_seq_peaks.bed
>    https://zenodo.org/record/6823059/files/Rep1_R1.fastq
>    https://zenodo.org/record/6823059/files/Rep1_R2.fastq
>    https://zenodo.org/record/6823059/files/Rep2_R1.fastq
>    https://zenodo.org/record/6823059/files/Rep2_R2.fastq
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 4. Add a tag called `#rep1` to the Rep1 files and a tag called `#rep2` to the Rep2 files.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
> 5. Check that the datatype of the 4 FASTQ files is `fastqsanger` and the peak file (chip_seq_peaks.bed) is `bed`. If they are not then change the datatype as described below.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 6. Create a paired collection named `2 PE fastqs`, and rename your pairs with the sample name followed by the attributes: `Rep1` and `Rep2`.
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
> 7. Rename the bed file `GATA1 ChIP-Seq peaks`.
>
{: .hands_on}

> <comment-title>FASTQ format</comment-title>
> If you are not familiar with FASTQ format, see the [Quality Control tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %})
{: .comment}
>
> <comment-title>BED / encode narrowPeak format</comment-title>
> If you are not familiar with BED format or encode narrowPeak format, see the [BED Format](https://genome.ucsc.edu/FAQ/FAQformat.html)
{: .comment}

## Quality Control

We first have to check if our data contains adapter sequences that we have to remove. A typical CUT&RUN experiment has a read length of 30-80 nt. We can check the raw data quality with **Falco**.

> <hands-on-title>Quality Control</hands-on-title>
>
> 1. {% tool [Flatten collection](__FLATTEN__) %} with the following parameters convert the list of pairs into a simple list:
>     - *"Input Collection"*: `2 PE fastqs`
>
> 2. {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} with the following parameters:
>       - {% icon param-collection %} *"Raw read data from your current history"*: Choose the output of **Flatten collection** {% icon tool %} selected as **Dataset collection**.
> 3. Inspect the web page output of **Falco** {% icon tool %} for the `Rep1_forward` sample. Check what adapters are found at the end of the reads.
>
>    > <question-title></question-title>
>    >
>    > 1. How many reads are in the FASTQ?
>    > 2. Which sections have a warning or an error?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are 300,000 reads.
>    > > 2. The 4 steps below have warnings:
>    > >
>    > >    1. **Per base sequence content**
>    > >
>    > >       CUT&RUN has sometimes base biases like ChIP-Seq.
>    > >
>    > >    2. **Sequence Duplication Levels**
>    > >
>    > >       The read library quite often has PCR duplicates that are introduced
>    > >       simply by the PCR itself. We will remove these duplicates later on.
>    > >
>    > >    3. **Overrepresented sequences**
>    > >
>    > >       Our data contains TruSeq adapter, Illumina PCR Primer, and a read from the mitochondrial chromosome.
>    > >
>    > >    4. **Adapter Content**
>    > >
>    > >       Our data contains an adapter (Illumina Universal Adapter) that we still have to remove.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
>    As it is tedious to inspect all these reports individually we will combine them with {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3) %}.
>
> 4. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3) %} to aggregate the FastQC reports with the following parameters:
>    - In *"Results"*:
>        - *"Results"*
>            - *"Which tool was used generate logs?"*: `FastQC`
>                - In *"FastQC output"*:
>                    - {% icon param-repeat %} *"Insert FastQC output"*
>                        - {% icon param-collection %} *"FastQC output"*: `Falco on collection N: Raw data` (output of **Falco** {% icon tool %})
>
{: .hands_on}

> <comment-title>Falco Results</comment-title>
> This is what you should expect from the **Adapter Content** section of multiQC:
> ![MultiQC screenshot of the Adapter Content section](../../images/cut_and_run/fastqc_adapter_content_plot_pre.png "MultiQC screenshot on the Adapter Content section")
{: .comment}

The MultiQC report (of FastQC) pointed out that we have in our data some standard Illumina adapter sequences, which we will remove with Trim Galore!.

## Trimming Reads

**Trim Galore!** is a handy tool that can automatically detect and trim standard Illumina adapters.

> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [Trim Galore!](toolshed.g2.bx.psu.edu/repos/bgruening/trim_galore/trim_galore/0.6.7+galaxy1) %} with the following parameters:
>    - *"Is this library paired- or single-end?"*: `Paired Collection`
>        - *"Select a paired collection"*: `2 PE fastqs`
>    - In *"Adapter sequence to be trimmed"*: `Illumina universal`
>    - *Avanced settings*: `Full parameter list`
>    - In *"Trim low-quality ends from reads in addition to adapter removal (Enter phred quality score threshold)"*: `30`
>    - In *"Discard reads that became shorter than length N"*: `15`
>    - In *"Generate a report file"*:   `Yes`
>
> 2. Click on the {% icon galaxy-eye %} (eye) icon of the report for `Rep1` and look for the info.
{: .hands_on}

> <question-title></question-title>
>
> 1. What percentage of reads contain adapters?
> 2. What percentage of reads are still longer than 15bp after the trimming?
>
> > <solution-title></solution-title>
> >
> > 1. 163,656 (54.6%) for Rep 1 and 169,776 (56.6%) for Rep 2
> > 2. The last line indicates that 10477 (3.49%) of pairs have been removed from Rep1 and 15775 (5.26%) pairs from Rep2.
> >
> {: .solution}
>
{: .question}

# Step 2: Mapping

## Mapping Reads to Reference Genome

Next, we map the trimmed reads to the human reference genome. Here we will use **Bowtie2**. We will extend the maximum fragment length (distance between read pairs) from 500 to 1000 because
we know some valid read pairs are from this fragment length. We will use the `--very-sensitive` parameter to have more chances to get the best match even if it takes a bit longer to run.
We will run the **end-to-end** mode because we trimmed the adapters, so we expect the whole read to map, no clipping of ends is needed. Regarding the genome to choose.
The hg38 version of the human genome contains [alternate loci](https://www.ncbi.nlm.nih.gov/grc/help/definitions/#ALTERNATE). This means that some regions of the genome
are present both in the canonical chromosome and on its alternate loci. The reads that map to these regions would map twice. To be able to filter reads falling into
repetitive regions but keep reads falling into regions present in alternate loci, we will map on the Canonical version of hg38 (only the chromosome with numbers, chrX, chrY, and chrM).

> <hands-on-title>Mapping reads to reference genome</hands-on-title>
>
> 1. {% tool [Bowtie2](toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.3+galaxy1) %} with the following parameters:
>    - *"Is this single or paired library"*: `Paired-end Dataset Collection`
>        - *"FASTQ Paired Dataset*: select the output of **Trim Galore!** {% icon tool %} *"paired reads"*
>        - *"Do you want to set paired-end options?"*: `Yes`
>            - *"Set the maximum fragment length for valid paired-end alignments"*: `1000`
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>        - *"Select reference genome"*: `Human (Homo sapiens): hg38 Canonical`
>    - *"Set read groups information?"*: `Do not set`
>    - *"Select analysis mode"*: `1: Default setting only`
>        - *"Do you want to use presets?"*: `Very sensitive end-to-end (--very-sensitive)`
>    - *"Do you want to tweak SAM/BAM Options?"*: `No`
>    - *"Save the bowtie2 mapping statistics to the history"*: `Yes`
>
> 2. Click on the {% icon galaxy-eye %} (eye) icon of the mapping stats `Rep1`.
{: .hands_on}

> <comment-title>Bowtie2 Results</comment-title>
> You should get similar results to this from Bowtie2:
> ![Mapping statistics of bowtie2](../../images/cut_and_run/mapping_stats.png "Mapping statistics of bowtie2")
{: .comment}

> <question-title></question-title>
>
> What percentage of read pairs from Rep1 mapped concordantly?
>
> > <solution-title></solution-title>
> >
> > 36.47+62.39=98.86%
> >
> {: .solution}
>
{: .question}

# Step 3: Filtering Mapped Reads and second level Quality Control

## Plot Enrichment signal

Before we apply any filters on our mapped data, let us check the cumulative enrichment of signal using **plotFingerprint** {% icon tool %} from the **deepTools** suite.

> <hands-on-title>Assess Signal CUT&RUN Enrichment using plotFingerprint</hands-on-title>
>
> 1. {% tool [plotFingerprint](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_fingerprint/deeptools_plot_fingerprint/3.5.4+galaxy0) %} with the following parameters:
>    - *"Sample order matters"*: `No`
>    - {% icon param-collection %} *"BAM/CRAM file"*: Select the output of  **Bowtie2** {% icon tool %} *"alignments"*
>
>
> 2. Check the plot.
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Do you see a high or low enrichment?
> 2. Do you expect such enrichment signal?
>
> > <solution-title></solution-title>
> >
> > ![plotFingerprint](../../images/cut_and_run/fingerprint.png "Signal Enrichment")
> >
> > 1. Both the replicates have almost identical enrichment profile indicating reproducibility. The sharp peak at the end indicates that only a few genomic regions (putative binding sites) were covered with most of the reads which is a very strong enrichment.
> > 2. There are mainly two reasons to expect such a high enrichment:
> >     1. Unlike ChIP-seq, CUT&RUN has low background noise.
> >     2. Our protein of interest (GATA1) in this data set is a transcription factor (TF). TFs usually show high binding specificity.
> >
> {: .solution}
>
{: .question}

## Filter Uninformative Reads

We apply some filters to the reads after the mapping. CUT&RUN datasets can have many reads that map to the mitochondrial genome because it is nucleosome-free and thus very accessible
to the pA(G)-MNase. CUT&RUN inserts adapter more easily in open chromatin regions due to the pA(G)-MNase activity. The mitochondrial genome is uninteresting for CUT&RUN,
so we remove these reads. We also remove reads with low mapping quality and reads that are not properly paired.

> <hands-on-title>Filtering of uninformative reads</hands-on-title>
>
> 1. {% tool [Filter BAM datasets on a variety of attributes](toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.5.2+galaxy2) %} with the following parameters:
>    - {% icon param-collection %} *"BAM dataset(s) to filter"*: Select the output of  **Bowtie2** {% icon tool %} *"alignments"*
>    - In *"Condition"*:
>        - {% icon param-repeat %} *"Condition"*
>            - In *"Filter"*:
>                - *"Filter"*
>                    - *"Select BAM property to filter on"*: `Mapping quality`
>                        - *"Filter on read mapping quality (phred scale)"*: `>=30`
>                - {% icon param-repeat %} *"Insert Filter"*
>                    - *"Select BAM property to filter on"*: `Proper Pair`
>                        - *"Select properly paired reads"*: `Yes`
>                - {% icon param-repeat %} *"Insert Filter"*
>                    - *"Select BAM property to filter on"*: `Reference name of the read`
>                        - *"Filter on the reference name for the read"*: `!chrM`
>    - *"Would you like to set rules?"*: `False`
>
>
> 2. Click on the input and the output BAM files of the filtering step. Check the size of the files.
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Based on the file size, what proportion of alignments was removed (approximately)?
> 2. Which parameter should be modified if you are interested in repetitive regions?
>
> > <solution-title></solution-title>
> >
> > 1. The original BAM files are 14-15 MB, the filtered ones are 5.8 and 7.1 MB. Approximately half of the alignments were removed.
> >
> > 2. You should modify the mapQuality criteria and decrease the threshold.
> >
> {: .solution}
>
{: .question}

## Filter Duplicate Reads

Because of the PCR amplification, there might be read duplicates (different reads mapping to the same genomic region) from over amplification of some regions. We will remove them with **Picard MarkDuplicates**.

> <hands-on-title>Remove duplicates</hands-on-title>
>
> 1. {% tool [MarkDuplicates](toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_MarkDuplicates/3.1.1.0) %} with the following parameters:
>    - {% icon param-collection %} *"Select SAM/BAM dataset or dataset collection"*: Select the output of  **Filter** {% icon tool %} *"BAM"*
>    - *"If true do not write duplicates to the output file instead of writing them with appropriate flags set"*: `Yes`
>
>    > <comment-title>Default of  **MarkDuplicates** {% icon tool %}</comment-title>
>    >
>    > By default, the tool will only "Mark" the duplicates. This means that it will change the Flag of the duplicated reads to enable them to be filtered afterwards. We use the parameter *"If true do not write duplicates to the output file instead of writing them with appropriate flags set"* to directly remove the duplicates.
>    {: .comment}
>
> 2. Click on the {% icon galaxy-eye %} (eye) icon of the MarkDuplicate metrics.
{: .hands_on}

> <comment-title>MarkDuplicates Results</comment-title>
> You should get a similar output to this from MarkDuplicates:
> ![Metrics of MarkDuplicates](../../images/cut_and_run/mark_duplicates_screen.png "Metrics of MarkDuplicates")
{: .comment}

> <tip-title>Formatting the MarkDuplicate metrics for readability</tip-title>
>
> 1. {% tool [Select lines that match an expression](Grep1) %} with the following parameters:
>    - {% icon param-collection %} *"Select lines from"*: Select The *tabular* output of  **MarkDuplicates** {% icon tool %}
>    - *"that*: `Matching`
>    - *"the pattern*: `(Library|LIBRARY)`
> 2. Check that the datatype is tabular. If not, change it.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 3. {% tool  [Transpose rows/columns in a tabular file](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.1.0) %}:
>    - {% icon param-collection %} *"Select lines from"*: Select the output of **Select** {% icon tool %}
>
> ![Metrics of MarkDuplicates](../../images/cut_and_run/mark_duplicates_transpose.png "Metrics of MarkDuplicates")
>
{: .tip}

> <question-title></question-title>
>
> 1. How many pairs were in the input?
> 2. How many pairs are duplicates?
>
> > <solution-title></solution-title>
> >
> > 1. 81460 for Rep1 and 100507 for Rep2
> > 2. 982 for Rep1 and 1042 for Rep2
> >
> {: .solution}
>
{: .question}

Once again, if you have a high number of duplicates it does not mean that your data are not good, it just means that you sequenced
too much compared to the diversity of the library you generated. Consequently, libraries with a high portion of duplicates should not be re-sequenced as this would not increase the amount of data.

## Check Deduplication and Adapter Removal

> <hands-on-title>Check Adapter Removal with Falco</hands-on-title>
>
> 1. {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} with the following parameters:
>       - {% icon param-collection %} *"Raw read data from your current history"*: select the output of **MarkDuplicates** BAM.
>
> 2. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3) %} to aggregate the FastQC reports with the following parameters:
>    - In *"Results"*:
>        - *"Results"*
>            - *"Which tool was used generate logs?"*: `FastQC`
>                - In *"FastQC output"*:
>                    - {% icon param-repeat %} *"Insert FastQC output"*
>                        - {% icon param-collection %} *"FastQC output"*: `Falco on collection N: Raw data` (output of **Falco** {% icon tool %})
>
{: .hands_on}

> <comment-title>FastQC Results</comment-title>
> Now, you should see under **Overrepresented sequences** that there is no more overrepresented sequences and under **Adapter Content** that the Illumina adapters are no longer present.
> ![MultiQC screenshot on the adapter content section after Trim Galore!](../../images/cut_and_run/fastqc_adapter_content_plot_post.png "MultiQC screenshot on the adapter content section after Trim Galore!")
> However, you may have noticed that you have a new section with warning: **Sequence Length Distribution**. This is expected as you trimmed part of the reads.
> Furthermore, you will notice under **Sequence Duplication Levels** that we removed nearly all PCR duplicates.
> ![MultiQC screenshot on the duplication level after MarkDuplicates](../../images/cut_and_run/fastqc_sequence_duplication_levels_plot.png "MultiQC screenshot on the duplication level after MarkDuplicates")
{: .comment}

## Check Insert Sizes

We will check the insert sizes with **Paired-end histogram** of insert size frequency. The insert size is the distance between the R1 and R2 read pairs. This tells us the size of the DNA fragment the read pairs came from. The fragment length distribution of a sample gives an excellent indication of the quality of the CUT and RUN experiment.

> <hands-on-title>Plot the distribution of fragment sizes</hands-on-title>
>
> 1. {% tool [Paired-end histogram](toolshed.g2.bx.psu.edu/repos/iuc/pe_histogram/pe_histogram/1.0.2) %} with the following parameters:
>    - {% icon param-collection %} *"BAM file"*: Select the output of  **MarkDuplicates** {% icon tool %} *"BAM output"*
>    - *"Lower bp limit (optional)"*: `0`
>    - *"Upper bp limit (optional)"*: `1000`
>
> 2. Click on the {% icon galaxy-eye %} (eye) icon of the lower one of the 2 outputs (the png file).
{: .hands_on}

> <comment-title>Paired-end histogram Results</comment-title>
> This is what you get from Paired-end histogram:
> ![Fragment size distribution](../../images/cut_and_run/insert_size.png "Fragment size distribution")
{: .comment}

> <question-title></question-title>
>
> Could you guess what the peaks at approximately 30bp correspond to?
>
> > <solution-title></solution-title>
> >
> > The first peak (30bp) corresponds to where the cleavage around our POI. Nevertheless, the cutting of our pA(G)-MNase-antibody-protein complex is not perfect. The arms can actually cut further away from the protein. Little bumps in the distribution are an indication of another protein(complex), such as a nucleosome that is located between the two cleavage sites.
> >
> {: .solution}
>
{: .question}

This fragment size distribution is a good indication if your experiment worked or not. The CUT&RUN yield can gradually increase over the digestion time {% cite Skene2017 %}, which of course depends on the enzyme used for the cleavage of the DNA.

![CUT and RUN yield](../../images/cut_and_run/cut_and_run_yield.png "Fragment length distributions of a CUT&RUN experiment with uniform digestion over time. No anti-FLAG primary antibody (No Ab). The track with No_Ab is the control experiment. Data is from two biological replicates. The studied protein showed fragments of size ≤120 bp corresponding to the binding sites, as seen on the left, whereas reads with ≥150 bp fragment size are randomly distributed in the local vicinity. The yield in the example increases from 5s to 10 minutes, which can be seen in the mapped profiles in the right. (Skene and Henikoff 2017 eLife)")

# Step 4: Peak calling

## Call Peaks

We have now finished the data preprocessing. Next, to find regions corresponding to potential binding sites, we want to identify regions where reads have piled up (peaks) greater than the background read coverage. [MACS2](https://github.com/taoliu/MACS) is widely used for CUT&RUN.

At this step, two approaches exist: Like in ATAC-Seq, we have to take a few things into account. The pA(G)-MNase does not need to bind the protein it can create fragments unspecifically a
few bases downstream and upstream away from the true binding site. This is to the *"long"* arms of the proteins. A big jump can happen if a nucleosome is between the protein binding site and the cutting site.

It is critical to re-center the signal since the pA(G)-MNase cuts in different distances. Since some read pairs are not symmetric (i.e., the first mate might be further away from the
binding site than the second mate) we will turn off the shifting mode of MACS2 and apply a direct shift. If we only assess the coverage of the 5' extremity of the reads, the data would be too sparse,
and it would be impossible to call peaks. Thus, we will extend the start sites of the reads by 200bp (100bp in each direction) to assess coverage.

### Using MACS2

We convert the BAM file to BED format because when we set the extension size in MACS2, it will only consider one read of the pair while here we would like to use the information from both.

> <hands-on-title>Convert the BAM to BED</hands-on-title>
>
> 1. {% tool [bedtools BAM to BED converter](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_bamtobed/2.31.1+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Convert the following BAM file to BED"*: Select the output of **MarkDuplicates** {% icon tool %} BAM output
>
{: .hands_on}

We call peaks with MACS2. To get the coverage centered on the 5' extended 100bp each side we will use `--shift -100` and `--extend 200`:
![MACS2 options to get 100bp each side](../../images/atac-seq/macs2Options.jpg "MACS2 options to get 100bp each side")

> <hands-on-title>Call peaks with MACS2</hands-on-title>
>
> 1. {% tool [MACS2 callpeak](toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.2.9.1+galaxy0) %} with the following parameters:
>    - *"Are you pooling Treatment Files?"*: `No`
>        - {% icon param-collection %} Select the output of **bedtools BAM to BED** converter {% icon tool %}
>    - *"Do you have a Control File?"*: `No`
>    - *"Format of Input Files"*: `Single-end BED`
>    - *"Effective genome size"*: `H. sapiens (2.7e9)`
>    - *"Build Model"*: `Do not build the shifting model (--nomodel)`
>        - *"Set extension size"*: `200`
>        - *"Set shift size"*: `-100`. It needs to be - half the extension size to be centered on the 5'.
>    - *"Additional Outputs"*:
>        - Check `Peaks as tabular file (compatible with MultiQC)`
>        - Check `Peak summits`
>        - Check `Scores in bedGraph files`
>    - In *"Advanced Options"*:
>        - *"Composite broad regions"*: `No broad regions`
>            - *"Use a more sophisticated signal processing approach to find subpeak summits in each enriched peak region"*: `Yes`
>        - *"How many duplicate tags at the exact same location are allowed?"*: `all`
>
>    > <comment-title>Why keeping all duplicates is important</comment-title>
>    >
>    > We previously removed duplicates using **MarkDuplicates** {% icon tool %} using paired-end information. If two pairs had identical R1 but different R2, we knew it was not a PCR duplicate. Because we converted the BAM to BED we lost the pair information. If we keep the default (removing duplicates) one of the 2 identical R1 would be filtered out as duplicate.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many peaks have been identified in each replicate?
>
> > <solution-title></solution-title>
> >
> > 1. 6314 for Rep1 and 7678 for Rep2
> >
> {: .solution}
>
{: .question}

# Step 5: Identifying Binding Motifs

## Prepare the Datasets


### Extracting individual replicates from your dataset collection

In the following step, we want to apply a tool to identify robust peaks between our two replicates. However, we still have one dataset collection, holding both replicates, which we have to split.

> <hands-on-title>Extract replicates from a dataset collection:</hands-on-title>
>
> 1. {% tool [Extract Dataset](__EXTRACT_DATASET__) %} with:
>    - {% icon param-collection %} *"Input List"*: `MACS2 callpeak on collection N (narrow Peaks)`
>    - *"How should a dataset be selected?"*: `Select by index`
>    - *"Element index:"*: `0`
>
>    You can use the arrow to rerun the same tool just changing the last parameter:
>
> 2. {% tool [Extract Dataset](__EXTRACT_DATASET__) %} with:
>    - {% icon param-collection %} *"Input List"*: `MACS2 callpeak on collection N (narrow Peaks)`
>    - *"How should a dataset be selected?"*: `Select by index`
>    - *"Element index:"*: `1`
{: .hands_on}

### Extract Robust Peaks
MACS helped us to identify potential binding sites for GATA1. Yet, our peak set will contain false positives because of technical and biological noise.
We can first try to reduce some noise using the results for our two biological replicates. Since we applied MACS we have created two peak files.
Yet, both replicates have peaks that are only present in one replicate and thus might be a potential false positive due to noise.
We can remove such peaks if we simply overlap the two peak files and consequently search for peaks that appear in both replicates.

> <hands-on-title>Select robust GATA1 peaks:</hands-on-title>
>
> 1. {% tool [bedtools Intersect intervals find overlapping intervals in various ways](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.31.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"File A to intersect with B"*: Select `Rep1`
>    - *"Combined or separate output files"*: `One output file per 'input B' file`
>        - {% icon param-file %} *"File B to intersect with A"*:  Select `Rep2`
>    - *"Calculation based on strandedness?"*: `Overlaps occurring on either strand`
>    - *"Required overlap"*: `Specify minimum overlap(s)`
>        - *"Minimum overlap required as a fraction of A"*: `0.5`
>        - *"Require that the fraction of overlap be reciprocal for A and B"*: `Yes`
>    - *"Write the original A entry _once_ if _any_ overlaps found in B.": `Yes`
>
> 2. Rename the datasets `Robust GATA1 CUT and RUN peaks`.
>
> > <question-title></question-title>
> >
> > 1. How many potential true positives do we obtain?
> > 2. How many potential false positives have we removed?
> > 3. Why have we set specified an overlap fraction of A with `0.5`?
> > 4. Why have we set **"Require that the fraction of overlap be reciprocal for A and B"**?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. 2,865 peaks
> > > 2. We started with 6,314 and 7,678 peaks. Thus, we could remove 3,449 and 4,813 peaks.
> > > 3. Finding robust peaks is a tricky process and not as simple as you think. Our approach defines a robust peak as an entity that is covered by two similar peaks in either one of the peak files of our two replicates. Our measurement for similarity is simply the amount of bases the peaks overlap. Yet, ask yourself, how much overlap do you need to state that they are similar? Is 1 base enough or maybe 10? The answer is not true or false and like so often needs more investigation. We define the overlap by a 50% fraction of each of the peak files. That means, if our peak in A is 100 bases then A has to overlap with 50 bases with the peak in file B.
> > > 4. **"Require that the fraction of overlap be reciprocal for A and B"** means that 50% of the peak in file A overlaps with the peak in file B and also 50% of the peak in file B overlaps with the peak in file A. We set this to make our filtering very conservative and filter for peaks that we can be sure are true positives. For example, A has a 10 bp peak that overlaps a peak in B with 100 bp. **Viewpoint of file A:** An overlap-fraction of 0.5 for A means that we accept peak A if it overlaps with 5 bp with the peak in B. This is very likely because the peak in B is 100 bp long. **Viewpoint of file B**: An overlap-fraction of 0.5 for B means that we reject A no matter what, even if A completely overlaps B because 10 bp are not enough to cover 50% of B. That is to say the option **"Require that the fraction of overlap be reciprocal for A and B"** takes care that we trust peaks that have in file A **and** in file B a fair overlap.
> > {: .solution}
> >
> {: .question}
{: .hands_on}

Note, that this is one way to define robust peaks. Yet, there are more sophisticated algorithms in Galaxy, such as
{% tool [IDR compare ranked list of identifications](toolshed.g2.bx.psu.edu/repos/iuc/idr/idr/2.0.3) %}, and other methods that can help to define a robust peak.
The tool **IDR** has not been tested so far for CUT and RUN.

### Extract Positive Peaks

We can further remove some noise with a positive control, that is why we have downloaded a peak set from a ChIP experiment of GATA1.

> <hands-on-title>Select GATA1 peaks from ChIP-Seq data:</hands-on-title>
>
> 1. {% tool [bedtools Intersect intervals find overlapping intervals in various ways](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.31.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"File A to intersect with B"*: Select **Robust GATA1 CUT and RUN peaks**
>    - *"Combined or separate output files"*: `One output file per 'input B' file`
>        - {% icon param-file %} *"File B to intersect with A"*:  Select the dataset `GATA1 ChIP-Seq peaks`
>    - *"Calculation based on strandedness?"*: `Overlaps occurring on either strand`
>    - *"Required overlap"*: `Specify minimum overlap(s)`
>        - *"Minimum overlap required as a fraction of A"*: `0.5`
>        - *"Require that the fraction of overlap be reciprocal for A and B"*: `No`
>    - *"Write the original A entry _once_ if _any_ overlaps found in B.": `Yes`
>
> 3. Rename the datasets `True GATA1 CUT and RUN peaks`.
>
> > <question-title></question-title>
> >
> > 1. How many potential true positives have we found (common between CUT&RUN and ChIP-seq) ?
> > 2. How many potential false positives have we removed?
> > 3. What is the precision of our analysis at this point? (Precision = True Positive / True Positive + False Positive)
> >
> > > <solution-title></solution-title>
> > >
> > > 1. 2,809 peaks
> > > 2. 56 peaks
> > > 3. Our precision is ~ 98%. A high precision is an indicator that we can predict true binding regions with high confidence.
> > {: .solution}
> >
> {: .question}
{: .hands_on}

### Extract DNA sequence

> <hands-on-title>Obtain DNA sequences from a BED file</hands-on-title>
>
> 1. {% tool [Extract Genomic DNA using coordinates from assembled/unassembled genomes](toolshed.g2.bx.psu.edu/repos/iuc/extract_genomic_dna/Extract genomic DNA 1/3.0.3+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Fetch sequences for intervals in"*: `True GATA1 CUT and RUN peaks`
>    - *"Interpret features when possible"*: `No`
>    - *"Choose the source for the reference genome": `locally cached`
>    - *"Using reference genome"*: `hg38`
>    - *"Select output format"*: `fasta`
>    - *"Select fasta header format"*: `bedtools getfasta default`
>
{: .hands_on}

## Motif detection with MEME-ChIP

Let's find out the sequence motifs of the TF GATA1. Studies have revealed that GATA1 has the DNA binding motif (T/A)GATA(A/G) {% cite Hasegawa2017 %}. In this last task we search for this motif in our data.

> <hands-on-title>Motif detection</hands-on-title>
>
> 1.  {% tool [MEME-ChIP](toolshed.g2.bx.psu.edu/repos/iuc/meme_chip/meme_chip/4.11.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Primary sequences"*: Select the output of **Extract Genomic DNA** {% icon tool %}
>    - *"Sequence alphabet"*: `DNA`
>    - *"Options Configuration"*: `Advanced`
>        - *"Should subsampling be random?"*: `Yes`
>            - *"Seed for the randomized selection of sequences"*: `123`
>        - *"E-value threshold for including motifs"*: `0.05`
>        - *"What is the expected motif site distribution?"*: `Zero or one occurrence per sequence`
>        - *"Minimum motif width"*: `5`
>        - *"Maximum motif width"*: `20`
>        - *"Maximum number of motifs to find"*: `5`
>        - *"Stop DREME searching after reaching this E-value threshold"*: `0.05`
>        - *"Stop DREME searching after finding this many motifs"*: `5`
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
> 2. View the **MEME-ChIP** HTML.
>
> > <question-title></question-title>
> >
> > 1. What is the E-value of the main motif?
> > 2. How many peaks support the main motif GATA?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. 1.1e-411
> > > 2. You need to click on the DREME link for the GATA motif. Then when you are in the DREME html, click on the arrow pointing down for the first motif (HGATAA) in the More column. Here, it gives you some numbers including how many peaks were positive for the motif. Here 1939.
> > {: .solution}
> >
> {: .question}
{: .hands_on}

![MEME-ChiP motifs for GATA1](../../images/cut_and_run/GATA_motif.png "Plot of the first five sequence motifs of MEME-ChIP.")

Let us first investigate the result of MEME-ChIP. **First column:** The x-axis of the sequence plots represents the nucleotide position of the motif.
The y-axis stands for the total information (uncertainty) and thus stands for the probability that the nucleotide at a certain position is the specific letter (for DNA: T,C,G,A).
Bigger letters stand for a higher probability. For more information please read [the corresponding Wikipedia page](https://en.wikipedia.org/wiki/Sequence_logo). **Second column:** Defines the algorithm that found the sequence motif.
MEME-ChIP uses two different approaches called MEME and DREME, which can find more complex or simpler motifs. More information about these tools can be found [in the Meme documentation](http://meme-suite.org/).
**Third column:** Defines the E-value, which represents the expected number of times we would find our sequence motif in a database (peak file), just by chance.
This means, that a small E-value corresponds to a very significant motif because the expected number we would find that motif in our peak file just by chance is very low.
Other sequences like repeats will have a high E-value on the other hand. **Fourth column:** This graph is the result of CentriMo, a tool that shows the distribution of
the best matches for that particular motif. The vertical line is the center of the motif. Therefore, if you have a long sequence motif CentriMo helps you to pinpoint a smaller sub-motif.
You can find more information on CentriMo [in the Meme documentation](https://meme-suite.org/meme/doc/centrimo.html).

The results of MEME-ChIP endorse the findings about the DNA binding motif (T/A)GATA(A/G) {% cite Hasegawa2017 %}. We also found other motifs, that might be secondary sequence motifs.
This, we could test with a deeper analysis.

# Step 6: Visualisation of Coverage

In the previous section we generated the `True GATA1 CUT and RUN peaks` file by eliminating the false positives. In this step, we will visualize the the coverage across the peaks using the `bedgraph` file generated by **MACS2** {% icon tool %}.

## Convert bedgraph from **MACS2** to bigwig
The BedGraph format is human-readable but can be quite large, making it slow to visualize specific regions. For better efficiency, we will convert it to the BigWig format, a binary format that allows for fast visualization of any genomic region.

> <hands-on-title>Convert BedGraphs to BigWig.</hands-on-title>
>
> 1. {% tool [Wig/BedGraph-to-bigWig](wig_to_bigWig) %} with the following parameters:
>    - {% icon param-file %} *"Convert"*: Select the output of **MACS2** {% icon tool %} (Bedgraph Treatment).
>    - *"Converter settings to use"*: `Default`
>
> 2. Rename the datasets `MACS2 bigwig`.
{: .hands_on}


## Create heatmap of coverage at TSS with deepTools

You might be interested in checking the coverage on binding GATA1 binding sites. For this, you can compute a heatmap. We will use the **deepTools plotHeatmap**. Here we make a heatmap centered on the transcription start sites (TSS).

### Generate computeMatrix

The input of **plotHeatmap** is a matrix in a hdf5 format. To generate it we use the tool **computeMatrix** that will evaluate the coverage at each GATA1 peak.

> <hands-on-title>Generate the matrix</hands-on-title>
>
> 1. {% tool [computeMatrix](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.5.4+galaxy0) %} with the following parameters:
>    - In *"Select regions"*:
>        - {% icon param-repeat %} *"Insert Select regions"*
>            - {% icon param-file %} *"Regions to plot"*: `True GATA1 CUT and RUN peaks`
>    - *"Sample order matters"*: `No`
>        - {% icon param-file %} *"Score file"*: `MACS2 bigwig`, output of **Wig/BedGraph-to-bigWig** {% icon tool %}.
>    - *"computeMatrix has two main output options"*: `reference-point`
>    - *"The reference point for the plotting"*: `beginning of region (e.g. TSS)`
>    - *"Show advanced output settings"*: `no`
>    - *"Show advanced options"*: `yes`
>        - *"Convert missing values to 0?"*: `Yes`
>
{: .hands_on}


### Plot with **plotHeatmap**

Now we will generate a heatmap. Each line will be a peak. The coverage will be summarized with a color code from red (no coverage) to blue (maximum coverage). All TSS will be aligned in the middle of the figure and only the 1 kb around the TSS will be displayed. Another plot, on top of the heatmap, will show the mean signal at the TSS. There will be one heatmap per replicate.

> <hands-on-title>Generate the heatmap</hands-on-title>
>
> 1. {% tool [plotHeatmap](toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_heatmap/deeptools_plot_heatmap/3.3.2.0.1) %} with the following parameters:
>    - {% icon param-file %} *"Matrix file from the computeMatrix tool"*: output of **computeMatrix** {% icon tool %}.
>    - *"Show advanced output settings"*: `no`
>    - *"Show advanced options"*: `Yes`
>    - *"The x-axis label”*: `distance from peak center (bp)`
>    - *"The y-axis label for the top panel"* : `GATA1 peaks`
>    - *"Reference point label”*: `peak center`
>    - *"Labels for the regions plotted in the heatmap"*: `GATA1_peaks`
{: .hands_on}

> <comment-title>plotHeatmap Results</comment-title>
> This is what you get from plotHeatmap:
> ![plotHeatmap output](../../images/cut_and_run/peak_coverage_heatmap.png "plotHeatmap output")
{: .comment}

> <question-title></question-title>
>
> 1. Is the coverage similar in replicates?
> 2. What does the coverage tell us about the binding sites?
>
> > <solution-title></solution-title>
> >
> > 1. Except a slightly more average coverae in replicate 2, both of them have similar sharp pattern.
> > 2. Such sharp coverage indicates that the binding sites are highly specific, which is expected for transcription factor binding.
> >
> {: .solution}
>
{: .question}


# Conclusion

In this training, you have learned the general principles of CUT&RUN data analysis. CUT&RUN is a method to analyse DNA-associated proteins. In contrast to ChIP-seq, CUT&RUN couples the
antibody with a pA(G)-MNase to reduce the size of the obtained sequences to increase the accuracy for the detection of the protein interaction. Based on this training material,
you gained some insights into the quality control of the data. You should know some pitfalls of CUT&RUN and look for adapter contamination and PCR duplicates, which disrupt the later enrichment analysis.
You learned about **Bowtie2** to do the alignment, and how to filter for properly paired, good-quality and reads that do not map to the mitochondrial genome. Furthermore, you learned that
an indicator for a good CUT&RUN experiment is the fragment size plot and that you optimize your protocol based on this parameter. With **MACS2** we identified enriched regions, but
we had to remove some noise. We performed some robust peak analysis with **bedtools intersect** and performed a check with a ChIP-Seq experiment to remove some false positives.
Finally, we used **MEME-ChIP** to identify our binding motifs.

![CUT&RUN workflow](../../images/cut_and_run/cut_and_run_mindmap.png)

tools_json:
["Flatten collection", "__FLATTEN__", "Falco", "toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3", "Trim Galore!", "toolshed.g2.bx.psu.edu/repos/bgruening/trim_galore/trim_galore/0.6.7+galaxy1", "Bowtie2", "toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.3+galaxy1", "plotFingerprint", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_fingerprint/deeptools_plot_fingerprint/3.5.4+galaxy0", "Filter BAM datasets on a variety of attributes", "toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.5.2+galaxy2", "MarkDuplicates", "toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_MarkDuplicates/3.1.1.0", "Select lines that match an expression", "Grep1", "Transpose rows/columns in a tabular file", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.1.0", "Paired-end histogram", "toolshed.g2.bx.psu.edu/repos/iuc/pe_histogram/pe_histogram/1.0.2", "bedtools BAM to BED converter", "toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_bamtobed/2.31.1+galaxy0", "MACS2 callpeak", "toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.2.9.1+galaxy0", "Extract Dataset", "__EXTRACT_DATASET__", "bedtools Intersect intervals find overlapping intervals in various ways", "toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.31.1+galaxy0", "IDR compare ranked list of identifications", "toolshed.g2.bx.psu.edu/repos/iuc/idr/idr/2.0.3", "Extract Genomic DNA using coordinates from assembled/unassembled genomes", "toolshed.g2.bx.psu.edu/repos/iuc/extract_genomic_dna/Extract genomic DNA 1/3.0.3+galaxy3", "MEME-ChIP", "toolshed.g2.bx.psu.edu/repos/iuc/meme_chip/meme_chip/4.11.2+galaxy1", "Wig/BedGraph-to-bigWig", "wig_to_bigWig", "computeMatrix", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_compute_matrix/deeptools_compute_matrix/3.5.4+galaxy0", "plotHeatmap", "toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_heatmap/deeptools_plot_heatmap/3.3.2.0.1"]


topic: epigenetics
tutorial: ewas-suite
title: Infinium Human Methylation BeadChip
difficulty: intermediate
time_estimation: 1h
requirements_json: []
tags_json: []
content_hash: 414269423520ff659eb984483268a833
last_modified: 2026-05-15T22:28:12.318269
zenodo_link: https://zenodo.org/record/1251211
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/epigenetics/tutorials/ewas-suite/tutorial.html

content:
This tutorial is based on Hugo W, Shi H, Sun L, Piva M et al.: Non-genomic and Immune Evolution of Melanoma Acquiring MAPKi Resistance {% cite Hugo2015 %}.

> <agenda-title></agenda-title>
> In this tutorial we will do:
> 1. TOC
> {:toc}
> We will use a small subset of the original data. If we run the tutorial on the orginal dataset, analysis will be time consuming and not reproducible Infinium Human Methylation BeadChip computation on the orginal data can be found at [case study](https://github.com/kpbioteam/case_study).
>
>
{: .agenda}

# Introduction


The field of cancer genomics has demonstrated the power of massively parallel sequencing techniques to inform on genes and specific alterations that drive tumor onset and progression. Although large comprehensive sequence data sets continue to be made increasingly available, data analysis remains an ongoing challenge, particularly for laboratories lacking dedicated resources and bioinformatics expertise. To address this, we have provided training based on Galaxy Infinium Human Methylation BeadChip tool that represents many popular algorithms for detecting somatic genetic alterations from genome and exome data.

![epimechanism](../../images/Epigenetic_mechanisms.jpg "How epigenetics mechanism can effect health (adapted from https://commonfund.nih.gov/epigenomics/figure)")

This exercise uses datasets from the Cell publication by {% cite Hugo2015 %}. with the goal being the identification of differentially methylated regions and positions associated with treatment resistant melanomas.
Datasets include the Infinium Human Methylation BeadChip array performed in melanoma tumors in a sample of patients pre and post MAPKi and BRAFi treatment with different outcomes (sensitive and resistant). For each sample there is raw green (methylated) and red (unmethylated) colour arrays containing the summarised bead information generated by the Infinium Human Methylation BeadChip scanner.

The Infinium Human Methylation BeadChip uses two different bead types to detect changes in DNA methylation levels. In the figure we can see M - methylated and U - unmethylated bead types. In our study unmethylated and methylated bead signals are reported as green and red colors respectively.

![methassay](../../images/methylationassay.png "Infinium Methylation Assay Overview (adapted from {% cite Illumina %})")


Accession  | Sensitivity | Treatment
--- | ---  | ---
GSM1588704 | baseline    | pre-treatment
GSM1588705 | baseline    | pre-treatment
GSM1588706 | resistant   | BRAFi
GSM1588707 | resistant   | BRAFi

The workflow combines 5 main steps, starting with raw intensity data loading (.idat) and then optional preprocessing and normalisation of the data. The next quality control step performs an additional sample check to remove low-quality data, which normalisation cannot detect. The workflow gives the user the opportunity to perform any of these preparation and data cleaning steps, including a highly recommended genetic variation annotation step resulting in single nucleotide polymorphism identification and removal. Finally, the dataset generated through all of these steps can be used to hunt (find) differentially-methylated positions (DMP) and regions (DMR) with respect to a phenotype covariate.

# Raw intensity data loading

The first step of the Infinium Human Methylation BeadChip array analysis is raw methylation data loading (intensity information files for each two colour micro array)

> <hands-on-title>Data Loading</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the following IDAT files from [Zenodo](https://zenodo.org/record/1251211#.WwREQ1Mvz-Y) or from the data library (ask your instructor)
>    - `GSM1588704_8795207135_R01C02_Red.idat`
>    - `GSM1588705_8795207119_R05C02_Red.idat`
>    - `GSM1588706_8795207135_R02C02_Red.idat`
>    - `GSM1588707_8795207119_R06C02_Red.idat`
>    - `GSM1588704_8795207135_R01C02_Grn.idat`
>    - `GSM1588705_8795207119_R05C02_Grn.idat`
>    - `GSM1588706_8795207135_R02C02_Grn.idat`
>    - `GSM1588707_8795207119_R06C02_Grn.idat`
>    - `phenotypeTable.txt`
>
>    > <details-title>List of Zenodo URLs</details-title>
>    > ```
>    > https://zenodo.org/record/1251211/files/GSM1588704_8795207135_R01C02_Red.idat
>    > https://zenodo.org/record/1251211/files/GSM1588706_8795207135_R02C02_Red.idat
>    > https://zenodo.org/record/1251211/files/GSM1588705_8795207119_R05C02_Red.idat
>    > https://zenodo.org/record/1251211/files/GSM1588707_8795207119_R06C02_Red.idat
>    > https://zenodo.org/record/1251211/files/GSM1588704_8795207135_R01C02_Grn.idat
>    > https://zenodo.org/record/1251211/files/GSM1588706_8795207135_R02C02_Grn.idat
>    > https://zenodo.org/record/1251211/files/GSM1588705_8795207119_R05C02_Grn.idat
>    > https://zenodo.org/record/1251211/files/GSM1588707_8795207119_R06C02_Grn.idat
>    > https://zenodo.org/record/1251211/files/phenotypeTable.txt
>    > ```
>    {: .details}
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>    > <comment-title>Phenotype table</comment-title>
>    > Phenotype table can be in different sizes with different arguments, however the second column is required to contain phenotype covariate information for each sample.
>    {: .comment}
>
> 3. Run **UCSC Main** {% icon tool %} to obtain the reference genome. The tool will take you to the **UCSC table browser**. Use the following parameters to extract the reference genome
>   - *"clade"*: `Mammal`
>   - *"genome"*: `Human`
>   - *"assembly"*: `Feb. 2009 (GRCh37/hg19)`
>   - *"group"*: `Regulation`
>   - *"track"*: `HAIB Methyl450`
>   - *"table"*: `GM12878 (wgEncodeHaibMethyl450Gm12878SitesRep1)`
>   - *"region"*: `genome`
>   - *"output format"*: `GTF - gene transfer (limited)`
>   - *"Send output to"*: `Galaxy` (only)
>   - Click on the **get output** button at the bottom of the screen
>   - On the next page, click on the **Send Query to Galaxy** button
>   - Wait for the upload to finish
>
>
> After exporting the reference genome from UCSC, we need to make sure that it is in the right dataset build.
>
> Click on the **Differentially_Methylated_Positions.bed** output in your history to expand it. \
> Set the database build of your dataset to `Human Feb. 2009 (GRCh37/hg19) (hg19)`(if it is not set automatically)
>
> {% snippet faqs/galaxy/datasets_change_dbkey.md dbkey="hg19" %}
>
> Click on `display at UCSC` towards the bottom of the history item.
> This will launch UCSC Genome Browser with your Custom Track
> ![Display at UCSC](../../images/ucsc.png "UCSC genome track showing differentialy methylated regions located on chromosome 6")
{: .hands_on}


# .idat preprocessing

Preprocessing and data quality assurance is an important step in Infinium Methylation Assay analysis. Idat dataset represents two colour data with a green and a red channel and can be converted into methylated and unmethylated signals or into Beta values. The Infinium Human Methylation BeadChip tool extracts and plots the quality control data frame with two columns mMed and uMed which are the medians of methylation signals (Meth and Unmeth). Comparing them against one another allows users to detect and remove low-quality samples.

![quality_control](../../images/quality_control.png)

> <comment-title>Normalisation of the data</comment-title>
> If your files require normalisation, you might prefer to use one of the other preprocessing tools provided in Infinium Human Methylation BeadChip tool i.e. Preprocess Funnorm or Preprocess Quantile look for recommendation at {% cite Hansen %}.
>
{: .comment}

# Differentially methylated regions and positions analysis

The main goal of the **Infinium Human Methylation BeadChip** analysis is to simplify the way differentially methylated loci sites are detected. The **Infinium Human Methylation BeadChip** pipeline contains differentially methylated positions (DMPs) detection with respect to a phenotype covariate, and more complex solutions for finding differentially methylated regions (DMRs). Genomic regions that are differentially methylated between two conditions can be tracked using a bumphunting algorithm. The algorithm first implements a t-statistic at each methylated loci location, with optional smoothing, then groups probes into clusters with a maximum location gap and a cutoff size to refer the lowest possible value of genomic profile hunted by our tool.

> <hands-on-title>detecting methylated loci sites</hands-on-title>
> 1. Run {% tool [Infinium Human Methylation BeadChip](toolshed.g2.bx.psu.edu/repos/kpbioteam/ewastools/minfi_analysis/2.1.0) %} with the following parameters to map the imported datasets against phenotype covariate and reference genome obtained from UCSC.
>    - {% icon param-files %} *"red channel files"*: all files ending in `_Red`
>    - {% icon param-files %} *"green channel files"*: all files ending in `Grn`
>
>
> Ilumina methylation array data can be mapped to the genome with or without additional preprocessing methods. Incomplete annotation of genetic variations such as single nucleotide polymorphism (SNP) may affect DNA measurements and disrupt downstream analysis of results. {% cite Hansen %} It is highly recommended to remove the probes that contain either an SNP at the methylated loci interrogation or at the single nucleotide extension. In this tutorial we will remove probes affected by genetic variation by selecting **(Optional) Preprocessing Method** {% icon tool %}.
>
>    - *"(Optional) Preprocessing Method"*: `Remove SNPS`
>
>
>    - *"Phenotype Table"*:`The phenotypeTable.txt file uploaded from Zenodo`
>    - *"maxGap Size"*:`250`
>    We will use the default gap of 250 base pairs (bps), i.e. any two points more than 250 bps away are put in a new cluster.
>    - *"Cutoff Size"*:`0.1`
>    In order to find segments that are positive, near zero, and negative. We need a cutoff which is one number in which case “near zero” default 0.1
>    - *"Number of Resamples"*:`0`
> Default value 0 for permutation method apply selection of randomized cases with replacement from the original data while using 'bootstrap' method.
>    - *"nullMethod"*:`permutation`
> Method used to generate null candidate regions, must be one of ‘bootstrap’ or
> ‘permutation’ (defaults to ‘permutation’).
>    - *"Phenotype Type"*:`categorical`
> Identify regions where methylation is associated with a continuous or categorical phenotype.
>    - *"qCutoff Size"*:`0.5`
> Diffrentialy methylated positions with an FDR q-value greater than this value will not be returned.
>    - *"Variance Shrinkage"*:` TRUE`
> Default TRUE as it is recommended when sample sizes are small <10
>    - *"Genome Table"*: `wgEncodeHaibMethyl450 ... `
>
{: .hands_on}

> <question-title></question-title>
> How do we define phenotype covariate?
> > <solution-title></solution-title>
> > Phenotype covariate is the set of observable characteristics of an individual resulting from the gene-environment interactions
> {: .solution}
{: .question}
# Annotation and visualization
In addition to downstream analysis users can annotate the differentially methylated loci at the promoter regions of genes with gene function descriptions, and relationships between these concepts.
> <hands-on-title> Annotate Differentially Methylated Position</hands-on-title>
> 1. Run {% tool [ChIPpeakAnno annoPeaks](toolshed.g2.bx.psu.edu/repos/kpbioteam/chipeakanno_annopeaks/chipeakanno_annopeaks/0.1.0) %} on the output of {% tool [Infinium Human Methylation BeadChip](toolshed.g2.bx.psu.edu/repos/kpbioteam/ewastools/minfi_analysis/2.1.0) %} with the following parameters
>   - {% icon param-file %} *"Differentialy methylated Positions"*: output of {% tool [Infinium Human Methylation BeadChip](toolshed.g2.bx.psu.edu/repos/kpbioteam/ewastools/minfi_analysis/2.1.0) %}
>   - *"bindingType"*: `StartSite`
>   - *"bindingRegionStart"*:`-5000`
>   - *"bindingRegionEnd"*:`3000`
>   - *"Additional Column of Score"*:`5`
>
>        Position of column of score optional value if it is required
>
> 2. Run {% tool [Cut](Cut1) %} on the previous output adjusting the following parameters to cut "gene_name" column from table of annotated peaks and then get a list of genes
>   - *"Cut columns"*: `c16`
>   - *"Delimited by"*: `Tab`
>   - {% icon param-file %} *"From"*: output of {% tool [ChIPpeakAnno annoPeaks](toolshed.g2.bx.psu.edu/repos/kpbioteam/chipeakanno_annopeaks/chipeakanno_annopeaks/0.1.0) %}
>
> 3. Use {% tool [Remove beginning](Remove beginning1) %} on `Gene List` with the following parameters
>   - *"Remove first"*: `1`
>   - {% icon param-file %} *"from"*: output of {% tool [Cut](Cut1) %}
>
> 4. Run {% tool [Cluster Profiler Bitr](toolshed.g2.bx.psu.edu/repos/kpbioteam/clusterprofiler_bitr/clusterprofiler_bitr/0.1.0) %} on the previous output adjusting the following parameters to convert the list of genes to list of entrez ID
>   - *"Input Type Gene ID"*: `SYMBOL`
>   - *"Output Type Gene ID"*: `ENTREZID`
>
> 5. Run a GO Enrichment Analysis using {% tool  [clusterProfiler go](toolshed.g2.bx.psu.edu/repos/kpbioteam/clusterprofiler_go/clusterprofiler_go/0.1.0) %} on the output of the {% tool  [Cluster Profiler Bitr](toolshed.g2.bx.psu.edu/repos/kpbioteam/clusterprofiler_bitr/clusterprofiler_bitr/0.1.0) %}
{: .hands_on}

![Functional annotations](../../images/funcann.jpg "Results of GO enrichments analysis for DMPs")

ID  | Description | pvalue | qvalue | geneID | Count
--- | ---  | --- | --- | --- | ---
GO:0048732 | gland development  | 1.38E-58 | 4.23E-55 | PTGS2 / KCNC1 / FZD1 /SLC22A18 /SLC22A3 (...) | 372
GO:1901652 | response to peptide | 3.99E-57 | 8.13E-54 | SULF1/ LAMA5/ MED1 /CFLAR/ MSX2 (...) | 359
GO:0048545 | response to steroid hormone | 1.38EE-54 | 2.11E-51 | HDAC9/ RAB10/ CFLAR/ WDTC1 (...) | 394

# Conclusion


Epigenetic aberrations which involve DNA modifications give researchers an interest in identifying novel non-genetic factors responsible for complex human phenotypes such as height, weight, and disease. To identify methylation changes researchers need to perform complicated and time consuming computational analysis. Here, the EWAS suite becomes a solution for this inconvenience and provides a simplified downstream analysis available as a ready to run pipline in supplementary materials. For more details, see the associated publication {% cite Murat2020 %}

tools_json:
["Infinium Human Methylation BeadChip", "toolshed.g2.bx.psu.edu/repos/kpbioteam/ewastools/minfi_analysis/2.1.0", "ChIPpeakAnno annoPeaks", "toolshed.g2.bx.psu.edu/repos/kpbioteam/chipeakanno_annopeaks/chipeakanno_annopeaks/0.1.0", "Cut", "Cut1", "Remove beginning", "Remove beginning1", "Cluster Profiler Bitr", "toolshed.g2.bx.psu.edu/repos/kpbioteam/clusterprofiler_bitr/clusterprofiler_bitr/0.1.0", "clusterProfiler go", "toolshed.g2.bx.psu.edu/repos/kpbioteam/clusterprofiler_go/clusterprofiler_go/0.1.0"]


topic: genome-annotation
tutorial: repeatmasker
title: Masking repeats with RepeatMasker
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: ["eukaryote", "biodiversity"]
content_hash: c53b148d7ae6ec35d43497c24950ee5c
last_modified: 2026-05-27T09:01:48.986312
zenodo_link: https://zenodo.org/record/7085837
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/repeatmasker/tutorial.html

content:
When you assemble a new genome, you get its full sequence in FASTA format, in the form of contigs, scaffolds, or even whole chromosomes if you are lucky. However genomes, in particular for eukaryote organisms, contain a varying but significant proportion of repeated elements all along the sequence. These elements belong to different classes, including:

- Tandem repeats: small sequences (<60 base pairs) repeated next to each other, found in many places in the genome, in particular centromeres and telomeres
- Interspersed repeats: sequences repeated in distant positions, including transposons, {SINEs} or {LINEs}

These repeats are interesting on their own: they can originate from transposons or viral insertions, and they can have direct effects on the expression of genes. But they are also the source of a lot of trouble when you work on genomics data. First when sequencing a genome, assembly tools often have problems reconstructing the genome sequence in regions containing repeats (in particular when repeats are longer than the read size). Then, when you have a good assembly, you want to annotate it to find the location of genes. Unfortunately many annotation tools have trouble identifying gene locations in regions rich in repeats.

The aim of repeat masking is to identify the location of all repeated elements along a genome sequence. Other tools (like annotation pipelines) can then take this information into account when producing their results.

The output of repeat masking tools is most often composed of a fasta file (with sometimes a BED or GFF file containing the position of each repeat). There is two types of masking, producing slightly different fasta output:

- Soft-masking: repeat elements are written in lower case
- Hard-masking: repeat elements are replaced by stretches of the letter N

Normal (non-repeated) sequences are always kept in uppercase. Doing hard-masking is destructive because you lose large parts of the sequence which are replaced by stretches of N. If you want to perform an annotation, it is best to choose soft-masking.

We call this operation "masking" because, by making repeats lowercase, or replacing them with Ns, you make them "invisible" by annotation tools (they are written to mostly ignore the regions marked like this).

Multiple tools exist to perform the masking: [RepeatMasker](https://www.repeatmasker.org/), [RepeatModeler](https://www.repeatmasker.org/RepeatModeler/), [REPET](https://urgi.versailles.inra.fr/Tools/REPET), ... Each one have specificities: some can be trained on specific genomes, some rely on existing databases of repeated elements signatures ([Dfam](https://www.dfam.org/), [RepBase](https://www.girinst.org/repbase/)).

In this tutorial you will learn how to soft-mask the genome sequence of a small eukaryote: Mucor mucedo (a fungal plant pathogen). You can learn how this genome sequence was assembled by following the [Flye assembly tutorial]({% link topics/assembly/tutorials/flye-assembly/tutorial.md %}). We will use two different tools, Red and RepeatMasker, which are probably two of the simplest solutions giving an acceptable result before annotating the genome in the [Funannotate annotation tutorial]({% link topics/genome-annotation/tutorials/funannotate/tutorial.md %}).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/record/7085837/files/genome_raw.fasta
>    https://zenodo.org/record/7085837/files/Muco_library_RM2.fasta
>    https://zenodo.org/record/7085837/files/Muco_library_EDTA.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

# Soft-masking using Red

First let's try Red, a tool than can mask repeats *de novo*. For that, select the input assembly in fasta format.

> <comment-title>*Ab initio* tool</comment-title>
>
> Red is an *ab initio* tool, it means that it will try to predict repeat elements using only the genomic sequence. It's perfect when you know nothing about the organism that you are working on.
>
{: .comment}

> <hands-on-title></hands-on-title>
>
> 1. {% tool [Red](toolshed.g2.bx.psu.edu/repos/iuc/red/red/2018.09.10+galaxy1) %} with the following parameters:
>   - {% icon param-file %} *"Genome sequence to mask"*: `genome_raw.fasta` (Input dataset)
>
{: .hands_on}

Red produces 2 output files :

- A *fasta* file: this is the soft-masked genome that you can use for future analysis. If you display it, you will notice that some portions of the sequence are in lowercase: these are the regions that were identified as repeats
- A *bed* file: this one contains the coordinates on the genome of each repeated loci

> <question-title></question-title>
>
> What proportion of the whole genome sequence is masked?
>
> > <solution-title></solution-title>
> >
> > You need to click on the {% icon galaxy-info %} on one of the output. You should find it at the end of the extended `Tool Standard Output` in **Job Information**. It should be ~ 30.62%
> >
> {: .solution}
>
{: .question}

> <question-title></question-title>
>
> How to hard-mask a genome with **Red** ?
>
> > <solution-title></solution-title>
> >
> > As you can see **Red** has no option to hard-mask your genome. However, one of the output is a `bed` file, so you can use {% tool bedtools MaskFastaBed %} to replace repeated regions with stretches of N:
> >
> > > <hands-on-title></hands-on-title>
> > >
> > > 1. {% tool [bedtools MaskFastaBed](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_maskfastabed/2.30.0) %} with the following parameters:
> > >   - {% icon param-file %} *"BED/bedGraph/GFF/VCF/EncodePeak file"*: `Red on data` (`bed` file produced by red)
> > >   - {% icon param-file %} *"FASTA file"*: `genome_raw.fasta`
> > >
> > {: .hands_on}
> >
> {: .solution}
>
{: .question}

Red uses only the sequence of the genome to detect repeated regions, and does not provide a detailed classification of the detected repeats. Let's use another tool that works differently: RepeatMasker.

# Soft-masking using RepeatMasker

Let's run RepeatMasker, by selected the input assembly in fasta format. We select the soft-masking option, and we choose to use the Dfam database.

> <comment-title>Choosing the right species</comment-title>
>
> We select the `Human (Homo sapiens)` species here, even though we are masking a fungi genome. It means RepeatMasker will identify very common repeats found in many organisms. For more precise results, you can consider selecting a species closer to the one you analyse in the drop down list, or using other more advanced tools like RepeatModeler.
>
{: .comment}

> <hands-on-title></hands-on-title>
>
> 1. {% tool [RepeatMasker](toolshed.g2.bx.psu.edu/repos/bgruening/repeat_masker/repeatmasker_wrapper/4.1.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Genomic DNA"*: `genome_raw.fasta` (Input dataset)
>    - *"Repeat library source"*: `DFam (curated only, bundled with RepeatMasker)`
>    - *"Select species name from a list?"*: `Yes`
>        - *"Species"*: `Human (Homo sapiens)`
>    - *"Output annotation of repeats in GFF format"*: `Yes`
>    - *"Perform soft-masking instead of hard-masking"*: `Yes`
>
{: .hands_on}

RepeatMasker produces 5 output files:

- `masked sequence`: this is the fasta file that you will use for future analysis. If you display it, you will notice that some portions of the sequence are in lowercase: these are the regions that were identified as repeats.
- `repeat statistics`: this one contains some statistics on the number of repeats found in each category, and the total number of base pairs masked.
- `output log`: this is a tabular file listing all repeats.
- `repeat catalogue`: this one contains the list of all repeat sequences that were identified, with their position, and their similarity with known repeats from the Dfam database.
- `repeat annotation` : this one contains the coordinate of each repeat element in GFF2 format.

> <question-title></question-title>
>
> What proportion of the whole genome sequence is masked?
>
> > <solution-title></solution-title>
> >
> > You should find it in the `repeat statistics` output. It should be ~2.41%.
> {: .solution}
>
{: .question}

As we have used a generic species (Human), we only identified the most common and simple repeats, not very specific to this species. If you compare with Red results, your are missing at least ~28% of repeated content in the genome. However, RepeatMasker gives interesting information about repeat classification which could be interesting for future analysis.

To boost RepeatMasker performance, we need a tailored repeat library for *Mucor mucedo*. This step can take from a few hours to a few days and a large number of tools could be used. We pre-computed two librairies:

- `Muco_library_RM2.fasta` using RepeatModeler ({% cite Flynn_2020 %})
- `Muco_library_EDTA.fasta` using EDTA ({% cite Ou_2019 %})

> <details-title>Preparing a library</details-title>
>
> You can prepare your own repeat library, specific to the genome you're using, by running RepeatModeler in Galaxy (it will take several hours to run):
>
> > <hands-on-title>Building a repeat library with RepeatModeler</hands-on-title>
> >
> > 1. {% tool [RepeatModeler](toolshed.g2.bx.psu.edu/repos/csbl/repeatmodeler/repeatmodeler/2.0.4+galaxy1) %} with the following parameters:
> >    - {% icon param-file %} *"Input genome fasta"*: `genome_raw.fasta` (Input dataset)
> {: .hands_on}
>
> The library is in the "consensus sequences" output dataset.
{: .details}

> <hands-on-title></hands-on-title>
>
> 1. {% tool [RepeatMasker](toolshed.g2.bx.psu.edu/repos/bgruening/repeat_masker/repeatmasker_wrapper/4.1.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Genomic DNA"*: `genome_raw.fasta` (Input dataset)
>    - *"Repeat library source"*: `Custom library of repeats`
>    - *"Custom library of repeats"*
>        - *"One of the two pre-computed libraires"*: `Muco_library_RM2.fasta` or `Muco_library_EDTA.fasta`
>    - *"Output annotation of repeats in GFF format"*: `Yes`
>    - *"Perform soft-masking instead of hard-masking"*: `Yes`
{: .hands_on}

> <question-title></question-title>
>
> Compare the different `repeat statistics` files produced, what is the highest library for RepeatMasker?
>
> > <solution-title></solution-title>
> >
> > The RepeatModeler library seems to have the highest percentage of repeats found with ~ 34.89%. It could be explained as RepeatModeler is specifically made to work with RepeatMasker.
> {: .solution}
>
{: .question}

Other tools might mask a greater proportion of the genome, at the cost of a more complex workflow with training steps. But masking more isn't always a positive results! In fact, large family of genes could be considered as a repeat by some tools or certain library. Only a manual curation can correct those mistakes, but this result is sufficient to perform an annotation by following the [Funannotate annotation tutorial]({% link topics/genome-annotation/tutorials/funannotate/tutorial.md %}).

# Conclusion



By following this tutorial you have learn how to mask an eukaryotic genome using Red and RepeatMasker, after assembling ([Flye assembly tutorial]({% link topics/assembly/tutorials/flye-assembly/tutorial.md %})) and before annotating it ([Funannotate annotation tutorial]({% link topics/genome-annotation/tutorials/funannotate/tutorial.md %})).

Often times, annotation tools prefer to use soft-masked genomes, as they primarily search for genes in non repeated regions, but tolerate that some genes overlap partially with these regions.

tools_json:
["Red", "toolshed.g2.bx.psu.edu/repos/iuc/red/red/2018.09.10+galaxy1", "bedtools MaskFastaBed", "toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_maskfastabed/2.30.0", "RepeatMasker", "toolshed.g2.bx.psu.edu/repos/bgruening/repeat_masker/repeatmasker_wrapper/4.1.5+galaxy0", "RepeatModeler", "toolshed.g2.bx.psu.edu/repos/csbl/repeatmodeler/repeatmodeler/2.0.4+galaxy1"]


topic: genome-annotation
tutorial: bacterial-comparative-genomics-dataset-construction
title: Dataset construction for bacterial comparative genomics
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: ["microgalaxy"]
content_hash: 86d2bd2666a7d77ab3c35e92ffa39af9
last_modified: 2026-05-27T09:01:48.732765
zenodo_link: https://zenodo.org/records/1
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/bacterial-comparative-genomics-dataset-construction/tutorial.html

content:
Introduction


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get genomes from GTDB

> <hands-on-title> Task description </hands-on-title>
>
> 1. 
>
{: .hands_on}



> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 3. {% tool [Import](upload1) %} the contig file from [Zenodo]({{ page.zenodo_link }}) or from Galaxy shared data libraries:
>
>    ```
>    {{ page.zenodo_link }}/files/DRR187559_contigs.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
{: .hands_on}



# Extract IDs for NCBI

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - *"Delimited by"*: `Comma`
>    - {% icon param-file %} *"From"*: `output` (Input dataset)
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}


> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c6`
>    - {% icon param-file %} *"From"*: `output` (Input dataset)
>
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}


> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Concatenate multiple datasets or collections](cat1) %} with the following parameters:
>    - {% icon param-file %} *"Concatenate Dataset"*: `out_file1` (output of **Cut** {% icon tool %})
>    - In *"Dataset"*:
>        - {% icon param-repeat %} *"Insert Dataset"*
>            - {% icon param-file %} *"Select"*: `out_file1` (output of **Cut** {% icon tool %})
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Sort](sort1) %} with the following parameters:
>    - {% icon param-file %} *"Sort Dataset"*: `out_file1` (output of **Concatenate datasets** {% icon tool %})
>    - *"on column"*: `c1`
> 1. {% tool [Unique](toolshed.g2.bx.psu.edu/repos/bgruening/unique/bg_uniq/0.3) %} with the following parameters:
>    - {% icon param-file %} *"from query"*: `out_file1` (output of **Sort** {% icon tool %})
>    - *"Advanced Options"*: `Hide Advanced Options`
>
>    ***TODO***: *Check parameter descriptions*
>
>    ***TODO***: *Consider adding a comment or tip box*
>
>    > <comment-title> short description </comment-title>
>    >
>    > A comment about the tool or something else. This box can also be in the main text
>    {: .comment}
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Remove beginning](Remove beginning1) %} with the following parameters:
>    - *"Remove first"*: `2`
>    - {% icon param-file %} *"from"*: `outfile` (output of **Unique** {% icon tool %})
>
>    ***TODO***: *Check parameter descriptions*
>
>    ***TODO***: *Consider adding a comment or tip box*
>
>    > <comment-title> short description </comment-title>
>    >
>    > A comment about the tool or something else. This box can also be in the main text
>    {: .comment}
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}

# Download genomes from NCBI

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [NCBI Datasets Genomes](toolshed.g2.bx.psu.edu/repos/iuc/ncbi_datasets/datasets_download_genome/16.42.0+galaxy0) %} with the following parameters:
>    - In *"Query"*:
>        - *"Choose how to find genomes to download"*: `By NCBI assembly or BioProject accession`
>            - *"Enter accession or read from file ?"*: `Read a list of NCBI Assembly accessions from a dataset`
>                - {% icon param-file %} *"Select dataset with list of NCBI Assembly accessions"*: `out_file1` (output of **Remove beginning** {% icon tool %})
>
>    ***TODO***: *Check parameter descriptions*
>
>    ***TODO***: *Consider adding a comment or tip box*
>
>    > <comment-title> short description </comment-title>
>    >
>    > A comment about the tool or something else. This box can also be in the main text
>    {: .comment}
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}


# Conclusion

Sum up the tutorial and the key takeaways here. We encourage adding an overview image of the
pipeline used.

tools_json:
["Import", "upload1", "Cut", "Cut1", "Concatenate multiple datasets or collections", "cat1", "Sort", "sort1", "Unique", "toolshed.g2.bx.psu.edu/repos/bgruening/unique/bg_uniq/0.3", "Remove beginning", "Remove beginning1", "NCBI Datasets Genomes", "toolshed.g2.bx.psu.edu/repos/iuc/ncbi_datasets/datasets_download_genome/16.42.0+galaxy0"]


topic: genome-annotation
tutorial: funannotate
title: Genome annotation with Funannotate
difficulty: intermediate
time_estimation: 8h
requirements_json: []
tags_json: ["gmod", "eukaryote", "jbrowse1", "biodiversity"]
content_hash: c66586eb60be50d062218c1b9e729fb2
last_modified: 2026-05-27T09:01:48.985597
zenodo_link: https://zenodo.org/record/7867921
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/funannotate/tutorial.html

content:
Genome annotation of eukaryotes is a little more complicated than for prokaryotes: eukaryotic genomes are usually larger than prokaryotes, with more genes. The sequences determining the beginning and the end of a gene are generally less conserved than the prokaryotic ones. Many genes also contain introns, and the limits of these introns (acceptor and donor sites) are not highly conserved.

In this tutorial we will use a software tool called Funannotate ({% cite jonathan_m_palmer_2020_4054262 %}) to annotate the genome sequence of a small eukaryote: [*Mucor mucedo*](https://en.wikipedia.org/wiki/Mucor_mucedo) (a fungal plant pathogen).

As explained on [Funannotate's website](https://funannotate.readthedocs.io/), "it was originally written to annotate fungal genomes (small eukaryotes ~ 30 Mb genomes), but has evolved over time to accomodate larger genomes". As other annotation tools like [Maker]({% link topics/genome-annotation/tutorials/annotation-with-maker/tutorial.md %}) ({% cite Campbell2014 %}) or Braker ({% cite braker %}), it works by aligning as many evidences as possible along the genome sequence, and then reconciliating all these signals to determine probable gene structures.

The evidences can be transcript or protein sequences from the same (or closely related) organism. These sequences can come from public databases (like NR or GenBank) or from your own experimental data (transcriptome assembly from an RNASeq experiment for example). Funannotate is also able to take into account repeated elements.

Funannotate uses ab-initio predictors ([Augustus](http://bioinf.uni-greifswald.de/augustus/), [SNAP](https://github.com/KorfLab/SNAP), [glimmerHMM](https://bio.tools/glimmer-hmm), [CodingQuarry](https://bmcgenomics.biomedcentral.com/articles/10.1186/s12864-015-1344-4) and [GeneMark-ES/ET](http://exon.gatech.edu/GeneMark/) (optional due to licensing)) to improve its predictions: these software tools are able to make gene structure predictions by analysing only the genome sequence with a statistical model.

While for [Maker]({% link topics/genome-annotation/tutorials/annotation-with-maker/tutorial.md %}) you need to perform training steps for the ab-initio predictors, Funannotate is able to take care of that for you, which makes it much easier to use.

In this tutorial, you will learn how to perform a structural genome annotation, and how to evaluate its quality. Then, you will learn how to run functional annotation, using EggNOG-mapper and InterProScan to automatically assign names and functions to the annotated genes. And you will also learn how Funannotate can prepare files ready for submission of your annotation to the NCBI. Finally, you will learn how to use the [JBrowse](http://jbrowse.org/) genome browser to visualise your new annotation.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data upload

To annotate our genome using Funannotate, we will use the following files:

- The **genome sequence** in fasta format. For best results, the sequence should be soft-masked beforehand. You can learn how to do it by following the [RepeatMasker tutorial]({% link topics/genome-annotation/tutorials/repeatmasker/tutorial.md %}). For this tutorial, we will try to annotate the genome assembled in the [Flye assembly tutorial]({% link topics/assembly/tutorials/flye-assembly/tutorial.md %}) and already masked for you using RepeatMasker.
- Some RNASeq data in fastq format. We will align them on the genome, and Funannotate will use it as evidence to annotate genes.
- A set of **protein sequences**, like UniProt/SwissProt. It is important to have good quality, curated sequences here, that's why, by default, Funannotate will use the UniProt/SwissProt databank. In this tutorial, we have prepared a subset of this databank to speed up computing, but you should use UniProt/SwissProt for real life analysis.

 Funannotate will take into account the position of mapped RNASeq reads, and the alignment of protein sequences on the genome sequence to determine gene positions.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/record/7867921/files/genome_masked.fasta
>    https://zenodo.org/record/7867921/files/rnaseq_R1.fq.gz
>    https://zenodo.org/record/7867921/files/rnaseq_R2.fq.gz
>    https://zenodo.org/record/7867921/files/SwissProt_subset.fasta
>    https://zenodo.org/record/7867921/files/alternate_annotation.gbk
>    https://zenodo.org/record/7867921/files/alternate_annotation.gff3
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

# Preparing RNASeq data

When you sequence a new genome, you usually sequence a few libraries of RNASeq data, from different tissues and in different conditions, because this data will help you in annotating the genome. Here, we are using data from one RNASeq dataset that is available on [Sequence Read Archive (SRA)](https://www.ncbi.nlm.nih.gov/sra): [SRR8534859](https://www.ncbi.nlm.nih.gov/sra/?term=SRR8534859).

You would normally get the Fastq files directly from SRA and use them in the following step. To speed up the tutorial (without impairing too much the quality of the results), we have reduced the size of the dataset into a single pair of (smaller) fastq files, available from Zenodo (or the GTN Data Libraries).

To make use of this RNASeq data, we need to map it on the genome. We will use **RNA star** {% icon tool %} and get a result in the form of a BAM file, that we will use in the rest of the tutorial.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [RNA STAR](toolshed.g2.bx.psu.edu/repos/iuc/rgrnastar/rna_star/2.7.8a+galaxy0) %} with the following parameters:
>    - *"Single-end or paired-end reads"*: `Paired-end (as individual datasets)`
>        - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file, forward reads"*: `rnaseq_R1.fq.gz` (Input dataset)
>        - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file, reverse reads"*: `rnaseq_R2.fq.gz` (Input dataset)
>    - *"Custom or built-in reference genome"*: `Use reference genome from history and create temporary index`
>        - {% icon param-file %} *"Select a reference genome"*: `genome_masked.fasta` (Input dataset)
>        - *"Length of the SA pre-indexing string"*: `11`
>
{: .hands_on}

We select `11` for the *"Length of the SA pre-indexing string"* parameter as it is the recommended value for a small genome of this size. If you select 14 (the default value), STAR will advise you to use `11` instead in its logs.

> <comment-title>What if I want to use multiple RNASeq libraries</comment-title>
>
> To get the best possible annotation, it is adviced to use multiple RNASeq libraries, sequences from different tissues in different conditions. To use them, you can map each one individually using STAR, just as in this tutorial. You will get one BAM file per RNASeq library, and you can then easily merge them into a single BAM file by using the {% tool [MergeSamFiles](toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_MergeSamFiles/2.18.2.1) %} tool. This single BAM file can then be used by Funannotate like we do in the next steps.
{: .comment}

Before we move on to the next step, we need to make sure that the mapping went well. Have a look at the `log` output of `RNA STAR`.

> <question-title></question-title>
>
> What proportion of reads were correctly mapped to the genome? Do you think it is enough to continue with this tutorial?
>
> > <solution-title></solution-title>
> >
> > Look for `Uniquely mapped reads %`: you should get ~96%. This is a very good score, because while reducing the size of the dataset for this tutorial, we have kept mostly reads properly mapping.
> > Anyway, with real data, you expect to have a vast majority of reads mapping uniquely on the genome. If it's not the case, check that you're using the correct RNASeq files, with the correct genome sequence.
> >
> {: .solution}
>
{: .question}

# Structural annotation

We can now run **Funannotate predict annotation** to perform the structural annotation of the genome.

We need to input the genome sequence, the mapped RNASeq data, and the proteins to align on the genome. We also specify the name of the species and strain (they can be used later for submission to NCBI).

There are other parameters to finely tune how Funannotate will run ab-initio predictors to predict genes, and to filter the final results based on various criteria. As Funannotate uses [BUSCO](http://busco.ezlab.org/) (Benchmarking Universal Single-Copy Orthologs) for initial training of ab-initio predictors, we select datasets close to the species we are annotating: `mucorales (orthodb 10)` and `rhizopus_oryzae`.

Funannotate is also able to use GeneMark to predict new genes, but to due to licensing restrictions, this software is not available on every Galaxy instance. We will ignore this for this tutorial, it will not impact the results too much.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [Funannotate predict annotation](toolshed.g2.bx.psu.edu/repos/iuc/funannotate_predict/funannotate_predict/1.8.15+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Assembly to annotate"*: `genome_masked.fasta` (Input dataset)
>    - *"Funannotate database"*: select the latest version available
>    - In *"Organism"*:
>        - *"Name of the species to annotate"*: `Mucor mucedo`
>        - *"Strain name"*: `muc1`
>        - *"Is it a fungus species?"*: `No`
>    - In *"Evidences"*:
>        - {% icon param-file %} *"RNA-seq mapped to genome to train Augustus/GeneMark-ET"*: `mapped.bam` (output of **RNA STAR** {% icon tool %})
>        - *"Select protein evidences"*: `Custom protein sequences`
>            - {% icon param-file %} *"Proteins to map to genome"*: `SwissProt_subset.fasta` (Input dataset)
>    - In *"Busco"*:
>        - *BUSCO models to align"*": `mucorales (orthodb 10)`
>        - *"Initial Augustus species training set for BUSCO alignment"*: `rhizopus_oryzae`
>    - *"Which outputs should be generated"*: Select all
>
{: .hands_on}

> <comment-title>on parameters</comment-title>
>
> - For *"Select protein evidences"* we select `Custom protein sequences` to reduce the computing time, but for real data analysis, you should select the default value: `Use UniProtKb/SwissProt (from selected Funannotate database)`.
> - It is possible to enable the *"Is it a fungus species?"* option in Funannotate: it launches an additional ab initio predictor (CodingQuerry) dedicated to fungi genomes. However it has proved to be unstable on the genome studied in this tutorial, and it can create a lot of fragmented gene models depending on the RNASeq data available. For this tutorial we leave this option to `No`. You can test it with real data, but be sure to compare the result with and without this option.
> - For real data analysis you can consider enabling the *"Augustus settings (advanced)"* > *"Run 'optimize_augustus.pl' to refine training (long runtime)"*. If you have enough data, you might get better results as there will be an additional training step for augustus (at the cost of a longer runtime).
{: .comment}

> <comment-title>Don't wait</comment-title>
>
> This step will take a bit of time to run. While it runs, we can already schedule the following functional annotation steps. Galaxy will run them automatically as soon as the structural annotation is ready.
{: .comment}

This tool produces several output dataset, in particular:

- the full structural annotation in Genbank, GFF3 or NCBI tbl formats: these files contain the position of all the genes that were found on the genome.
- the CDS, transcript and protein sequences of all the genes predicted by Funannotate (fasta files)
- some statistics and reports

Let's have a closer look at the output of our annotation. First display the `stats` dataset: the first par of the file contains some information on how funannotate was launched. If you go to the bottom, you'll find a few interesting numbers in the `annotation` section:

- the total number of genes and mRNA
- the average length of genes, exons, proteins
- the number of single/multiple exon transcripts

These number alone are interesting, but not fully informative on the quality of the annotation. For example, for the number of genes, you want a 'good' number based on what you expect for this species (a too big number can mean that genes are fragmented, and a too small number can mean that some genes were not annotated at all). These numbers can help when comparing an annotation with other ones performed with other parameters or tools. You can also compare these numbers with annotation of closely related species.

To get a better picture of the quality of the result, we will run BUSCO in the next step.

Before moving on, have a quick look at the `tbl2asn error summary report` output: it lists a few potential problems that were identified by Funannotate in the results it generated. For example, Funannotate can tell you when it predicted genes that contain very short exons, or that use a rare splice site sequence. You can have a detailed list of identified potential problems in the `tbl2asn genome validation report` dataset. It does not mean that each listed gene is wrong, but it means that you might want to give a closer look at these ones. If you have time to manually check each gene, Apollo can help you in doing this, see the note in the conclusion for this.

{% include topics/genome-annotation/tutorials/functional/content.md short=false %}

# Submission to NCBI

If you plan to submit the final genome sequence and annotation to NCBI, there are a few steps to follow.

> <warning-title>Please do not submit this genome to NCBI!</warning-title>
> In this tutorial we will **not** perform a real submission, as we don't want to duplicate the annotation in Genbank every time someone follows this tutorial!
>
> But here's a description of the main steps, and how Funannotate can help you in this process. NCBI provides a complete documentation for [genome and annotation submission](https://www.ncbi.nlm.nih.gov/genbank/genomesubmit/).
{: .warning}

- First, you should have created a BioProject and a BioSample [on the NCBI portal](https://submit.ncbi.nlm.nih.gov/subs/bioproject/), corresponding to your scientific project, and the sample(s) you have sequenced
- The raw reads used for the assembly, and the RNASeq ones should be deposited on SRA, and taggued with the BioProject and BioSample ids
- Ideally you should start submitting the assembly *before* performing the annotation. The reason is simple: NCBI performs some validation on the genome sequence before accepting it into GenBank. It means that you may be forced to make modifications to the genome sequence (e.g. remove contigs, split contigs where you have adapter contamination, etc). You will save time and trouble doing the annotation on a fully validated and frozen genome sequence.
- You should get a `locus_tag` for your genome: it is the unique prefix that is used at the beginning of each gene name. By default, Funannotate uses `FUN_` (e.g. `FUN_000001`), but to submit to NCBI you need to have a prefix specific to your genome. NCBI should provide it to you when you create your BioProject.
- You then need to prepare a file containing a few metadata. This can be done online on [https://submit.ncbi.nlm.nih.gov/genbank/template/submission/](https://submit.ncbi.nlm.nih.gov/genbank/template/submission/). You need to fill the form with some basic information, like that for example (for real data, you should of course write real information!):

![NCBI submission template](../../images/ncbi_template.png "Example of a filled form to generate an NCBI submission template.")

When filled, you can click on the **Create template** button: you will be able to download a `.sbt` file that you should then upload to your Galaxy history.

The next step will show you how Funannotate can use this `.sbt` file, and the `locus_tag`, to generate the needed file for the submission of the annotation to the NCBI. These files will need to be submitted on the [NCBI genome submission portal](https://submit.ncbi.nlm.nih.gov/subs/genome/).

# Integrating structural and functional annotation

Now we have a structural annotation, and the results of both **EggNOG Mapper** and **InterProScan**. Each one is in a separate file, so we will now combine all this data into a single file that will contain both the structural *and* the functional annotation. This will be the final output of our annotation pipeline, ready to be submitted to the NCBI reference database.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [Funannotate functional](toolshed.g2.bx.psu.edu/repos/iuc/funannotate_annotate/funannotate_annotate/1.8.15+galaxy1) %} with the following parameters:
>    - *"Input format"*: `GenBank (from 'Funannotate predict annotation' tool)`
>        - {% icon param-file %} *"Genome annotation in genbank format"*: `annotation (genbank)` (output of **Funannotate predict annotation** {% icon tool %})
>    - *"Funannotate database"*: select the latest version available
>    - {% icon param-file %} *"NCBI submission template file"*: `template.sbt` (the file downloaded from NCBI website, or leave empty if you didn't generate it)
>    - {% icon param-file %} *"Eggnog-mapper annotations file"*: `annotations` (output of **eggNOG Mapper** {% icon tool %})
>    - {% icon param-file %} *"InterProScan5 XML file"*: `InterProScan XML` (output of **InterProScan** {% icon tool %})
>    - {% icon param-file %} *"BUSCO models"*: `mucorales (orthodb 10)`
>    - *"Strain name"*: `muc1`
>    - *"locus_tag from NCBI to rename GFF gene models with"*: `MMUCEDO_` (we consider NCBI has assigned us this `locus_tag`)
>    - *"Which outputs should be generated"*: Select all
>
{: .hands_on}

This tool produces several output dataset, in particular:

- The full structural and functional annotation in Genbank and GFF3 format
- NCBI tbl and sqn (`NCBI Sequin genome`) formats: these files can be used for submission to NCBI
- The CDS, transcript and protein sequences of all the genes predicted by Funannotate
- Some statistics and reports

If you display the GFF3 output, you will notice that the functional information, including gene names, is now stored in this file.

## Evaluation with **Busco**

We now have a complete annotation, including functional annotation, but it's time to evaluate the quality of this annotation. [BUSCO](http://busco.ezlab.org/) (Benchmarking Universal Single-Copy Orthologs) is a tool allowing to evaluate the quality of a genome assembly or of a genome annotation. By comparing genomes from various more or less related species, the authors determined sets of ortholog genes that are present in single copy in (almost) all the species of a clade (Bacteria, Fungi, Plants, Insects, Mammalians, ...). Most of these genes are essential for the organism to live, and are expected to be found in any newly sequenced and annotated genome from the corresponding clade. Using this data, BUSCO is able to evaluate the proportion of these essential genes (also named BUSCOs) found in a set of (predicted) transcript or protein sequences. This is a good evaluation of the "completeness" of the annotation.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.4.6+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Sequences to analyse"*: `protein sequences` (output of **Funannotate functional** {% icon tool %})
>    - *"Mode"*: `annotated gene sets (protein)`
>    - *"Auto-detect or select lineage?"*: `Select lineage`
>        - *"Lineage"*: `Mucorales`
>    - In *"Advanced Options"*:
>        - *"Which outputs should be generated"*: `shortsummary text` and `summary image`
>
{: .hands_on}

> <question-title></question-title>
>
> How many BUSCO genes were found complete in the annotation? Do you think the quality of the annotation is good?
>
> > <solution-title></solution-title>
> >
> > On a total of 2449, you should find ~2312 BUSCO genes identifed as complete in the annotation, with 2281 being in single copy, and 31 being duplicated.
> >
> > That's a quite good result as running BUSCO on the genome itself gives a very close number (2327 Complete BUSCOs) (see Flye assembly tutorial). It means the annotation process was able to detect most of the genes it was supposed to find.
> >
> > To improve the result you can consider using more RNASeq data, and using the *"Augustus settings (advanced)"* > *"Run 'optimize_augustus.pl' to refine training (long runtime)"* option.
> {: .solution}
>
{: .question}

# Visualisation with a genome browser

With Galaxy, you can visualize the annotation you have generated using JBrowse genome browser. This allows you to navigate along the chromosomes of the genome and see the structure of each predicted gene. We also add an RNASeq track, using the BAM file created with **RNA STAR** {% icon tool %}.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: `genome_masked.fasta` (Input dataset)
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Annotation`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `gff3` (output of **Funannotate functional** {% icon tool %})
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `RNASeq`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BAM Pileups`
>                        - {% icon param-file %} *"BAM Track Data"*: `mapped.bam` (output of **RNA STAR** {% icon tool %})
>                        - *"Autogenerate SNP Track"*: `Yes`
>
{: .hands_on}

Click on the newly created dataset's eye to display it. You will see a JBrowse genome browser. You can have a look at the [JBrowse tutorial]({% link topics/visualisation/tutorials/jbrowse/tutorial.md %}) for a more in-depth description of JBrowse.

Enable the annotation track, and the SNPs/Coverage tracks on the left side of JBrowse, then navigate along the genome. You will see the different gene models predicted by Funannotate.

If you click on a gene, a popup will appear with detailed information on the selected gene: position and sequence, but also the functional information retrieved from the **EggNOG mapper** and **InterProScan** results.

If you zoom to a specific gene, and look at the RNASeq tracks, you will see light grey regions corresponding to portions of the genomes where RNASeq were mapped, and darker grey regions corresponding to introns (= regions were some reads were found to match both the end of an exon, and the start of the next one). You can enable the other RNASeq track to display each individual read that was mapped on the genome.

If you navigate along the genome, you will find genes with very low RNASeq coverage: this demonstrates how Funannotate is able to predict genes not only with RNASeq, but also by comparing to protein sequences (from SwissProt) and using ab initio predictors, trained using RNASeq data.

# Comparing annotations

Earlier, we have seen how general statistics and BUSCO results can help to evaluate the quality of an annotation. But when annnotating a new genome, you might want to try different annotation methods and parameters. We will see now how to compare multiple annotations between them. As an example, we will compare the annotation we have generated, with an alternate one, that you have uploaded from Zenodo at the beginning of the tutorial. Comparison is only possible for annotations performed on the same genome sequence.

## Comparing with **AEGeAn ParsEval**

Let's run **AEGeAn ParsEval** first: it compares two annotations in GFF3 format.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [AEGeAn ParsEval](toolshed.g2.bx.psu.edu/repos/iuc/aegean_parseval/aegean_parseval/0.16.0) %} with the following parameters:
>    - {% icon param-file %} *"Reference GFF3 file"*: `gff3` (output of **Funannotate functional** {% icon tool %})
>    - {% icon param-file %} *"Prediction GFF3 file"*: `alternate_annotation.gff3` (Input dataset)
>    - *"Select the output type"*: `HTML`
>
{: .hands_on}

**AEGeAn ParsEval** {% icon tool %} compares gene loci spread all along the genome sequence, each locus containing gene(s) on the "reference" (=the annotation we have generated) and/or the "prediction" (=the alternate one) annotation.

The output is a web page where you can see how many loci or genes are identical or different between the two annotations. By clicking on the `(+)` links, or on the scaffold names, you can see the results more in detail, even at the gene level.

> <hands-on-title></hands-on-title>
>
> 1. Display the report
> 2. Click on `scaffold_11`
> 3. Click on the first locus (between positions 637 and 7730)
>
{: .hands_on}

> <question-title></question-title>
>
> What can you tell from the comparison at this gene locus?
>
> > <solution-title></solution-title>
> >
> > You should see something like this:
> >
> > ![Scaffold 11 - 637..7730](../../images/aegean_scaffold_11_637-7730.png "Comparison of the two annotations on Scaffold 11 - 637..7730")
> >
> > As you can see, at this position, the alternate annotation as found only one big gene, while the annotation you have generated has found 4 different genes, not all on the same strand.
> >
> > This can be explained by the way the alternate annotation was performed: it was done by running Funannotate without any RNASeq data, and choosing a wrong value (`insecta`) in the busco parameter. The annotation you have generated is probably the good one here.
> {: .solution}
>
{: .question}

## Comparing with **Funannotate compare**

**Funannotate compare** {% icon tool %} is another tool to compare several annotations. It uses genbank files, as generated by **Funannotate functional** {% icon tool %}.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [Funannotate compare](toolshed.g2.bx.psu.edu/repos/iuc/funannotate_compare/funannotate_compare/1.8.15+galaxy1) %} with the following parameters:
>    - {% icon param-files %} *"Genome annotations in genbank format"*: `alternate_annotation.gbk` (Input dataset) and `gbk` (output of **Funannotate functional** {% icon tool %})
>    - *"Funannotate database"*: select the latest version available
>
{: .hands_on}

The output is a web page with different tabs:

- `Stats`: general statistics on the two annotations.
- `Orthologs`: orthology relations between genes of the two annotations.
- `InterPro` and `PFAM`: count the number of times each InterPro/PFAM signature is identified in each annotation. You can download a PDF file at the top, showing an {NMDS} analysis of this data.
- `Merops` and `CAZymes`: count the number of members of each Merops/CAZymes family in each annotation. You can download PDF files at the top, showing colorful representations of this data.
- `GO`: list Gene Ontology terms that were found to be over or under represented in each annotation.

> <question-title></question-title>
>
> What can you tell from this report?
>
> > <solution-title></solution-title>
> >
> > The alternate annotation contains less genes than the one you generated, and misses a lot of functional annotations. It confirms that this alternate annotation have a lower quality than the one you performed.
> {: .solution}
>
{: .question}

# Conclusion


Congratulations for reaching the end of this tutorial! Now you know how to perform a structural and functional annotation of a new eukaryotic genome, using Funannotate, EggNOG mapper and InterProScan. You also learned how Funannotate can help you in the submission process to NCBI. And you learned how to visualise your new annotation using JBrowse, and how to compare it with another annotation.

An automatic annotation of an eukaryotic genome is unfortunately rarely perfect. If you inspect some predicted genes (or look at the `tbl2asn genome validation report` output of Funannotate), you may find some mistakes made by Funannotate, or potential problems, e.g. wrong exon/intron limits, splitted genes, or merged genes. Setting up a manual curation project using [Apollo](http://genomearchitect.org/) can help a lot to manually fix these errors. Check out the [Apollo tutorial]({% link topics/genome-annotation/tutorials/apollo-euk/tutorial.md %}) for more details.

tools_json:
["RNA STAR", "toolshed.g2.bx.psu.edu/repos/iuc/rgrnastar/rna_star/2.7.8a+galaxy0", "MergeSamFiles", "toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_MergeSamFiles/2.18.2.1", "Funannotate predict annotation", "toolshed.g2.bx.psu.edu/repos/iuc/funannotate_predict/funannotate_predict/1.8.15+galaxy1", "Funannotate functional", "toolshed.g2.bx.psu.edu/repos/iuc/funannotate_annotate/funannotate_annotate/1.8.15+galaxy1", "Busco", "toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.4.6+galaxy0", "JBrowse", "toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1", "AEGeAn ParsEval", "toolshed.g2.bx.psu.edu/repos/iuc/aegean_parseval/aegean_parseval/0.16.0", "Funannotate compare", "toolshed.g2.bx.psu.edu/repos/iuc/funannotate_compare/funannotate_compare/1.8.15+galaxy1"]


topic: genome-annotation
tutorial: gene-centric
title: Comparative gene analysis in unannotated genomes
difficulty: intermediate
time_estimation: 30M
requirements_json: []
content_hash: 0ba8180122a89b5b15c814e626fa7f29
last_modified: 2026-05-27T09:01:48.985913
zenodo_link: https://zenodo.org/record/7034885
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/gene-centric/tutorial.html

content:
Despite the rapidly increasing number of fully assembled genomes few genomes are well annotated. This is especially true for large eukaryotic genomes with their complex gene structure and abundance of pseudogenes. And of course do not forget about the [Murthy's law](https://en.wikipedia.org/wiki/Murphy%27s_law): if you are interested in a particular gene the chances are that it will not be annotated in your genome of interest. In this tutorial we will demonstrate how to compare gene structures across a set of vertebrate genomes. So ...

> > <code-in-title>What I have</code-in-title>
> > - I know the gene's name
> > - I know which species I'm interested in
> > - I know where to find genomes of these species
> {: .code-in}
>
> > <code-out-title>What I want</code-out-title>
> > - I work with a gene _X_
> > - I would like to compare the structure of gene _X_ across _N_ genomes
> {: .code-out}
>
{: .code-2col}

> <tip-title>What I will get</tip-title>
> - Interactive graphs showing location of the gene across your species of choice. These will allow you to see the absence/presence of the genes across genomes, to detect potential duplications, predogenization events, re-arrangements etc.
> - Phylogenetic trees for individual exons of the gene. The trees will give you an idea of potential unusual evolutionary dynamics for the gene.
{: .tip}

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Logic

The analysis follow the following logic (also see the following figure):

1. Pick a gene and select genomes to analyze
1. Get amino acid translations for all exons of my gene of interest
1. Identify genomes of interest
1. Extract amino acid sequences and genome coordinates for all possible open reading frames (ORFs) from all genomes in my set

    The following steps 5, 6, and 7 are performed using a single Galaxy workflow (grey area in the following figure).

1. Align - Find matches between exons of the gene of interest and ORFs
1. Intersect - Compute genome coordinates of matches
1. Build phylogenetic trees for individual exons

    The final step is performed using Jupyter notebook

1. Create a comparative plot showing distribution of matches across genomes of interest

![A schematic diagram of the above workflow](../../images/gene-centric/gene_analysis.svg)

------

# Example history

This [example history](https://usegalaxy.org/u/cartman/h/genecomparisonxbp1) contains results of the analysis described in this tutorial.

{% snippet faqs/galaxy/histories_import.md %}

You can use this history to understand the input datasets as well as outputs of the entire analysis. The key items in the history are labelled with <kbd>tags</kbd>:

> <code-in-title>dataset in the example history</code-in-title>
> - <kbd>EXONS</kbd> - amino acid translation of exons of the gene of interest (*XBP-1*)
> - <kbd>ORF_BED</kbd> - coordinates of predicted ORFs in the genomes of interest
> - <kbd>DiamondDB</kbd> - database and amino acid translations of predicted ORFs in the genomes of interest
{: .code-in}

> <code-out-title>in the sample history</code-out-title>
> - <kbd>PlottingData</kbd> - summary necessary for plotting comparative genome graphs
> - <kbd>Trees</kbd> - phylogenetic trees for each exon
{: .code-out}

> <warning-title>A suggestion!</warning-title>
> Importing and looking around this history is very helpful for understanding how this analysis works!
{: .warning}

-------

# Practice

## Step 1: Pick a gene and select genomes to analyze

In this example we will compare structure of X-box protein 1 gene (*XBP-1*) across a set of five vertebrate genomes sequenced by [VGP](https://vertebrategenomesproject.org/) consortium ({% cite Rhie2021-ks %}).

## Step 2: Get amino acid translations for all exons of my gene of interest

This step is manual and is performed outside Galaxy.

The best annotated vertebrate is ... human. To obtain amino acid translation of individual exons of *XBP-1* you can use [UCSC Genome browser](https://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&lastVirtModeType=default&lastVirtModeExtraState=&virtModeType=default&virtMode=0&nonVirtPosition=&position=chr22%3A28794555%2D28800597&hgsid=1433476153_N1Iet0RvAaPQIpPuVl2qbi74BrC3). The "UCSC annotations of RefSeq RNAs (NM_\* and NR_\*)" track shows two alternative transcripts produced by *XBP-1* : spliced and unspliced (here 'spliced' and 'unspliced' do not refer to the "normal' splicing process. To learn more about this fascinating gene read here).

![XBP-1](../../images/gene-centric/xbp-1.svg)

Clicking on the names of the transcripts open a UCSC page specific to that transcript. From there you can obtain genomic sequences corresponding to all exons. You can then translate these sequences using any available online tool (e.g., [NCBI ORFinder](https://www.ncbi.nlm.nih.gov/orffinder/)).

In this particular case we did not create translation of all exons. Instead, we created translation of exon 1  and two terminal exons. The first exon is shared between the two transcripts. It will allow us to anchor the beginning of the gene. The two terminal exons are different between the two transcripts. Because of this we create two alternative translation per exon: `s-p2` and `s-p12` for the spliced version and `u-p1` and `u-p2` for unspliced transcript. A FASTA file containing all translation is shown below:

```
>xbp-1u-p1
GNEVRPVAGSAESAALRLRAPLQQVQAQLSPLQNISPWILAVLTLQIQ
>xbp-1u-p2
LISCWAFWTTWTQSCSSNALPQSLPAWRSSQRSTQKDPVPYQPPFLCQWG
RHQPSWKPLMN
>xbp-1s-p12
GNEVRPVAGSAESAAGAGPVVTPPEHLPMDSGGIDSSDSE
>xbp-1s-p3
SDILLGILDNLDPVMFFKCPSPEPASLEELPEVYPEGPSSLPASLSLSVG
TSSAKLEAINELIRFDHIYTKPLVLEIPSETESQANVVVKIEEAPLSPSE
NDHPEFIVSVKEEPVEDDLVPELGISNLLSSSHCPKPSSCLLDAYSDCGY
GGSLSPFSDMSSLLGVNHSWEDTFANELFPQLISV
>xbp1_ex1
MVVVAAAPNPADGTPKVLLLSGQPASAAGAPAGQALPLMVPAQRGASPEA
ASGGLPQARKRQRLTHLSPEEKALR
```

## Step 3: Identify and upload genomes of interest

This tutorial has been initially designed for the analysis of data produced by the Vertebrate Genome Consortium [VGP](https://vertebrategenomesproject.org/). However, it is equally suitable for any genomic sequences (including prokaryotic ones).

In this section we first show how to upload sample datasets. These datasets were intentionally made small. All subsequent steps of this tutorial are performed using these sample data. We then demonstrate how to upload full size genomes from the VGP data repository called [GenomeArk](https://vgp.github.io/genomeark/).

### Uploading sample data

> <hands-on-title>Sample Data upload</hands-on-title>
>
> - Create a new history for this tutorial
> - Import the files from [Zenodo]({{ page.zenodo_link }}) using the following URLs:
>
>   ```
>   https://zenodo.org/record/7034885/files/aGasCar1.fa
>   https://zenodo.org/record/7034885/files/bTaeGut2.fa
>   https://zenodo.org/record/7034885/files/fScoJap1.fa
>   https://zenodo.org/record/7034885/files/mCynVol1.fa
>   https://zenodo.org/record/7034885/files/mHomSapT2T.fa
>   ```
>   {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}

These correspond to fragments of genomes from:

 - `aGasCar1` - _Gastrophryne carolinensis_ (Eastern narrow-mouth toad)
 - `bTaeGut` - _Taeniopygia guttata_ (Zebra finch)
 - `fScoJap` - _Scomber japonicus_ (Chub mackerel)
 - `mCynVol` - _Cynocephalus volans_ (Philippine flying lemur)
 - `mHomSap` - _Homo sapiens_ (Human)

### Uploading VGP data from GenomeArk

Galaxy provides a direct connection to [GenomeArk](https://vgp.github.io/genomeark/) from its "Upload Data" tool. To access GenomeArk data you need to:

> <hands-on-title>Upload VGP data from GenomeArk</hands-on-title>
>
> - Create a new history for this tutorial
> - Import genome assembly FASTA files from GenomeArk:
>
>    - Open the file {% icon galaxy-upload %} __Upload Data__ menu
>    - Click on __Choose remote files__ button at the bottom of the Upload interface
>    - Choose __Genome Ark__
>    - Pick assembles you would like to include in your analysis (use Search box on top)
>
>
> Generally for a given species you want the assembly with the highest version number. For example, Human (_Homo sapiens_) had three assemblies at the time of writing: `mHomSap1`, `mHomSap2`, and `mHomSap3`. Pick `mHomSap3` in this situation.
>
> Inside each assembly folder you will see typically see different sub-folders. The one you should generally be interested in will be called `assembly_curated`. Inside that folder you will typically see separate assemblies for maternal (indicated with `mat`) and paternal (indicated with `pat`) haplotypes. In the analyses here we are choosing as assembly representing heterogametic haplotype (`pat` in the case of Human).
>
> - Repeat this process for all assemblies you are interested in.
>
{: .hands_on}

### Combine uploaded sequences into a dataset collection

After uploading data from GenomeArk you will end up with a number of FASTA datasets in your history. To proceed with the analysis you will need to combine these into a single dataset collection:

> <hands-on-title>Combine genome assemblies into a dataset collection</hands-on-title>
>
> To create a collection from genome assemblies you just uploaded into GenomeArk follow the steps in the video below. Obviously, you only want to click check-boxes next to FASTA files (assemblies) you want to include and leave everything else out.
>
> {% snippet faqs/galaxy/collections_build_list.md %}
>
{: .hands_on}

## Step 4: Extract amino acid sequences and genome coordinates for all ORFs

### Extract ORFs from genomic sequences

Now we can go ahead and find all ORFs in all genome assemblies we have bundled into a collection. For this we will use tool called [ORFiPy](https://github.com/urmi-21/orfipy):

> <hands-on-title>Find ORFs witrh ORFiPy</hands-on-title>
>
> Run {% tool [ORFiPy](toolshed.g2.bx.psu.edu/repos/iuc/orfipy/orfipy/0.0.4+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Find ORFs in"*: The collection containing genome assemblies we just created (click "Dataset collection" {% icon param-collection %} button on left-side of this input field)
>    - *"Select options"*: check the box next to `BED` and `Peptides`
>    - *"Minimum length of ORFs"*: type the number `99`. We do not want ORFs to be too short as their number will be too high
>    - *"Output ORFs bound by Stop codons?"*: set this option to `Yes`.
>
{: .hands_on}

Because this operation is performed on a dataset collection. it will produce a dataset collection as an output. Actually, it will produce two because we selected `BED` and `Peptides` options. One collection will contain amino acid sequences for all ORFs (it will typically be called `ORFs on collection ... (FASTA format)`) and another their coordinates (called `ORFs on collection ... (BED format)`)
)

This will produce two new dataset collections in your history: one containing coordinates of ORFs and the other containing their amino acid translations. Because genomes are large these will likely be tens of millions of ORFs identified this way.

### Creating __Diamond__ database

Because we will be using the [Diamond](https://github.com/bbuchfink/diamond) tool to find matches between our gene of interest and ORF translations we need to convert FASTA files into Diamond database using __Diamond makedb__ tool:

> <hands-on-title>Create Diamond database</hands-on-title>
>
> Run {% tool [Diamond makedb](toolshed.g2.bx.psu.edu/repos/bgruening/diamond/bg_diamond_makedb/2.0.15+galaxy0) %} on the collection containing amino acid translations of ORFs generated using __ORFiPy__:
>    - {% icon param-collection %} "Input reference file in FASTA format"*: Select collections containing amino acid (FASTA Protein) output of __ORFiPy__
>
{: .hands_on}

At this point we have three input datasets that would allow us to find and visualize matches between the gene of interest and genome asseblies:

1. Diamond database of ORF translations from genome assemblies
2. Coordinates and frame information about ORFs in BED format
3. Amino acid translation of exons from the gene of interest

## Steps 5, 6, and 7: Finding matches and building trees

To find location of genes, [we will use the following workflow]({% link topics/genome-annotation/tutorials/gene-centric/workflows/index.md %}) that is available as a part of this tutorial. To use this workflow you need to import it into your Galaxy instance.

![WF](../../images/gene-centric/wf.png)

The workflow takes three inputs:

> <code-in-title>Workflow inputs</code-in-title>
>
> 1. <kbd>EXONS</kbd> - Amino acid translation of exons from the gene of interest ([Step 2](#step-2-get-amino-acid-translations-for-all-exons-of-my-gene-of-interest) of this tutorial)
> 1. <kbd>DiamondDB</kbd> - Diamond database of ORF translations from genome assemblies ([Step 4](#step-4-extract-amino-acid-sequences-and-genome-coordinates-for-all-orfs) of this tutorial)
> 1. <kbd>ORF BED</kbd> - Coordinates and frame information about ORFs in BED format ([Step 4](#step-4-extract-amino-acid-sequences-and-genome-coordinates-for-all-orfs)  of this tutorial)
>
{: .code-in}

It produces two primary outputs:

> <code-out-title>Results</code-out-title>
>
> 1. <kbd>Trees</kbd> - Phylogenetic trees for each input exon as [Newick](https://en.wikipedia.org/wiki/Newick_format) file
> 1. <kbd>PlottingData</kbd> - A summary table of exon matches for each genome
>
{: .code-out}

The overall logic of the workflow is as follows:

1. Perform [__Diamond__](https://github.com/bbuchfink/diamond) search to identify matches between the gene of interest (<kbd>EXONS</kbd>) and ORF sets from each genomes (<kbd>DiamondDB</kbd>).
2. Intersect information about the matches with BED file containing ORF coordinates (<kbd>ORF BED</kbd>). This allows us to know genomic position of the ORFs and their frames (1, 2, 3 or -1, -2, -3).
3. Extract matching parts of the sequences and generated multiple alignments using [__MAFFT__](https://mafft.cbrc.jp/alignment/software/). This is done for each amino acid sequence in <kbd>EXONS</kbd>. Thus in the case of *XBP-1* there will be five sets of alignments.
4. Build phylogenetic tree for each alignment produced in the previous step using the [Neighbor Joining method](https://en.wikipedia.org/wiki/Neighbor_joining) - a simple and quick way to obtain a rough phylogeny.
5. Use information from step 1 to compute genome coordinates of matches. This is done by combining the information about genomic positions of ORFs from the <kbd>ORF BED</kbd> file and local alignment information generated during step 1.

## Step 7: Looking at the trees

After running the workflow phylogenetic trees will be saved into a collection named `Join neighbors on ...`. This collection will also be labelled with tag <kbd>Trees</kbd>. To visualize the trees:

> <hands-on-title>Visualize the trees</hands-on-title>
>
> - Expand the collection by clicking on it
> - Click on any dataset
> - Click the {% icon galaxy-barchart %} icon:
>     ![Graph icon](../../images/gene-centric/display_tree.png)
> - A selection of tree visualization tools will appear in the center pane of the interface
> - Select `Phylogenetic Tre Visualization` and play with it:
>     ![The tree](../../images/gene-centric/tree.png)
{: .hands_on}

## Step 8: Generating and interpreting the comparative genome graph

Another workflow output will represent a single file summarizing genomic location of matches between each of the genomes in our dataset and amino acid translation of exons from the gene of interest. It will be called `Mapping report` and will have tag <kbd>PlottingData</kbd> associated with it. To plot the data contained in this file we will use external Jupyter notebook (note that Jupyter can be run directly from Galaxy, but to make this tutorial runnable on any Galaxy instance we will use an internal notebook server).

### Starting notebook

{% include _includes/cyoa-choices.html option1="UseGalaxy.org" option2="Galaxy Europe, Australia, and Others" default="UseGalaxy.org"
       text="If you are on a server with Interactive Tool support (e.g. Galaxy Europe, Australia, and many others), you can choose that option for a more integrated Jupyter Notebook experience." %}


<div class="UseGalaxyorg" markdown="1">

The notebook can be accessed [from here](https://colab.research.google.com/drive/1smTpRejBb7c02LiIxMPNVDjOucLAsD81?usp=sharing). You do need to have a Google Account to be able to use the notebook.

</div>
<div class="Galaxy-Europe-Australia-and-Others" markdown="1">

{% snippet faqs/galaxy/interactive_tools_jupyter_launch.md %}
{% capture ipynbpath %}{{ site.url }}{{ site.baseurl }}/{{ page.path | replace: "tutorial.md", "gene_image.ipynb" }}{% endcapture %}

1. Once the notebook has started, open a Terminal in Jupyter **File → New → Terminal**

2. Run the following command:

   ```
   wget {{ ipynbpath }}
   ```

3. Switch to the Notebook that appears in the list of files on the left

</div>


### Proving data

The data is provided to the notebook by setting the `dataset_url` parameter in this cell:

```python
# Paste link to the dataset here
dataset_url = 'https://usegalaxy.org/api/datasets/f9cad7b01a472135a8abd43f91f8d3cf/display?to_ext=tabular'
```

This is a URL pointing to one of the workflow outputs: `Mapping report` with the <kbd>PlottingData</kbd> tag. To copy the URL click on the {% icon galaxy-link %} icon adjacent to the dataset:

![Copying the link](../../images/gene-centric/copy_link.png)

Running the notebook will generate two graphs explained in the next section.

## Interpreting the graphs

Analysis of sample data associated with this tutorial will produce the genome graph shown below. In this graph the Y-axis represents ORFs on positive (1, 2, 3 in red color) and negative (-1, -2, -3 in blue color) strands. The X-axis is genomic coordinates.  Boxes represent matches between amino acid sequences of exons and ORFs they are superimposed to. The color of boxes reflect the extent of amino acid identity. The color key is shown in the left upper corner of the plot. The image is interactive so you can zoom in and out.

<html>
<head>
  <script src="https://cdn.jsdelivr.net/npm/vega@5.22.1"></script>
  <script src="https://cdn.jsdelivr.net/npm/vega-lite@5.2.0"></script>
  <script src="https://cdn.jsdelivr.net/npm/vega-embed@6.20.8"></script>
</head>
<body>
  <div id="vis"/>
  <script>
    const spec = {
  "config": {
    "view": {"continuousWidth": 400, "continuousHeight": 300},
    "title": {"anchor": "start", "fontSize": 10}
  },
  "vconcat": [
    {
      "layer": [
        {
          "mark": {"type": "rule", "strokeWidth": 2},
          "encoding": {
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {
              "field": "start",
              "scale": {"domain": [52, 10726]},
              "title": null,
              "type": "quantitative"
            },
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "selection": {
            "selector001": {
              "type": "interval",
              "bind": "scales",
              "encodings": ["x", "y"]
            }
          },
          "title": "aGasCar1.fascaffold_1"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "start", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "aGasCar1.fascaffold_1"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "end", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "aGasCar1.fascaffold_1"
        },
        {
          "mark": {"type": "rule", "opacity": 0.9, "strokeWidth": 10},
          "encoding": {
            "color": {
              "field": "id",
              "scale": {"scheme": "blueorange"},
              "type": "quantitative"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "matchStart", "type": "quantitative"},
            "x2": {"field": "matchEnd"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "aGasCar1.fascaffold_1"
        },
        {
          "mark": {"type": "text", "yOffset": -10},
          "encoding": {
            "color": {"value": "black"},
            "text": {"field": "exon", "type": "nominal"},
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "midpoint", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "aGasCar1.fascaffold_1"
        }
      ],
      "data": {"name": "data-4c922bb95694b1514f353a1acd2ab409"},
      "height": 100,
      "width": 800
    },
    {
      "layer": [
        {
          "mark": {"type": "rule", "strokeWidth": 2},
          "encoding": {
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {
              "field": "start",
              "scale": {"domain": [-228, 3860]},
              "title": null,
              "type": "quantitative"
            },
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "selection": {
            "selector002": {
              "type": "interval",
              "bind": "scales",
              "encodings": ["x", "y"]
            }
          },
          "title": "bTaeGut2.fascaffold_18"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "start", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "bTaeGut2.fascaffold_18"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "end", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "bTaeGut2.fascaffold_18"
        },
        {
          "mark": {"type": "rule", "opacity": 0.9, "strokeWidth": 10},
          "encoding": {
            "color": {
              "field": "id",
              "scale": {"scheme": "blueorange"},
              "type": "quantitative"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "matchStart", "type": "quantitative"},
            "x2": {"field": "matchEnd"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "bTaeGut2.fascaffold_18"
        },
        {
          "mark": {"type": "text", "yOffset": -10},
          "encoding": {
            "color": {"value": "black"},
            "text": {"field": "exon", "type": "nominal"},
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "midpoint", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "bTaeGut2.fascaffold_18"
        }
      ],
      "data": {"name": "data-cd3fa6c014d625fc2f85cfeb0119b9e0"},
      "height": 100,
      "width": 800
    },
    {
      "layer": [
        {
          "mark": {"type": "rule", "strokeWidth": 2},
          "encoding": {
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {
              "field": "start",
              "scale": {"domain": [314, 3976]},
              "title": null,
              "type": "quantitative"
            },
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "selection": {
            "selector003": {
              "type": "interval",
              "bind": "scales",
              "encodings": ["x", "y"]
            }
          },
          "title": "fScoJap1.fascaffold_13"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "start", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "fScoJap1.fascaffold_13"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "end", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "fScoJap1.fascaffold_13"
        },
        {
          "mark": {"type": "rule", "opacity": 0.9, "strokeWidth": 10},
          "encoding": {
            "color": {
              "field": "id",
              "scale": {"scheme": "blueorange"},
              "type": "quantitative"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "matchStart", "type": "quantitative"},
            "x2": {"field": "matchEnd"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "fScoJap1.fascaffold_13"
        },
        {
          "mark": {"type": "text", "yOffset": -10},
          "encoding": {
            "color": {"value": "black"},
            "text": {"field": "exon", "type": "nominal"},
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "midpoint", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "fScoJap1.fascaffold_13"
        }
      ],
      "data": {"name": "data-33f25281b1f78907be251e80305165ce"},
      "height": 100,
      "width": 800
    },
    {
      "layer": [
        {
          "mark": {"type": "rule", "strokeWidth": 2},
          "encoding": {
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {
              "field": "start",
              "scale": {"domain": [-71, 6632]},
              "title": null,
              "type": "quantitative"
            },
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "selection": {
            "selector004": {
              "type": "interval",
              "bind": "scales",
              "encodings": ["x", "y"]
            }
          },
          "title": "mHomSapT2T.fachr22"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "start", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mHomSapT2T.fachr22"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "end", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mHomSapT2T.fachr22"
        },
        {
          "mark": {"type": "rule", "opacity": 0.9, "strokeWidth": 10},
          "encoding": {
            "color": {
              "field": "id",
              "scale": {"scheme": "blueorange"},
              "type": "quantitative"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "matchStart", "type": "quantitative"},
            "x2": {"field": "matchEnd"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mHomSapT2T.fachr22"
        },
        {
          "mark": {"type": "text", "yOffset": -10},
          "encoding": {
            "color": {"value": "black"},
            "text": {"field": "exon", "type": "nominal"},
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "midpoint", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mHomSapT2T.fachr22"
        }
      ],
      "data": {"name": "data-b9417f0d40ef923551b8b4d9ff9101ea"},
      "height": 100,
      "width": 800
    },
    {
      "layer": [
        {
          "mark": {"type": "rule", "strokeWidth": 2},
          "encoding": {
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {
              "field": "start",
              "scale": {"domain": [315, 2819]},
              "title": null,
              "type": "quantitative"
            },
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "selection": {
            "selector005": {
              "type": "interval",
              "bind": "scales",
              "encodings": ["x", "y"]
            }
          },
          "title": "mCynVol1.fascaffold_2"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "start", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mCynVol1.fascaffold_2"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "end", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mCynVol1.fascaffold_2"
        },
        {
          "mark": {"type": "rule", "opacity": 0.9, "strokeWidth": 10},
          "encoding": {
            "color": {
              "field": "id",
              "scale": {"scheme": "blueorange"},
              "type": "quantitative"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "matchStart", "type": "quantitative"},
            "x2": {"field": "matchEnd"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mCynVol1.fascaffold_2"
        },
        {
          "mark": {"type": "text", "yOffset": -10},
          "encoding": {
            "color": {"value": "black"},
            "text": {"field": "exon", "type": "nominal"},
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "midpoint", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mCynVol1.fascaffold_2"
        }
      ],
      "data": {"name": "data-4ee088ff88102dae61c31ea38e9d47e2"},
      "height": 100,
      "width": 800
    },
    {
      "layer": [
        {
          "mark": {"type": "rule", "strokeWidth": 2},
          "encoding": {
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {
              "field": "start",
              "scale": {"domain": [-74, 5866]},
              "title": null,
              "type": "quantitative"
            },
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "selection": {
            "selector006": {
              "type": "interval",
              "bind": "scales",
              "encodings": ["x", "y"]
            }
          },
          "title": "mCynVol1.fascaffold_3"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "start", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mCynVol1.fascaffold_3"
        },
        {
          "mark": {"type": "point", "shape": "arrow", "strokeWidth": 2},
          "encoding": {
            "angle": {
              "condition": {"value": 270, "test": "(datum['strand'] === '-')"},
              "value": 90
            },
            "color": {
              "condition": {
                "value": "blue",
                "test": "(datum['strand'] === '-')"
              },
              "value": "red"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "end", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mCynVol1.fascaffold_3"
        },
        {
          "mark": {"type": "rule", "opacity": 0.9, "strokeWidth": 10},
          "encoding": {
            "color": {
              "field": "id",
              "scale": {"scheme": "blueorange"},
              "type": "quantitative"
            },
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "matchStart", "type": "quantitative"},
            "x2": {"field": "matchEnd"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mCynVol1.fascaffold_3"
        },
        {
          "mark": {"type": "text", "yOffset": -10},
          "encoding": {
            "color": {"value": "black"},
            "text": {"field": "exon", "type": "nominal"},
            "tooltip": [
              {"field": "orf", "title": "ORF id", "type": "nominal"},
              {"field": "genome", "title": "Genome", "type": "nominal"},
              {"field": "chr", "title": "Chromosome", "type": "nominal"},
              {"field": "id", "title": "% identity", "type": "nominal"}
            ],
            "x": {"field": "midpoint", "type": "quantitative"},
            "x2": {"field": "end"},
            "y": {"field": "frame", "title": null, "type": "nominal"}
          },
          "title": "mCynVol1.fascaffold_3"
        }
      ],
      "data": {"name": "data-729a5f2f10472b0a903d29284fd22ba5"},
      "height": 100,
      "width": 800
    }
  ],
  "resolve": {"scale": {"y": "shared"}},
  "$schema": "https://vega.github.io/schema/vega-lite/v4.17.0.json",
  "datasets": {
    "data-4c922bb95694b1514f353a1acd2ab409": [
      {
        "genome": "aGasCar1.fa",
        "chr": "scaffold_1",
        "start": 552,
        "end": 684,
        "orf": "aGasCar1.fascaffold_1_ORF.2",
        "frame": 1,
        "strand": "+",
        "midpoint": null,
        "matchStart": null,
        "matchEnd": null,
        "exon": null,
        "id": null,
        "genome_chr": "aGasCar1.fascaffold_1",
        "clade": "a"
      },
      {
        "genome": "aGasCar1.fa",
        "chr": "scaffold_1",
        "start": 762,
        "end": 1029,
        "orf": "aGasCar1.fascaffold_1_ORF.3",
        "frame": 1,
        "strand": "+",
        "midpoint": null,
        "matchStart": null,
        "matchEnd": null,
        "exon": null,
        "id": null,
        "genome_chr": "aGasCar1.fascaffold_1",
        "clade": "a"
      },
      {
        "genome": "aGasCar1.fa",
        "chr": "scaffold_1",
        "start": 8880,
        "end": 9045,
        "orf": "aGasCar1.fascaffold_1_ORF.31",
        "frame": 1,
        "strand": "+",
        "midpoint": "8970",
        "matchStart": "8904",
        "matchEnd": "9036",
        "exon": "xbp-1u-p1",
        "id": "75.6",
        "genome_chr": "aGasCar1.fascaffold_1",
        "clade": "a"
      },
      {
        "genome": "aGasCar1.fa",
        "chr": "scaffold_1",
        "start": 8880,
        "end": 9045,
        "orf": "aGasCar1.fascaffold_1_ORF.31",
        "frame": 1,
        "strand": "+",
        "midpoint": "8923",
        "matchStart": "8904",
        "matchEnd": "8943",
        "exon": "xbp-1s-p12",
        "id": "85.7",
        "genome_chr": "aGasCar1.fascaffold_1",
        "clade": "a"
      },
      {
        "genome": "aGasCar1.fa",
        "chr": "scaffold_1",
        "start": 9270,
        "end": 9432,
        "orf": "aGasCar1.fascaffold_1_ORF.32",
        "frame": 1,
        "strand": "+",

tools_json:
["ORFiPy", "toolshed.g2.bx.psu.edu/repos/iuc/orfipy/orfipy/0.0.4+galaxy0", "Diamond makedb", "toolshed.g2.bx.psu.edu/repos/bgruening/diamond/bg_diamond_makedb/2.0.15+galaxy0"]

tags_json:
["evolution", "annotation", "vgp", "cookbook", "eukaryote", "prokaryote", "microgalaxy", "biodiversity"]


topic: genome-annotation
tutorial: annotation-with-maker
title: Genome annotation with Maker
difficulty: advanced
time_estimation: 4h
content: {% include topics/genome-annotation/tutorials/annotation-with-maker/content.md short=false %}
tools_json: []
requirements_json: []
tags_json: ["gmod", "eukaryote", "jbrowse1", "maker"]
content_hash: 3ac46864e10bda17a1cfd1c2c83e9061
last_modified: 2026-05-15T22:28:13.462988
zenodo_link: https://doi.org/10.5281/zenodo.4406623
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/annotation-with-maker/tutorial.html


topic: genome-annotation
tutorial: genome-annotation
title: Genome Annotation
difficulty: introductory
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: ["prokaryote"]
content_hash: f1f384d1e62e05ae1d954fa1a7c2b569
last_modified: 2026-05-15T22:28:13.507976
zenodo_link: https://doi.org/10.5281/zenodo.1250793
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/genome-annotation/tutorial.html

content:
Genome annotation is the process of attaching biological information to sequences.
It consists of three main steps:

 - identifying portions of the genome that do not code for proteins
 - identifying elements on the genome, a process called gene prediction, and
 - attaching biological information to these elements.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Introduction into File Formats

## FASTA

DNA and protein sequences are written in FASTA format where you have in the first line a ">" followed by the description. In the second line the sequence starts.

![FASTA file](../../images/fasta_format.png)

## GFF3

The general feature format (gene-finding format, generic feature format, GFF) is a file format used for describing genes and other features of DNA, RNA and protein sequences.

<img src="../../images/gff3_format.png" alt="GFF3 overview" width="70%">

## GENBANK

The genbank sequence format is a rich format for storing sequences and associated annotations.

![genbank file](../../images/gb_full.png)

# Structural Annotation

For the genome annotation we use a piece of the *Aspergillus fumigatus* [genome sequence](https://zenodo.org/record/1250793/files/Aspergillus_sequence.fasta) as input file.

## Sequence Features

First we want to get some general information about our sequence.

> <hands-on-title>Sequence composition</hands-on-title>
>
> 1. Count the number of bases in your sequence (**compute sequence length**)
> 2. Check for sequence composition and GC content (**geecee**).
> 3. Plot the sequence composition as bar chart.
>
> <img src="../../images/barchart_sequencecomposition.png" alt="Bar chart output of the sequence" width="30%">
>
{: .hands_on}

## Gene Prediction

At first you need to identify those structures of the genome which code for proteins. This step of annotation is called “structural annotation”. It contains the identification and location of open reading frames (ORFs), identification of gene structures and coding regions, and the location of regulatory motifs. Galaxy contains several tools for the structural annotation. Tools for gene prediction are **Augustus** (for eukaryotes and prokaryotes) and **glimmer3** (only for prokaryotes).

> <hands-on-title>Gene prediction</hands-on-title>
>
> We use **Augustus** for gene prediction.
> 1. Use the genome sequence (FASTA file) as input.
> 2. Choose the right *model organism*, *gff* format output.
> 3. Select all possible output options.
>
> ![augustus](../../images/augustus.png)
>
> Augustus will provide three output files: *gff3*, *coding sequences* (CDS) and *protein sequences*.
>
>    > <question-title></question-title>
>    >
>    > How many genes are predicted?
>    >
>    > > <solution-title></solution-title>
>    > > Check the output: [augustus_output](../../images/augustus_out.png)
>    > {: .solution }
> {: .question}
>
{: .hands_on}

> <hands-on-title>tRNA and tmRNA Prediction</hands-on-title>
>
> Use **Aragorn** for tRNA and tmRNA prediction.
> 1. As input file use the *Aspergillus* genome sequence. You can choose the genetic code (e.g. bacteria).
> 2. Select the topology of your genome (circular or linear).
>
>    > <question-title></question-title>
>    >
>    > Are there tRNAs or tmRNAs in the sequence?
>    >
>    {: .question}
>
{: .hands_on}

> <details-title>Aragorn in depth</details-title>
>
> read more about **Aragorn** [in this publication](https://nar.oxfordjournals.org/content/32/1/11.full.pdf+html).
{: .details}

# Functional Annotation

## Similarity Searches (BLAST)

Functional gene annotation means the description of the biochemical and biological function of proteins. Possible analyses to annotate genes can be for example:

- similarity searches
- gene cluster prediction for secondary metabolites
- identification of transmembrane domains in protein sequences
- finding gene ontology terms
- pathway information

For similarity searches we use *NCBI BLAST+ blastp* to find similar proteins in a protein database.

> <hands-on-title> Similarity search</hands-on-title>
>
> 1. {% icon tool %} As input file, select the protein sequences from Augustus.
> 2. Choose the protein BLAST database *SwissProt* and the output format *xml*.
>
> <img src="../../images/blastP.png" alt="blastp tool interface and parameters" width="70%">
>
> 3. Parsing the xml output (**Parse blast XML output**) results in changing the format style into tabular.
>
>    > <question-title></question-title>
>    >
>    > What information do you see in the BLAST output?
>    >
>    {: .question}
>
>
> From BLAST search results we want to get only the best hit for each protein.
> 4. {% icon tool %} Therefore apply the tool **BLAST top hit descriptions** with *number of descriptions =1* on the xml output file.
>
>    > <question-title></question-title>
>    >
>    > For how many proteins we do not get a BLAST hit?
>    >
>    {: .question}
>
> 5. {% icon tool %} Choose the tool **Select lines that match an expression** and enter the following information: *Select lines from* [select the BLAST top hit descriptions result file]; *that* [not matching]; *the pattern* [gi].
>
>    <img src="../../images/selectlines.png" alt="Select lines that match an expression tool interface and parameters" width="50%">
>
>    > <comment-title>Results file</comment-title>
>    > The result file will contain all proteins which do not have an entry in the second column and therefore have no similar protein in the SwissProt database.
>    {: .comment}
>
>    > <comment-title>Obtaining unannotated proteins for analysis</comment-title>
>    > For functional description of those proteins we want to search for motifs or domains which may classify them more. To get a protein sequence FASTA file with only the not annotated proteins, use the tool **Filter sequences by ID from a tabular file** and select for *Sequence file to filter on the identifiers* [Augustus protein sequences] and for *Tabular file containing sequence identifiers* the protein file with not annotated sequences. The output file is a FASTA file with only those sequences without description.
>    {: .comment}
>
{: .hands_on}

This file will be the input for more detailed analysis:

- **Interproscan** is a functional prediction tool. Select all applications and run it on your protein file.

- **WolfPSort** predicts eukaryote protein subcellular localization. Filter the result file for the best ranked localization hit. Use **Filter data on any column using simple expressions** with *c4==1*. The parameter c4==1 means: filter and keep all results where in column 4 is a "1".

- **TMHMM** finds transmembrane domains in protein sequences. The number of amino acids in transmembrane helices should be >18. This information can be found in column 3. Filter the result file *c3>17.99*.

- **BLAST2GO** maps BLAST results to GO annotation terms.

### BLAST Programs

![BLAST programs](../../images/blastprograms.png)

![BLAST databases](../../images/blast_database.png)

> <details-title>Organism not available in a BLAST database</details-title>
>
> If you have an organism which is not available in a BLAST database, you can use its genome sequence in FASTA file for BLAST searches "sequence file against sequence file". If you need to search in these sequences on a regularly basis, you can create a own BLAST database from the sequences of the organism. The advantage of having a own database for your organism is the duration of the BLAST search which speeds up a lot.
{: .details}

**NCBI BLAST+ makeblastdb** creates a BLAST database from your own FASTA sequence file. Molecule type of input is protein or nucleotide.

> <details-title>Further Reading about BLAST Tools in Galaxy</details-title>
>
> Cock et al. (2015): [NCBI BLAST+ integrated into Galaxy](http://biorxiv.org/content/early/2015/05/04/014043.full-text.pdf+html)
>
> Cock et al. (2013): [Galaxy tools and workflows for sequence analysis with applications in molecular plant pathology](https://peerj.com/articles/167/)
{: .details}

## More Similarity Search Tools in Galaxy

* **VSEARCH**: For processing metagenomic sequences, including searching, clustering, chimera detection, dereplication, sorting, masking and shuffling. VSEARCH stands for vectorized search, as the tool takes advantage of parallelism in the form of SIMD vectorization as well as multiple threads to perform accurate alignments at high speed. VSEARCH uses an optimal global aligner (full dynamic programming Needleman-Wunsch), in contrast to USEARCH which by default uses a heuristic seed and extend aligner. This results in more accurate alignments and overall improved sensitivity (recall) with VSEARCH, especially for alignments with gaps.

> <details-title>`vsearch` in depth</details-title>
>
> Documentation for vsearch available via their [GitHub repository](https://github.com/torognes/vsearch).
{: .details}

* **Diamond**: Diamond is a high-throughput program for aligning a file of short reads against a protein reference database such as NR, at 20,000 times the speed of Blastx, with high sensitivity.

> <details-title>Diamond in depth</details-title>
>
> [Buchfink et al. (2015): Fast and sensitive protein alignment using Diamond.](https://www.nature.com/nmeth/journal/v12/n1/abs/nmeth.3176.html)
{: .details}

* **Kraken**: Kraken BLAST is a highly scalable, extremely fast, commercial, parallelized implementation of the NCBI BLAST application.

## Identification of Gene Clusters

For identification of gene clusters, **antiSMASH** is used. The tool uses genbank file as input files and predicts gene clusters. Output files are a html visualization and the gene cluster proteins.

> <hands-on-title>antiSMASH analysis</hands-on-title>
>
> {% icon tool %} Import this [dataset](../../input_data/Streptomyces_coelicolor_part.genbank) into your Galaxy history and run **antiSMASH** to detect gene clusters. The genbank file contains a part of the *Streptomyces coelicolor* genome sequence.
>
{: .hands_on}

> <question-title></question-title>
>
> Which gene clusters are identified?
>
{: .question}

When you have a whole genome **antiSMASH** analysis, your result may look like this:

<img src="../../images/antismash_full.png" alt="The result of antiSMASH" width="80%">

At the end, you can extract a reproducible workflow out of your history. The workflow should look like this:

![GenomeAnnotation Workflow](../../images/work_flow_Screenshot_from_2015-06-23_09-33-23.png)


topic: genome-annotation
tutorial: bacterial-pangenomics
title: Bacterial pangenomics
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: []
content_hash: e9ac617f92e0ada230bed4759722cb5a
last_modified: 2026-05-27T09:01:48.736155
zenodo_link: https://zenodo.org/records/1
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/bacterial-pangenomics/tutorial.html

content:
General introduction about the topic and then an introduction of the
tutorial (the questions and the objectives). It is nice also to have a
scheme to sum up the pipeline used during the tutorial. The idea is to
give to trainees insight into the content of the tutorial and the (theoretical
and technical) key concepts they will learn.

You may want to cite some publications; this can be done by adding citations to the
bibliography file (`tutorial.bib` file next to your `tutorial.md` file). These citations
must be in bibtex format. If you have the DOI for the paper you wish to cite, you can
get the corresponding bibtex entry using [doi2bib.org](https://doi2bib.org).

With the example you will find in the `tutorial.bib` file, you can add a citation to
this article here in your tutorial like this:
{% raw %} `{% cite Batut2018 %}`{% endraw %}.
This will be rendered like this: {% cite Batut2018 %}, and links to a
bibliography section which will automatically be created at the end of the
tutorial.


**Please follow our
[tutorial to learn how to fill the Markdown]({{ site.baseurl }}/topics/contributing/tutorials/create-new-tutorial-content/tutorial.html)**

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Galaxy and data preparation

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 3. {% tool [Import](upload1) %} the contig file from [Zenodo]({{ page.zenodo_link }}) or from Galaxy shared data libraries:
>
>    ```
>    {{ page.zenodo_link }}/files/DRR187559_contigs.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
{: .hands_on}

# Pangenomics

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [PPanGGOLiN all](toolshed.g2.bx.psu.edu/repos/iuc/ppanggolin_all/ppanggolin_all/2.2.1+galaxy1) %} with the following parameters:
>    - {% icon param-collection %} *"Select genome files"*: Input dataset collection
>
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}

# Multi-sequence alignment

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [PPanGGOLiN MSA](toolshed.g2.bx.psu.edu/repos/iuc/ppanggolin_msa/ppanggolin_msa/2.2.1+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input pangenome h5 file"*: `pangenome_h5` (output of **PPanGGOLiN all** {% icon tool %})
>    - *"Partition"*: `Core`
>
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}

# Conclusion

Sum up the tutorial and the key takeaways here. We encourage adding an overview image of the
pipeline used.

tools_json:
["Import", "upload1", "PPanGGOLiN all", "toolshed.g2.bx.psu.edu/repos/iuc/ppanggolin_all/ppanggolin_all/2.2.1+galaxy1", "PPanGGOLiN MSA", "toolshed.g2.bx.psu.edu/repos/iuc/ppanggolin_msa/ppanggolin_msa/2.2.1+galaxy1"]


topic: genome-annotation
tutorial: bacterial-genome-quality-control
title: Bacterial genome quality control
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: ["microgalaxy"]
content_hash: b882c6aa7dc71cb8321dd0b795a41052
last_modified: 2026-05-27T09:01:48.735967
zenodo_link: https://zenodo.org/records/1
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/bacterial-genome-quality-control/tutorial.html

content:
General introduction about the topic and then an introduction of the
tutorial (the questions and the objectives). It is nice also to have a
scheme to sum up the pipeline used during the tutorial. The idea is to
give to trainees insight into the content of the tutorial and the (theoretical
and technical) key concepts they will learn.

You may want to cite some publications; this can be done by adding citations to the
bibliography file (`tutorial.bib` file next to your `tutorial.md` file). These citations
must be in bibtex format. If you have the DOI for the paper you wish to cite, you can
get the corresponding bibtex entry using [doi2bib.org](https://doi2bib.org).

With the example you will find in the `tutorial.bib` file, you can add a citation to
this article here in your tutorial like this:
{% raw %} `{% cite Batut2018 %}`{% endraw %}.
This will be rendered like this: {% cite Batut2018 %}, and links to a
bibliography section which will automatically be created at the end of the
tutorial.


**Please follow our
[tutorial to learn how to fill the Markdown]({{ site.baseurl }}/topics/contributing/tutorials/create-new-tutorial-content/tutorial.html)**

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Galaxy and data preparation

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 3. {% tool [Import](upload1) %} the contig file from [Zenodo]({{ page.zenodo_link }}) or from Galaxy shared data libraries:
>
>    ```
>    {{ page.zenodo_link }}/files/DRR187559_contigs.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
{: .hands_on}

# Quality control

It comes first a description of the step: some background and some theory.
Some image can be added there to support the theory explanation:

The idea is to keep the theory description before quite simple to focus more on the practical part.

***TODO***: *Consider adding a detail box to expand the theory*

> <details-title> More details about the theory </details-title>
>
> But to describe more details, it is possible to use the detail boxes which are expandable
>
{: .details}

A big step can have several subsections or sub steps:

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [checkm2](toolshed.g2.bx.psu.edu/repos/iuc/checkm2/checkm2/1.0.2+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Input MAG/SAG datasets"*: Input dataset collection
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}

# Dereplication

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [dRep dereplicate](toolshed.g2.bx.psu.edu/repos/iuc/drep_dereplicate/drep_dereplicate/3.5.0+galaxy1) %} with the following parameters:
>    - {% icon param-collection %} *"Genomes"*: `output` (Input dataset collection)
>    - *"Genome quality filtering"*: `Run checkM`
>    - In *"Genome comparison and clustering"*:
>        - *"Steps in genome comparison"*: `Default: Run MASH clustering and a secondary clustering`
>            - *"Algorithm for secondary clustering comparisons"*: `ANImf: Align whole genomes with nucmer; filter alignment; compare aligned regions - RECOMMENDED`
>    - *"Select outputs"*: ``
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}


# Conclusion

Sum up the tutorial and the key takeaways here. We encourage adding an overview image of the
pipeline used.

tools_json:
["Import", "upload1", "checkm2", "toolshed.g2.bx.psu.edu/repos/iuc/checkm2/checkm2/1.0.2+galaxy0", "dRep dereplicate", "toolshed.g2.bx.psu.edu/repos/iuc/drep_dereplicate/drep_dereplicate/3.5.0+galaxy1"]


topic: genome-annotation
tutorial: amr-gene-detection
title: Identification of AMR genes in an assembled bacterial genome
difficulty: introductory
time_estimation: 2h
requirements_json: []
tags_json: ["gmod", "illumina", "amr", "one-health", "jbrowse1", "microgalaxy"]
content_hash: f25b876d874f66951f79ea66b784961d
last_modified: 2026-05-27T09:01:48.731566
zenodo_link: https://zenodo.org/record/10572227
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/amr-gene-detection/tutorial.html

content:
Antimicrobial resistance (AMR) is a global phenomenon with no geographical or species boundaries, which poses an important threat to human, animal and environmental health. It is a complex and growing problem that compromises our ability to treat bacterial infections.

AMR gene content can be assessed from whole genome sequencing to detect known resistance mechanisms and potentially identify novel mechanisms.

To illustrate the process to identify AMR gene in a bacterial genome, we take an assembly of a bacterial genome (KUN1163 sample) generated by following a [bacterial genome assembly tutorial]({% link topics/assembly/tutorials/mrsa-illumina/tutorial.md %}) from data produced in "Complete Genome Sequences of Eight Methicillin-Resistant *Staphylococcus aureus* Strains Isolated from Patients in Japan" ({% cite Hikichi_2019 %}).

> Methicillin-resistant *Staphylococcus aureus* (MRSA) is a major pathogen
> causing nosocomial infections, and the clinical manifestations of MRSA
> range from asymptomatic colonization of the nasal mucosa to soft tissue
> infection to fulminant invasive disease. Here, we report the complete
> genome sequences of eight MRSA strains isolated from patients in Japan.
{: .quote cite="{% cite_url Hikichi_2019 %}"}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Galaxy and data preparation

Any analysis should get its own Galaxy history. So let's start by creating a new one and get the data (contig file from the assembly) into it.

> <hands-on-title>Prepare Galaxy and data</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 3. {% tool [Import](upload1) %} the contig file from [Zenodo]({{ page.zenodo_link }}) or from Galaxy shared data libraries:
>
>    ```
>    {{ page.zenodo_link }}/files/DRR187559_contigs.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
{: .hands_on}

# Identification of AMR genes

To identify AMR genes in contigs, tools like ABRicate or staramr ({% cite bharat2022correlation %}) can be used. In this tutorial, we use [**staramr**](https://github.com/phac-nml/staramr). **Staramr** scans bacterial genome contigs against the **ResFinder** ({% cite zankari2012identification %}), **PointFinder** ({% cite zankari2017pointfinder %}), and **PlasmidFinder** ({% cite carattoli2014plasmidfinder %}) databases (used by the ResFinder webservice) and compiles a summary report of detected antimicrobial resistance genes.

> <comment-title></comment-title>
>
> If you are interested in learning more about using ABRicate for AMR gene detection, you can check the sub-section on AMR in "Gene based pathogenic identification" of ["Pathogen detection from (direct Nanopore) sequencing data using Galaxy - Foodborne Edition" tutorial]({% link topics/microbiome/tutorials/pathogen-detection-from-nanopore-foodborne-data/tutorial.md %})
>
{: .comment}

> <hands-on-title>Run staramr</hands-on-title>
>
> 1. {% tool [staramr](toolshed.g2.bx.psu.edu/repos/nml/staramr/staramr_search/0.10.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"genomes"*: Contig file
>
{: .hands_on}

**staramr** generates 8 outputs:

1. `summary.tsv`:  A summary of all detected AMR genes/mutations/plasmids/sequence type in each genome, one genome per line and the following columns:

   * __Isolate ID__: The id of the isolate/genome file(s) passed to `staramr`.
   * __Quality Module__: The isolate/genome file(s) pass/fail result(s) for the quality metrics
   * __Genotype__: The AMR genotype of the isolate.
   * __Predicted Phenotype__: The predicted AMR phenotype (drug resistances) for the isolate.
   * __CGE Predicted Phenotype__: The CGE-predicted AMR phenotype (drug resistances) for the isolate (CGE = [Center for Genomic Epidemiology](http://www.genomicepidemiology.org/))

   * __Plasmid__: Plasmid types that were found for the isolate.
   * __Scheme__: The MLST scheme used

      MLST stands for MultiLocus Sequence Typing. It is a technique for the typing of multiple loci, using DNA sequences of internal fragments of multiple housekeeping genes to characterize isolates of microbial species.

      Here, **starmr** uses [mlst](https://github.com/tseemann/mlst) to scan the contig files against traditional [PubMLST](https://pubmlst.org/) typing schemes. The correspondance between the scheme and the bacteria genus and species is accessible in the [map](https://github.com/tseemann/mlst/blob/master/db/scheme_species_map.tab)

   * __Sequence Type__: The sequence type that's assigned when combining all allele types
   * __Genome Length__: The isolate/genome file(s) genome length(s)
   * __N50 value__: The isolate/genome file(s) N50 value(s)
   * __Number of Contigs Greater Than Or Equal To 300 bp__: The number of contigs greater or equal to 300 base pair in the isolate/genome file(s)
   * __Quality Module Feedback__: The isolate/genome file(s) detailed feedback for the quality metrics

    > <hands-on-title>Inspect the staramr output</hands-on-title>
    >
    > 1. View {% icon galaxy-eye %} the `summary.tsv` file
    >
    {: .hands_on}

    > <question-title>Inspect the staramr output</question-title>
    >
    > 1. How many genomes are there?
    > 2. Has the genome failed or passed the quality? Why?
    > 3. What are the predicted AMR drug resistances? Are they similar to the one found for KUN1163 in [Table 1](https://journals.asm.org/doi/10.1128/mra.01212-19#tab1) in {% cite Hikichi_2019 %}?
    > 4. What is the MLST scheme? Does that correspond to the expected species and genus?
    >
    > > <solution-title>Inspect the staramr output</solution-title>
    > >
    > > 1. There is one genome (1 line)
    > > 2. The genome has failed the quality (column 2), because the genome length is not within the acceptable length range (last column).
    > > 3. We can summarize starmr output and Table 1 in {% cite Hikichi_2019 %}:
    > >
    > >    Antibiotic name | Abbreviation | staramr | {% cite Hikichi_2019 %}
    > >    --- | --- | --- | ---
    > >    Amikacin | | Yes |
    > >    Azithromycin | | Yes |
    > >    Cefazolin | CEZ | | Yes
    > >    clindamycin | CLDM | | Yes
    > >    Erythromycin | EM | Yes | Yes
    > >    Gentamicin | GM | Yes | Yes
    > >    Imipenem | IPM | | Yes
    > >    Levofloxacin | LVFX | | Yes
    > >    Oxacillin | MPIPC | | Yes
    > >    Penicillin | | Yes |
    > >    Spectinomycin | | Yes |
    > >    Tetracycline | | Yes |
    > >    Tobramycin | | Yes |
    > >
    > > 4. The scheme is saureus, so *Staphylococcus aureus* (given the [scheme genus map](https://github.com/tseemann/mlst/blob/master/db/scheme_species_map.tab)), which is coherent with MRSA
    > {: .solution}
    >
    {: .question}

2. `detailed_summary.tsv`: A detailed summary of all detected AMR genes/mutations/plasmids/sequence type in each genome, one gene per line and the following columns:

    * __Isolate ID__: The id of the isolate/genome file(s) passed to `staramr`.
    * __Data__: The particular gene detected from ResFinder, PlasmidFinder, PointFinder, or the sequence type.
    * __Data Type__: The type of gene (Resistance or Plasmid), or MLST.
    * __Predicted Phenotype__: The predicted AMR phenotype (drug resistances) found in ResFinder/PointFinder. Plasmids will be left blank by default.
    * __CGE Predicted Phenotype__: The CGE-predicted AMR phenotype (drug resistances) found in ResFinder/PointFinder. Plasmids will be left blank by default.
    * __%Identity__: The % identity of the top BLAST HSP to the gene.
    * __%Overlap__: THe % overlap of the top BLAST HSP to the gene (calculated as __hsp length/total length * 100__).
    * __HSP Length/Total Length__ The top BLAST HSP length over the gene total length (nucleotides).
    * __Contig__: The contig id containing this gene.
    * __Start__: The start of the gene (will be greater than __End__ if on minus strand).
    * __End__: The end of the gene.
    * __Accession__: The accession of the gene from either ResFinder or PlasmidFinder  database.

    > <hands-on-title>Inspect the staramr output</hands-on-title>
    >
    > 1. View {% icon galaxy-eye %} the `detailed_summary.tsv` file
    >
    {: .hands_on}

    > <question-title>Inspect the staramr output</question-title>
    >
    > 1. What are the different sequence types that have been identified?
    > 2. Where are located the plasmid genes?
    > 3. Where are located the resistances genes?
    > 4. By looking for the accession number (M13771) on NCBI ([https://www.ncbi.nlm.nih.gov/nuccore/](https://www.ncbi.nlm.nih.gov/nuccore/)), where does the first resistance come from?
    >
    > > <solution-title>Inspect the staramr output</solution-title>
    > >
    > > 1. There are:
    > >    - 1 MSLT
    > >    - 5 plasmid *rep* genes
    > >    - 7 resistance genes
    > > 2. The plasmid genes are on contig00019 (3 genes - coherent with {% cite Lozano_2012 %}), contig00024, and contig00002.
    > > 3. The resistance genes are:
    > >    - 4/7 on contigs with plasmid genes (contig00002, contig00019, contig00024)
    > >    - 3/7 on contigs without plasmid genes (contig00022, contig00021)
    > > 4. M13771 is a 6'-aminoglycoside acetyltransferase phosphotransferase (AAC(6')-APH(2')) bifunctional resistance protein from *Streptococcus faecalis*. It might come from horizontal transfer from *Streptococcus faecalis* on a plasmid of *Staphylococcus aureus*.
    > {: .solution}
    >
    {: .question}

3. `resfinder.tsv`: A tabular file of each AMR gene and additional BLAST information from the **ResFinder** database, one gene per line and the following columns:

   * __Isolate ID__: The id of the isolate/genome file(s) passed to `staramr`.
   * __Gene__: The particular AMR gene detected.
   * __Predicted Phenotype__: The predicted AMR phenotype (drug resistances) for this gene.
   * __CGE Predicted Phenotype__: The CGE-predicted AMR phenotype (drug resistances) for this gene.
   * __%Identity__: The % identity of the top BLAST HSP to the AMR gene.
   * __%Overlap__: THe % overlap of the top BLAST HSP to the AMR gene (calculated as __hsp length/total length * 100__).
   * __HSP Length/Total Length__ The top BLAST HSP length over the AMR gene total length (nucleotides).
   * __Contig__: The contig id containing this AMR gene.
   * __Start__: The start of the AMR gene (will be greater than __End__ if on minus strand).
   * __End__: The end of the AMR gene.
   * __Accession__: The accession of the AMR gene in the ResFinder database.
   * __Sequence__: The AMR Gene sequence
   * __CGE Notes__: Any CGE notes associated with the prediction.

4. `plasmidfinder.tsv`: A tabular file of each AMR plasmid type and additional BLAST information from the **PlasmidFinder** database, one plasmid type per line with the following columns:

   * __Isolate ID__: The id of the isolate/genome file(s) passed to `staramr`.
   * __Plasmid__: The particular plasmid type detected.
   * __%Identity__: The % identity of the top BLAST HSP to the plasmid type.
   * __%Overlap__: The % overlap of the top BLAST HSP to the plasmid type (calculated as    __hsp length/total length * 100__).
   * __HSP Length/Total Length__ The top BLAST HSP length over the plasmid type total length (nucleotides).
   * __Contig__: The contig id containing this plasmid type.
   * __Start__: The start of the plasmid type (will be greater than __End__ if on minus strand).
   * __End__: The end of the plasmid type.
   * __Accession__: The accession of the plasmid type in the PlasmidFinder database.

5. `mlst.tsv`: A tabular file of each multi-locus sequence type (MLST) and it's corresponding locus/alleles, one genome per line with the following columns:

   * __Isolate ID__: The id of the isolate/genome file(s) passed to `staramr`.
   * __Scheme__: The scheme that `MLST` has identified.
   * __Sequence Type__: The sequence type that's assigned when combining all allele types
   * __Locus #__: A particular locus in the specified MLST scheme.

6. `settings.txt`: The command-line, database versions, and other settings used to run `staramr`.
7. `results.xlsx`: An Excel spreadsheet containing the previous files as separate worksheets.
8. `hits`: A collection with fasta files of the BLAST HSP nucleotides for the entries listed in the **resfinder.tsv** and **pointfinder.tsv** files. There are up to two files per input genome, one for ResFinder and one for PointFinder.

# Getting more information about ARG via the CARD database

To get more information about the antibiotic resistant genes (ARG), we can check the [CARD database](https://card.mcmaster.ca) (**C**omprehensive **A**ntibiotic **R**esistance **D**atabase) ({% cite Jia2016 %})

CARD can be very helpful to check all the resistance genes and check if it is logical to find the resistance gene in a specific bacteria.

> <question-title></question-title>
>
> 1. To what family does [mecA](https://card.mcmaster.ca/ontology/36911) belong?
> 2. Do you expect to find this gene in this MRSA strain and why?
> 3. Is the accession number of the entry related to the accession reported by staramr?
>
> > <solution-title></solution-title>
> >
> > 1. [Methicillin resistant PBP2](https://card.mcmaster.ca/ontology/37589)
> > 2. The strain we use is a Methicillin(multi) resistant Staphylococcus aureus. As `mecA` has a perfect resistome mach with *S. aureus*, and the AMR Gene Family is methicillin resistant PBP2, we expect to see mecA in MRSA.
> > 3. No, these are completely unrelated. Unfortunately this is a **very** common issue in bioinformatics. Everyone builds their own numbering system for entries in their database (usually calling them 'accessions'), and then someone else needs to build a service to link these databases.
> >
> {: .solution}
{: .question}

# Visualization of the ARGs and plasmid genes in their genomic context

We would like to look at the ARGs and plasmid genes in their genomic context. To do that, we will usie [JBrowse](https://jbrowse.org/jb2/) ({% cite diesh2023jbrowse %}) with several information:

1. Assembly as the reference
2. ARGs location
3. Contigs annotation (genes, etc)
4. Coverage of the contigs from the raw reads

## Extraction of the ARG and plasmid genes location

The first step is to extract the location of the ARGs and plasmid genes on the contigs. This information is available on the `detailed_summary.tsv` output of **starmar**. The genes and their location are on the lines with a decimal value on column 6 or 7. So to only get this information, we need to select lines with a decimal value (`###.##`) followed by a tab character, the column separator in Galaxy. As a result, any lines without an identity or overlap value will be filtered out.

> <hands-on-title>Extract ARGs and plasmid genes</hands-on-title>
> 1. {% tool [Select lines that match an expression](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `detailed_summary.tsv` **staramr** output
>    - *"that"*: Matching
>    - *"the pattern"*: `[0-9]+\.[0-9]+\t`
{: .hands_on}

> <question-title></question-title>
>
> How many genes have been kept?
>
> > <solution-title></solution-title>
> >
> > 12
> >
> {: .solution}
{: .question}

This table can not be used directly in JBrowse. It first needs to be transformed in a standard format: GFF3, a file format used for describing genes and other features of DNA, RNA and protein sequences.


> <comment-title>GFF3 file format</comment-title>
>
> A GFF is a tab delimited file with 9 fields per line:
> 1. **seqid**: The name of the sequence where the feature is located.
> 2. **source**: The algorithm or procedure that generated the feature. This is typically the name of a software or database.
> 3. **type**: The feature type name, like "gene" or "exon". In a well structured GFF file, all the children features always follow their parents in a single block (so all exons of a transcript are put after their parent "transcript" feature line and before any other parent transcript line). In GFF3, all features and their relationships should be compatible with the standards released by the Sequence Ontology Project.
> 4. **start**: Genomic start of the feature, with a 1-base offset. This is in contrast with other 0-offset half-open sequence formats, like BED.
> 5. **end**: Genomic end of the feature, with a 1-base offset. This is the same end coordinate as it is in 0-offset half-open sequence formats, like BED.
> 6. **score**: Numeric value that generally indicates the confidence of the source in the annotated feature. A value of "." (a dot) is used to define a null value.
> 7. **strand**: Single character that indicates the strand of the feature. This can be "+" (positive, or 5'->3'), "-", (negative, or 3'->5'), "." (undetermined), or "?" for features with relevant but unknown strands.
> 8. **phase**: phase of CDS features; it can be either one of 0, 1, 2 (for CDS features) or "." (for everything else). See the section below for a detailed explanation.
> 9. **attributes**: A list of tag-value pairs separated by a semicolon with additional information about the feature.
{: .comment}

> <hands-on-title>Create a GFF file</hands-on-title>
> 3. {% tool [Table to GFF3](toolshed.g2.bx.psu.edu/repos/iuc/tbl2gff3/tbl2gff3/1.2) %}
>    - {% icon param-file %} *"Table"*: output of the above **Select lines** {% icon tool %} step
>    - *"Record ID column or value"*: `9`
>    - *"Start column or value"*: `10`
>    - *"End column or value"*: `11`
>    - *"Type column or value"*: `3`
>    - *"Score column or value"*: `6`
>    - *"Source column or value"*: `3`
>    - {% icon param-repeat %} *"Insert Qualifiers"*
>        - *"Name"*: `name`
>        - *"Qualifier value column or raw text"*: `2`
>    - {% icon param-repeat %} *"Insert Qualifiers"*
>        - *"Name"*: `phenotype`
>        - *"Qualifier value column or raw text"*: `4`
>    - {% icon param-repeat %} *"Insert Qualifiers"*
>        - *"Name"*: `accession`
>        - *"Qualifier value column or raw text"*: `12`
>
{: .hands_on}

> <question-title></question-title>
>
> How many genes have been found per contigs?
>
> > <solution-title></solution-title>
> >
> > - Contig 19: 3 plasmid genes, 1 ARG
> > - Contig 24: 1 plasmid gene, 2 ARGs
> > - Contig 2: 1 plasmid gene, 1 ARG
> > - Contig 22: 2 ARGs
> > - Contig 21: 1 ARG
> >
> {: .solution}
{: .question}

## Annotation of the contigs

In addition to the ARGs and plasmid genes, it would be good to have extra information about other genes on the contigs. Several tools exists to do that: Prokka ({% cite seemann2014prokka %}), Bakta ({% cite schwengers2021bakta %}), etc. Here, we use **Bakta** as recommended by Torsten Seeman as the successor of Prokka.

{% include _includes/cyoa-choices.html option1="No" option2="Yes" default="No" text="Have you already run Bakta on this dataset before?"%}

<div class="No" markdown="1">

**Bakta**  is a tool for the rapid & standardized annotation of bacterial genomes and plasmids from both isolates and MAGs.

> <hands-on-title>Annotate the contigs</hands-on-title>
>
> 1. {% tool [Bakta](toolshed.g2.bx.psu.edu/repos/iuc/bakta/bakta/1.8.2+galaxy0) %} with the following parameters:
>    - In *"Input/Output options"*:
>        - *"The bakta database"*: latest one
>        - *"The amrfinderplus database"*: latest one
>        - {% icon param-file %} *"Select genome in fasta format"*: Contig file
>    - In *"Optional annotation"*:
>        - *"Keep original contig header (--keep-contig-headers)"*: `Yes`
{: .hands_on}

> <question-title></question-title>
>
> How many features have been identified?
>
> > <solution-title></solution-title>
> >
> > 51k+ (number of lines in the GFF)
> >
> {: .solution}
{: .question}

</div>

<div class="Yes" markdown="1">

If you already ran **Bakta** on your contigs, you can just upload here the GFF generated by **Bakta**.

> <hands-on-title>Import annotation GFF</hands-on-title>
>
> 1. {% tool [Import](upload1) %} the GFF file generated by **Bakta**
>
{: .hands_on}

![Bakta - circular plot of the annotation, with the contigs on the outside ordered clockwise by length size.](./images/bakta.svg)

</div>

## Mapping the raw reads on the contigs

To get information about the coverage of the contigs and genes, we map the reads used to build the contigs to the contigs using **Bowtie2** ({% cite langmead2009ultrafast %}).

> <hands-on-title>Annotating the Genome</hands-on-title>
> 1. {% tool [Import](upload1) %} the reads (after quality control) from another history, from [Zenodo]({{ page.zenodo_link }}) or from Galaxy shared data libraries:
>
>    ```
>    {{ page.zenodo_link }}/files/DRR187559_after_fastp_1.fastq.gz
>    {{ page.zenodo_link }}/files/DRR187559_after_fastp_2.fastq.gz
>    ```
>    
> 2. Create a paired collection named `Paired Reads`
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
> 3. {% tool [Bowtie2](toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.4+galaxy0) %} with the following parameters:
>    - *"Is this single or paired library"*: `Paired-end`
>        - {% icon param-collection %} *"FASTQ Paired Dataset"*: `Paired Reads`
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from the history and build index`
>        - {% icon param-file %} *"Select reference genome"*: Contig file
>    - *"Save the bowtie2 mapping statistics to the history"*: `Yes`
>
{: .hands_on}

> <question-title></question-title>
>
> What is the alignment rate? What does that mean for the assembly?
>
> > <solution-title></solution-title>
> >
> > The alignment rate is 99.77%. Most of the reads have been mapped on the contigs. They have been then used to build the contigs.
> >
> {: .solution}
{: .question}

## Visualisation of the ARGs

We can now visualize the contigs, the mapping coverage, and the genes, using JBrowse and different information track.

> <hands-on-title>Visualize the Genome</hands-on-title>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: Contig file
>    - *"Genetic Code"*: `11. The Bacterial, Archaeal and Plant Plastid Code`
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Bakta`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `Annotation_and_sequences` (output of **Bakta** {% icon tool %})
>                        - *"JBrowse Track Type [Advanced]"*: `Neat Canvas Features`
>                        - *"Track Visibility"*: `On for new users`
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `ARGs and plasmid genes`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: Output of **Table to GFF3**
>                        - *"JBrowse Track Type [Advanced]"*: `Neat Canvas Features`
>                        - *"Track Visibility"*: `On for new users`
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Coverage`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BAM Pileups`
>                        - {% icon param-file %} *"BAM Track Data"*: Bowtie2's output
>                        - *"Autogenerate SNP Track"*: `Yes`
>
> 3. View the output of JBrowse
>
{: .hands_on}

In the output of the JBrowse you can view the mapped reads and the found genes against the reference genome. With the search tools you can easily find genes of interest. JBrowse can handle many inputs and can be very useful. Using the Bowtie2 mapping output, low coverage regions can be detected. This SNP detection can also give a clear view of where the data was less reliable or where variations were located.

If it takes too long to build the JBrowse instance, you can view an embedded one here. (**Warning**: feature name search will not work.)

{% snippet topics/visualisation/faqs/visualizations_jbrowse.html datadir="data" loc="contig00019:81..23760" tracks="DNA,878c908bab1edf92fb1577fad80f7bae_0,a7fe8304cddfe5fbececdc7a2bd7ed18_0," %}

> <question-title></question-title>
>
> 1. What is the name for the gene found by Bakta corresponding to the rep16 gene found by staramr?
> 2. What are the NCBI Protein id and UniRef id for aac(6')-aph(2'')?
>
> > <solution-title></solution-title>
> >
> > 1. By zooming at the beginning of the contig, the name for rep16 is DUF536 domain-containing protein
> > 2. aac(6')-aph(2'') is on contig00019:20311..21749. The corresponding gene found by Bakta is bifunctional aminoglycoside N-acetyltransferase AAC(6')-Ie/aminoglycoside O-phosphotransferase APH(2'')-Ia. When clicking on it, we can find the dbxref attributes:
> >   - NCBI Protein: WP_001028144.1
> >   - UniRef: UniRef50_P0A0C2
> {: .solution}
{: .question}

# Conclusion

In this tutorial, contigs were scaned for AMR and plasmid genes. The genes were then visualized in their genomic context after contig annotation.

tools_json:
["Import", "upload1", "staramr", "toolshed.g2.bx.psu.edu/repos/nml/staramr/staramr_search/0.10.0+galaxy1", "Select lines that match an expression", "Grep1", "Table to GFF3", "toolshed.g2.bx.psu.edu/repos/iuc/tbl2gff3/tbl2gff3/1.2", "Bakta", "toolshed.g2.bx.psu.edu/repos/iuc/bakta/bakta/1.8.2+galaxy0", "Bowtie2", "toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.4+galaxy0", "JBrowse", "toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1"]


