# tutorials

topic: transcriptomics
tutorial: rna-seq-counts-to-genes
title: 2: RNA-seq counts to genes
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: ["limma-voom", "mouse", "QC"]
content_hash: b1ebd5eacb05d1434dfd55031fa49949
last_modified: 2026-02-26T21:12:18.571036
zenodo_link: https://zenodo.org/record/4273218
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/rna-seq-counts-to-genes/tutorial.html

content:
Measuring gene expression on a genome-wide scale has become common practice over the last two decades or so, with microarrays predominantly used pre-2008. With the advent of next generation sequencing technology in 2008, an increasing number of scientists use this technology to measure and understand changes in gene expression in often complex systems. As sequencing costs have decreased, using RNA-Seq to simultaneously measure the expression of tens of thousands of genes for multiple samples has never been easier. The cost of these experiments has now moved from generating the data to storing and analysing it.

There are many steps involved in analysing an RNA-Seq experiment. The analysis begins with sequencing reads (FASTQ files). These are usually aligned to a reference genome, if available. Then the number of reads mapped to each gene can be counted. This results in a table of counts, which is what we perform statistical analyses on to determine differentially expressed genes and pathways. The purpose of this tutorial is to demonstrate how to perform differential expression on count data with **limma-voom**. How to generate counts from reads (FASTQs) is covered in the accompanying tutorial [RNA-seq reads to counts]({% link topics/transcriptomics/tutorials/rna-seq-reads-to-counts/tutorial.md %}).

## Mouse mammary gland dataset

The data for this tutorial comes from a Nature Cell Biology paper by {% cite Fu2015 %}. Both the raw data (sequence reads) and processed data (counts) can be downloaded from Gene Expression Omnibus database (GEO) under accession number [GSE60450](http://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE60450).

This study examined the expression profiles of basal and luminal cells in the mammary gland of virgin, pregnant and lactating mice. Six groups are present, with one for each combination of cell type and mouse status. Note that two biological replicates are used here, two independent sorts of cells from the mammary glands of virgin, pregnant or lactating mice, however three replicates is usually recommended as a minimum requirement for RNA-seq. In this tutorial we will use the GEO counts file as a starting point for our analysis. Alternatively, you could create a count matrix from the raw sequence reads, as demonstrated in the [RNA-seq reads to counts tutorial]({% link topics/transcriptomics/tutorials/rna-seq-reads-to-counts/tutorial.md %}). The GEO count file was generated from aligning the reads to the mouse `mm10` genome with the Rsubread aligner ({% cite Liao2019 %}), followed by counting reads mapped to RefSeq genes with featureCounts ({% cite liao2013featurecounts %}), see the [Fu paper](https://www.nature.com/articles/ncb3117) for details.

We will use **limma-voom** ({% cite Law2014 %}) for identifying differentially expressed genes here. Other popular alternatives are edgeR ({% cite robinson2010edger %}) and DESeq2 ({% cite love2014moderated %}). Limma-voom has been shown to be perform well in terms of precision, accuracy and sensitivity ({% cite CostaSilva2017 %}) and, due to its speed, it's particularly recommended for large-scale datasets with 100s of samples ({% cite Chen2016 %}).

This is a Galaxy tutorial based on material from the [COMBINE R RNAseq workshop](http://combine-australia.github.io/RNAseq-R/06-rnaseq-day1.html), first taught [in this course](http://combine-australia.github.io/2016-05-11-RNAseq/).

![Tutorial Dataset](../../images/rna-seq-reads-to-counts/mouse_exp.png "Tutorial Dataset")


> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}


{% snippet faqs/galaxy/analysis_results_may_vary.md %}

# Preparing the inputs

We will use three files for this analysis:

 * **Count matrix** (genes in rows, samples in columns)
 * **Sample information** file (sample id, group)
 * **Gene annotation** file (gene id, symbol, description)

## Import data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this RNA-seq exercise e.g. `RNA-seq with limma-voom`
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import the mammary gland counts table and the associated sample information file.
>
>    To import the files, there are two options:
>    - Option 1: From a shared data library if available (`GTN - Material -> {{ page.topic_name }} -> {{ page.title }}`)
>    - Option 2: From [Zenodo](https://zenodo.org/record/4273218)
>
>      {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    - You can paste both links below into the **Paste/Fetch** box:
>
>    ```
>    https://zenodo.org/record/4273218/files/countdata.tsv
>    https://zenodo.org/record/4273218/files/factordata.tsv
>    ```
>
> 2. Rename the counts dataset as `countdata` and the sample information dataset as `factordata` using the {% icon galaxy-pencil %} (pencil) icon.
> 3. Check that the datatype is `tabular`.
>    If the datatype is not `tabular`, please change the file type to `tabular`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}


Let’s take a look at the data. The `countdata` file contains information about genes (one gene per row), the first column has the Entrez gene id and the remaining columns contain information about the number of reads aligning to the gene in each experimental sample. There are two replicates for each cell type and time point (detailed sample info can be found in file “GSE60450_series_matrix.txt” from the GEO website). The first few rows and columns of the seqdata file are shown below.

![countdata file](../../images/rna-seq-counts-to-genes/countdata.png "Count file")

The `factordata` file contains basic information about the samples that we will need for the analysis. See below. Note that the sample ids must match exactly with the ids in the counts file.

![factordata file](../../images/rna-seq-counts-to-genes/factordata.png "Sample information file")


> <details-title>Formatting the data</details-title>
>
> The files above have been formatted for you. If you are interested to know how they were formatted the information is below.
>
> The original files are available at
>
> ```
> https://zenodo.org/record/4273218/files/GSE60450_Lactation-GenewiseCounts.txt
> https://zenodo.org/record/4273218/files/SampleInfo.txt
> ```
>
>![seqdata file](../../images/rna-seq-counts-to-genes/seqdata.png "Count file (before formatting)")
>
>![sampleinfo file](../../images/rna-seq-counts-to-genes/sampleinfo.png "Sample information file (before formatting)")
>
>To create the file, `countdata`, that contains only the counts for the 12 samples i.e. we'll remove the gene length column with the **Cut columns from a table (cut)** tool. The sample names are also pretty long so we'll use the **>Replace Text in entire line** tool to shorten these to contain only the relevant information about each sample. We will also replace the hyphen in the sample names with a dot so they match the names in the sample information file.
>
>> <hands-on-title>Format the counts data</hands-on-title>
>>
>> 1. **Cut columns from a table (cut)** {% icon tool %} with the following parameters:
>>      - {% icon param-file %} *"File to cut"*: `seqdata`
>>      - {% icon param-select %} *"Operation"*: `Discard`
>>      - {% icon param-select %} *"List of fields"*: Select `Column:2`
>> 2. **Replace Text in entire line** {% icon tool %} with the following parameters:
>>      - {% icon param-file %} *"File to process"*: output of **Cut** {% icon tool %}
>>      - {% icon param-text %} *"Find pattern"*: `_B[A-Z0-9_]+`
>> 3. **Replace Text in entire line** {% icon tool %} with the following parameters:
>>      - {% icon param-file %} *"File to process"*: output of **Replace Text** {% icon tool %}
>>      - {% icon param-text %} *"Find pattern"*: `-`
>>      - {% icon param-text %} *"Replace with"*: `.`
>> 4. Rename file as `countdata` using the {% icon galaxy-pencil %} (pencil) icon.
>{: .hands_on}
>
>To create the file, `factordata`, that contains the groups information that we need for the limma-voom tool. We'll combine the cell type and mouse status to make 6 groups e.g. we'll combine the CellType `basal` with the Status `pregnant` for the group `basalpregnant`. We'll use the **Merge Columns** tool to combine the cell type and mouse status columns in the sample information file, making a column with the 6 group names.
>
>> <hands-on-title>Format the sample information file</hands-on-title>
>>
>> 1. **Merge Columns together** {% icon tool %} with the following parameters:
>>      - {% icon param-file %} *"Select data"*: `sampleinfo`
>>      - {% icon param-select %} *"Merge column"*: `Column: 3`
>>      - {% icon param-select %} *"with column"*: `Column: 4`
>> 2. **Cut columns from a table (cut)** {% icon tool %} with the following parameters:
>>      - {% icon param-file %} *"File to cut"*: output of **Merge Columns** {% icon tool %}
>>      - {% icon param-select %} *"Operation"*: `Keep`
>>      - {% icon param-select %} *"List of fields"*: Select `Column:2` and `Column:5`
>> 3. Rename file as `factordata` using the {% icon galaxy-pencil %} (pencil) icon.
>{: .hands_on}
{: .details}

## Get gene annotations

Optionally, gene annotations can be provided to the limma-voom tool and if provided the annotation will be available in the output files. We'll get gene symbols and descriptions for these genes using the Galaxy **annotateMyIDs** tool, which provides annotations for human, mouse, fruitfly and zebrafish.

> <hands-on-title>Get gene annotations</hands-on-title>
>
> 1. **annotateMyIDs** {% icon tool %} with the following parameters:
>      - {% icon param-file %} *"File with IDs"*: `countdata`
>      - {% icon param-check %} *"File has header"*: `Yes`
>      - {% icon param-select %} *"Organism"*: `Mouse`
>      - {% icon param-select %} *"ID Type"*: `Entrez`
>      - {% icon param-check %} "*Output columns"*: tick
>          - `ENTREZID`
>          - `SYMBOL`
>          - `GENENAME`
> 2. Rename file as `annodata` using the {% icon galaxy-pencil %} (pencil) icon. The file should look like below.
>    ![annodata file](../../images/rna-seq-counts-to-genes/annodata.png "Gene annotation file")
> 3. Check the number of lines shown on the datasets in the history, there should be 27,180 lines in both. There must be the same number of lines (rows) in the counts and annotation.
{: .hands_on}

# Differential expression with limma-voom

## Filtering to remove lowly expressed genes

It is recommended to filter for lowly expressed genes when running the limma-voom tool. Genes with very low counts across all samples provide little evidence for differential expression and they interfere with some of the statistical approximations that are used later in the pipeline. They also add to the multiple testing burden when estimating false discovery rates, reducing power to detect differentially expressed genes. These genes should be filtered out prior to further analysis.

There are a few ways to filter out lowly expressed genes. When there are biological replicates in each group, in this case we have a sample size of 2 in each group, we favour filtering on a minimum counts-per-million (CPM) threshold present in at least 2 samples. Two represents the smallest sample size for each group in our experiment. In this dataset, we choose to retain genes if they are expressed at a CPM above 0.5 in at least two samples. The CPM threshold selected can be compared to the raw count with the CpmPlots (see below).

> <details-title>More details on filtering</details-title>
>
> The limma tool uses the `cpm` function from the edgeR package ({% cite robinson2010edger %}) to generate the CPM values which can then be filtered. Note that by converting to CPMs we are normalizing for the different sequencing depths for each sample. A CPM of 0.5 is used as it corresponds to a count of 10-15 for the library sizes in this data set. If the count is any smaller, it is considered to be very low, indicating that the associated gene is not expressed in that sample. A requirement for expression in two or more libraries is used as each group contains two replicates. This ensures that a gene will be retained if it is only expressed in one group. Smaller CPM thresholds are usually appropriate for larger libraries. As a general rule, a good threshold can be chosen by identifying the CPM that corresponds to a count of 10, which in this case is about 0.5. You should filter with CPMs rather than filtering on the counts directly, as the latter does not account for differences in library sizes between samples.
{: .details}

## Normalization for composition bias

In an RNA-seq analysis, the counts are normalized for different sequencing depths between samples. Normalizing to eliminate composition biases between samples is also typically performed. Composition biases can occur, for example, if there are a few highly expressed genes dominating in some samples, leading to less reads from other genes. By default, TMM normalization ({% cite robinson2010scaling %}) is performed by the limma tool using the edgeR `calcNormFactors` function (this can be changed under **Advanced Options**). TMM stands for Trimmed Mean of M values, where a weighted trimmed mean of the log expression ratios is used to scale the counts for the samples. See the figure from the TMM paper below. Note the plot (Figure 1c) that shows how a few highly expressed genes in the liver sample (where the arrow is) results in the majority of other genes in the sample having the appearance of being expressed lower in liver. The mid-line through the points is offset from the expected zero and the TMM normalization factor (red line) scales the counts to adjust for this.

![TMM normalization](../../images/rna-seq-counts-to-genes/TMM.png "TMM normalization (Robinson and Oshlack 2010)")

## Specify Contrast(s) of interest

Since we are interested in differences between groups, we need to specify which comparisons we want to test. For example, if we are interested in knowing which genes are differentially expressed between the pregnant and lactating group in the basal cells we specify `basalpregnant-basallactate` for the *Contrast of Interest*. Note that the group names in the contrast must exactly match the names of the groups in the `factordata` file. More than one contrast can be specified using the `Insert Contrast` button, so we could look at more comparisons of the groups here, but first we'll take a look at `basalpregnant-basallactate`.

> <hands-on-title>Differential expression with limma-voom</hands-on-title>
>
> 1. **limma** {% icon tool %} with the following parameters:
>      - {% icon param-file %} *"Differential Expression Method"*: `limma-voom`
>      - {% icon param-select %} *"Count Files or Matrix?*": `Single Count Matrix`
>          - {% icon param-file %} *"Count Matrix"*: Select `countdata`
>      - {% icon param-select %} *"Input factor information from file?"*: `Yes`
>          - {% icon param-file %} *"Factor File"*: Select `factordata`
>      - {% icon param-select %} *"Use Gene Annotations?"*: `Yes`
>          - {% icon param-file %} *"Factor File"*: Select `annodata`
>      - {% icon param-text %} *"Contrast of Interest"*: `basalpregnant-basallactate`
>      - {% icon param-select %} *"Filter lowly expressed genes?"*: `Yes`
>          - {% icon param-select %} *"Filter on CPM or Count values?"*: `CPM`
>          - {% icon param-text %} *"Minimum CPM"*: `0.5`
>          - {% icon param-text %} *"Minimum Samples"*: `2`
> 2. Inspect the `Report` produced by clicking on the {% icon galaxy-eye %} (eye) icon
{: .hands_on}


> <tip-title>Multiple factors</tip-title>
>
> If we need to account for additional sources of variation, for example, batch, sex, genotype etc, we can input that information as additional factors. For example, if we were interested in the genes differentially expressed between the luminal and basal cell types, we could include an additional column to account for the variation due to the different stages.
> ![Multiple factors](../../images/rna-seq-counts-to-genes/multiplefactors_factordata.png "Multiple factors")
>
{: .tip}


# QC of count data

Before we check out the differentially expressed genes, we can look at the `Report` information to check that the data is good quality and that the samples are as we would expect.

## Multidimensional scaling plot

By far, one of the most important plots we make when we analyse RNA-Seq data are MDS plots. An MDS plot is a visualisation of a principal components analysis, which determines the greatest sources of variation in the data. A principal components analysis is an example of an unsupervised analysis, where we don’t need to specify the groups. If your experiment is well controlled and has worked well, what we hope to see is that the greatest sources of variation in the data are the treatments/groups we are interested in. It is also an incredibly useful tool for quality control and checking for outliers. This Galaxy limma tool outputs an MDS plot by default in the `Report` and a link is also provided to a PDF version (`MDSPlot_CellTypeStatus.pdf`). A scree plot is also produced that shows how much variation is attributed to each dimension. If there was a batch effect for example, you may see high values for additional dimensions, and you may choose to include batch as an additional factor in the differential expression analysis. The limma tool plots the first two dimensions by default (1 vs 2), however you can also plot additional dimensions 2 vs 3 and 3 vs 4 using under **Output Options** *Additional Plots* `MDS Extra`. These are displayed in the `Report` along with a link to a PDF version (MDSPlot_extra.pdf). Selecting the `Glimma Interactive Plots` will generate an interactive version of the MDS plot, see the plots section of the report below. If outlier samples are detected you may decide to remove them. Alternatively, you could downweight them by choosing the option in the limma tool *Apply voom with sample quality weights?*. The voom sample quality weighting is described in the paper *Why weight? Modelling sample and observational level variability improves power in RNA-seq analyses ({% cite Liu2015 %})*.

![MDS Plot](../../images/rna-seq-counts-to-genes/mdsscree.png "MDS Plot")

Take a look at the MDS plot coloured by group.

> <question-title></question-title>
>
> Do you notice anything about the samples in this plot?
>
>    > <solution-title></solution-title>
>    >
>    > Two samples don't appear to be in the right place.
>    >
>    {: .solution}
{: .question}

It turns out that there has been a mix-up with two samples, they have been mislabelled in the sample information file. This shows how the MDS plot can also be useful to help identify if sample mix-ups may have occurred. We need to redo the limma-voom analysis with the correct sample information.

> <hands-on-title>Use the Rerun button to redo steps</hands-on-title>
>
> 1. Import the correct sample information file from `https://zenodo.org/record/4273218/files/factordata_fixed.tsv` and name as `factordata_fixed`
> 2. Use the **Rerun** {% icon galaxy-refresh %} button in the History to run **limma** {% icon tool %} as before with the `factordata_fixed` file and adding the following parameters:
>      - **Output Options**
>          - {% icon param-check %} *"Additional Plots"* tick:
>              - `Glimma Interactive Plots`
>              - `Density Plots (if filtering)`
>              - `CpmsVsCounts Plots (if filtering on cpms)`
>              - `Box Plots (if normalising)`
>              - `MDS Extra (Dims 2vs3 and 3vs4)`
>              - `MD Plots for individual samples`
>              - `Heatmaps (top DE genes)`
>              - `Stripcharts (top DE genes)`
> 3. Delete the incorrect `factordata` file and its limma outputs to avoid any confusion.
{: .hands_on}

In the `Report` you should then see the correct MDS plot as below.

![MDS Plot](../../images/rna-seq-counts-to-genes/mdsscree_corr.png "MDS Plot (correct samples)")


> <question-title></question-title>
>
> What is the greatest source of variation in the data (i.e. what does dimension 1 represent)?
> What is the second greatest source of variation in the data?
>
>    > <solution-title></solution-title>
>    >
>    > Dimension 1 represents the variation due to cell type, basal vs luminal. Dimension 2 represents the variation due to the stages, virgin, pregnant or lactating.
>    >
>    {: .solution}
{: .question}

Next, scroll down the `Report` to take a look at the **Additional information** and **Summary of experimental data** sections near the bottom. It should look similar to below. Here you can check that the correct samples have been assigned to the correct groups, what settings were used (e.g. filters, normalization method) and also how many genes were filtered out due to low expression.

![Report summary](../../images/rna-seq-counts-to-genes/report_summary.png){: width="600px"}

> <question-title></question-title>
>
> How many genes have been filtered out for low expression?
>
>    > <solution-title></solution-title>
>    >
>    > 11375 genes were filtered out as insignificant as they were without more than 0.5 CPM in at least 2 samples.
>    >
>    {: .solution}
{: .question}


## Density plots

Density plots can be output in the `Report` if *Filter lowly expressed genes* is selected. A link is also provided in the `Report` to a PDF version (`DensityPlots.pdf`). These plots allow comparison of the counts distributions before and after filtering. The samples are coloured by the groups. Count data is not normally distributed, so if we want to examine the distributions of the raw counts we need to log the counts. We typically check the distribution of the read counts on the log2 scale. A CPM value of 1 is equivalent to a log-CPM value of 0 and the CPM we used of 0.5 is equivalent to a log-CPM of -1. It can be seen in the Raw counts (before filtering) plot below, that a large proportion of genes within each sample are not expressed or lowly-expressed, and the Filtered counts plot shows our filter of CPM of 0.5 (in at least 2 samples) removes a lot of these uninformative genes.

![Density Plots](../../images/rna-seq-counts-to-genes/densityplots.png "Density Plots")

The `Report` provides links to PDFs of all plots shown in the `Report` and also to the rest of the additional plots selected to be output.

![Report Outputs](../../images/rna-seq-counts-to-genes/report_plots.png "Report outputs")

> <details-title>Cpm plots</details-title>
>
>We can also have a look more closely to see whether our threshold of 0.5 CPM does indeed correspond to a count of about 10-15 reads in each sample with the plots of CPM versus raw counts.
>
>Click on the `CpmPlots.pdf` link in the `Report`. You should see 12 plots, one for each sample. Two of the plots are shown below. From these plots we can see that 0.5 CPM is equivalent to ~10 counts in each of the 12 samples, so 0.5 seems to be an appropriate threshold for this dataset (these samples all have sequencing depth of 20-30 million, see the `Library information` file below, so a CPM value of 0.5 would be ~10 counts).
>
>![CPM threshold Plots](../../images/rna-seq-counts-to-genes/cpmsvscounts.png "CPM vs Raw Counts Plots")
>
>> <comment-title>Thresholds</comment-title>
>>
>> * A threshold of 1 CPM in at least minimum group sample size is a good rule of thumb for samples with about 10 million reads. For larger library sizes increase the CPM theshold and for smaller library sizes decrease it. Check the CpmPlots to see if the selected threshold looks appropriate for the samples (equivalent to ~10 reads).
>>
>{: .comment}
{: .details}


## Box plots

We can also use box plots to check the distributions of counts in the samples. Box plots can be selected to be output by the Galaxy limma-voom tool if normalization is applied (TMM is applied by default). The plots are output in the `Report` and a link is also provided to a PDF version (`BoxPlots.pdf`). The samples are coloured by the groups. With the box plots for these samples we can see that overall the distributions are not identical but still not very different. If a sample is really far above or below the blue horizontal line we may need to investigate that sample further.

![Box Plots](../../images/rna-seq-counts-to-genes/boxplots.png "Box Plots")

> <question-title></question-title>
>
> Compare the box plots before and after TMM normalisation. Can you see any differences?
>
>    > <solution-title></solution-title>
>    >
>    > After the normalization more of the samples are closer to the median horizontal line.
>    >
>    {: .solution}
{: .question}


> <details-title>Normalization factors</details-title>
>
>The TMM normalization generates normalization factors, where the product of these factors and the library sizes defines the effective library size. TMM normalization (and most scaling normalization methods) scale relative to one sample. The normalization factors multiply to unity across all libraries. A normalization factor below one indicates that the library size will be scaled down, as there is more suppression (i.e., composition bias) in that library relative to the other libraries. This is also equivalent to scaling the counts upwards in that sample. Conversely, a factor above one scales up the library size and is equivalent to downscaling the counts. We can see the normalization factors for these samples in the `Library information` file if we select to output it with *"Output Library information file?"*: `Yes`. Click on the {% icon galaxy-eye %} (eye) icon to view.
>
>![Library Info file](../../images/rna-seq-counts-to-genes/libinfo.png "Library information file")
>
>> <question-title></question-title>
>>
>> Which sample has the largest normalization factor? Which sample has the smallest?
>>
>>    > <solution-title></solution-title>
>>    >
>>    > MCL1.LA has the largest normalization factor and MCL1.LE the smallest.
>>    >
>>    {: .solution}
>{: .question}
{: .details}

> <details-title>MD plots for samples</details-title>
>
>It is considered good practice to make mean-difference (MD) plots for all the samples as a quality check, as described in this [edgeR workflow article](https://f1000research.com/articles/5-1438/v2). These plots allow expression profiles of individual samples to be explored more closely. An MD plot shows the log-fold change between a sample against the average expression across all the other samples. This visualisation can help you see if there are genes highly upregulated or downregulated in a sample. If we look at mean difference plots for these samples, we should be able to see the composition bias problem. The mean-difference plots show average expression (mean: x-axis) against log-fold-changes (difference: y-axis).
>
>Click on the `MDPlots_Samples.pdf` link in the `Report`. You should see 12 MD plots, one for each sample. Let's take a look at the plots for the two samples MCL1.LA and MCL1.LE that had the largest and smallest normalization factors. The MD plots on the left below show the counts normalized for library size and the plots on the right show the counts after the TMM normalization has been applied. MCL1.LA had the largest normalization factor and was above the median line in the unnormalized by TMM box plots. MCL1.LE had the smallest normalization factor and was below the median line in the box plots. These MD plots help show the composition bias problem has been addressed.
>![MD Plot LA](../../images/rna-seq-counts-to-genes/mdsampleLA.png "MD Plots for MCL1.LA before and after TMM normalization")
>![MD Plot LE](../../images/rna-seq-counts-to-genes/mdsampleLE.png "MD Plots for MCL1.LE before and after TMM normalization")
{: .details}

## Voom variance plot

This plot is generated by the voom method and displayed in the `Report` along with a link to a PDF version (`VoomPlot.pdf`). Each dot represents a gene and it shows the mean-variance relationship of the genes in the dataset. This plot can help show if low counts have been filtered adequately and if there is a lot of variation in the data, as shown in the **More details on Voom variance plots** box below.

![Voom Plot](../../images/rna-seq-counts-to-genes/voomplot.png "Voom Plot")

The SA plot below plots log2 residual standard deviations against mean log-CPM values. The average log2 residual standard deviation is marked by a horizontal blue line. This plot shows how the dependence between the means and variances has been removed after the voom weights are applied to the data.

![SA Plot](../../images/rna-seq-counts-to-genes/saplot.png "SA Plot")

> <details-title>More details on Voom variance plots</details-title>
>
> If we didn't filter this dataset for the lowly expressed genes the variance plot would look like below.
>
>   ![Voom Nofilter Plot](../../images/rna-seq-counts-to-genes/voomplot_nofilt.png "Voom Plot unfiltered counts")
>
> If we look at the plot generated with the two samples mixed up we can see there's more variation.
>
>   ![Voom Mixedup Plot](../../images/rna-seq-counts-to-genes/voomplot_mixed.png "Voom Plot mixed-up samples")
>
> More examples of the variation this plot can show can be seen in Figure 1 from the limma-voom paper ({% cite Law2014 %}), shown below.
>
>   ![Voom Plot Examples](../../images/rna-seq-counts-to-genes/voom_variance_examples.jpg "Voom Plot more examples")
>
>   *Figure 1: Mean-variance relationships. Gene-wise means and variances of RNA-seq data are represented by black points with a LOWESS trend. Plots are ordered by increasing levels of biological variation in datasets. (a) voom trend for HBRR and UHRR genes for Samples A, B, C and D of the SEQC project; technical variation only. (b) C57BL/6J and DBA mouse experiment; low-level biological variation. (c) Simulation study in the presence of 100 upregulating genes and 100 downregulating genes; moderate-level biological variation. (d) Nigerian lymphoblastoid cell lines; high-level biological variation. (e) Drosophila melanogaster embryonic developmental stages; very high biological variation due to systematic differences between samples. (f) LOWESS voom trends for datasets (a)–(e). HBRR, Ambion’s Human Brain Reference RNA; LOWESS, locally weighted regression; UHRR, Stratagene’s Universal Human Reference RNA*.
{: .details}

## MD and Volcano plots for DE results

Genome-wide plots that are useful for checking differentially expressed (DE) results are MD plots (or MA plots) and Volcano plots. There are functions in limma for generating these plots and they are used by this tool. These plots are output by default and shown in the `Report` along with a link to PDF versions (`MDPlot_basalpregnant-basallactate.pdf` and `VolcanoPlot_basalpregnant-basallactate.pdf`). In the volcano plot the top genes (by adjusted p-value) are highlighted. The number of top genes is 10 by default and the user can specify the number of top genes to view (up to 100) under **Advanced Options**. For further information on volcano plots see the [Volcano Plot tutorial]({{ site.baseurl }}/topics/transcriptomics/tutorials/rna-seq-viz-with-volcanoplot/tutorial.html)

![MDVol Plot](../../images/rna-seq-counts-to-genes/mdvolplot_basalpregnant-basallactate.png "MD Plot and Volcano Plot")

The MD Plot highlighted genes are significant at an adjusted p-value (adj.P) threshold of 0.05 and exhibit log2-fold-change (lfc) of at least 0. These thresholds can be changed under **Advanced Options**.


> <question-title></question-title>
>
> How many genes are differentially expressed at the default thresholds of adj.P=0.05 and lfc=0?
>
>    > <solution-title></solution-title>
>    >
>    > The number of DE genes at these adj.P and lfc thresholds is shown in the table in the `Report` as below.
>    >
>    > ![DE counts](../../images/rna-seq-counts-to-genes/decounts.png){: width="500px"}
>    >
>    {: .solution}
{: .question}


> <comment-title>on adjusted P value</comment-title>
>
> **A note about deciding how many genes are significant**: In order to decide which genes are differentially expressed, we usually take a cut-off (e.g. 0.05 or 0.01) on the adjusted p-value, NOT the raw p-value. This is because we are testing many genes (more than 15000 genes here), and the chances of finding differentially expressed genes is very high when you do that many tests. Hence we need to control the false discovery rate, which is the adjusted p-value column in the results table. What this means is that, if we choose an adjusted p-value cut-off of 0.05, and if 100 genes are significant at a 5% false discovery rate, we are willing to accept that 5 will be false positives.
{: .comment}

# Testing relative to a threshold (TREAT)

When there is a lot of differential expression, sometimes we may want to cut-off on a fold change threshold, as well as a p-value threshold, so that we follow up on the most biologically significant genes. However, it is not recommended to simply rank by p-value and then discard genes with small logFC’s, as this has been shown to increase the false discovery rate. In other words, you are not controlling the false discovery rate at 5% any more. There is a function called `treat` in limma that performs this style of analysis correctly ({% cite McCarthy2009 %}). TREAT will simply take a user-specified log fold change cut-off and recalculate the moderated t-statistics and p-values with the new information about logFC. There are thousands of genes differentially expressed in this `basalpregnant-basallactate` comparison, so let's rerun the analysis applying TREAT and similar thresholds to what was used in the Fu paper: an adjusted P value of 0.01 (1% false discovery rate) and a log-fold-change cutoff of 0.58 (equivalent to a fold change of 1.5).

> <hands-on-title>Testing relative to a threshold (TREAT)</hands-on-title>
>
> 1. Use the **Rerun** {% icon galaxy-refresh %} button in the History to rerun **limma** {% icon tool %} with the following parameters:
>      - *"**Advanced Options**"*
>          - {% icon param-text %} *"Minimum Log2 Fold Change"*: `0.58`
>          - {% icon param-text %} *"P-Value Adjusted Threshold"*: `0.01`
>          - {% icon param-check %} *"Test significance relative to a fold-change threshold (TREAT)"*: `Yes`
> 2. Add a tag `#treat` to the `Report` output and inspect the report
>
> {% snippet faqs/galaxy/datasets_add_tag.md type="name" %}
>
{: .hands_on}

We can see that much fewer genes are now highlighted in the MD plot and identified as differentially expressed.

![TREAT MDVol Plot](../../images/rna-seq-counts-to-genes/TREAT_mdvolplot_basalpregnant-basallactate.png "TREAT MD and Volcano plots")

![TREAT DE counts](../../images/rna-seq-counts-to-genes/TREATdecounts.png "TREAT differentially expressed genes")


# Visualising results

In addition to the plots already discussed, it is recommended to have a look at the expression levels of the individual samples for the genes of interest, before following up on the DE genes with further lab work. The Galaxy limma tool can auto-generate heatmaps of the top genes to show the expression levels across the *samples*. This enables a quick view of the expression of the top differentially expressed genes and can help show if expression is consistent amongst replicates in the groups.

## Heatmap of top genes

Click on the `Heatmap_basalpregnant-basallactate.pdf` link in the `Report`. You should see a plot like below.

![Heatmap](../../images/rna-seq-counts-to-genes/heatmap.png "Heatmap of top genes")

## Stripcharts of top genes

The limma-voom tool can also auto-generate stripcharts to view the expression of the top genes across the groups. Click on the `Stripcharts_basalpregnant-basallactate.pdf` link in the `Report`. You should see 10 plots, one for each top gene. Four are shown below. Note that here you can see if replicates tend to group together and how the expression compares to the other groups.

![Stripchart Plot](../../images/rna-seq-counts-to-genes/stripcharts.png "Stripcharts of top genes")


## Interactive DE plots (Glimma)

Interactive versions of the MD and Volcano plots can be output by the limma-voom tool via the Glimma package ({% cite Su2017 %}), if a gene annotation file is provided and `Glimma Interactive Plots` is selected. Links to the Glimma html pages are generated in the `Report`.

Let's take a look at the interactive MD plot. Click on the `Glimma_MDPlot_basalpregnant-basallactate.html` link in the `Report`. You should see a two-panel interactive MD plot like below. The left plot shows the log-fold-change vs average expression. The right plot shows the expression levels of a particular gene of each sample by groups (similar to the stripcharts). Hovering over points on left plot will plot expression level for corresponding gene, clicking on points will fix the expression plot to gene. Clicking on rows on the table has the same effect as clicking on the corresponding gene in the plot.

<iframe src="../../images/rna-seq-counts-to-genes/glimma/MD-Plot.html" width="100%" height="1000"></iframe>

> <hands-on-title>Search for a gene of interest</hands-on-title>
>
> `Egf` was a gene identifed as very highly expressed in the Fu paper and confirmed with qRT-PCR, see Fig. 6c from the paper below.
> ![Fu EGF gene](../../images/rna-seq-counts-to-genes/fu_egf.png "Fu et al, Nat Cell Biol 2015")
>
> Search for `Egf` in the Glimma interactive table. You should see something similar to below.
>
> ![Glimma EGF gene](../../images/rna-seq-counts-to-genes/glimma_egf.png "Glimma EGF gene")
>
>  Notice that in the plot on the right above, showing `Egf` expression for all samples, we can see it is highly expressed in the luminal lactate group but not the other samples.
{: .hands_on}

Multiple contrasts can be run with the limma tool. For example, we can compare the pregnant and lactating conditions for both the basal and luminal cells. So let's rerun the limma-voom TREAT analysis (adj.P <0.01 and lfc=0.58) and this time use the `Insert Contrast` button to include the additional contrast `luminalpregnant - luminallactate`. We can then see how the number of differentially expressed genes in the luminal cells compares to the basal cells.

> <hands-on-title>Run multiple contrasts</hands-on-title>
>
> 1. Use the **Rerun** {% icon galaxy-refresh %} button in the History to rerun the **limma** {% icon tool %} `#treat` analysis adding the following parameters *(i.e. run with 2 contrasts)*:
>      - {% icon param-text %} *"Contrast of Interest"*: `basalpregnant-basallactate`
>      - {% icon param-text %} *"Contrast of Interest"*: `luminalpregnant-luminallactate`
> 2. Add a tag `#multiple-contrasts` to the `Report` output and inspect the report
>
>   You should find that there are more genes differentially expressed for the luminal cells than the basal. There are ~274 genes DE in basal cells versus ~ 1610 in the luminal cells.
> ![Basal Luminal DE counts](../../images/rna-seq-counts-to-genes/basal_luminal_decounts.png "Basal vs Luminal DE counts")
>
>   This is similar to what Fu et al found, many more genes differentially expressed in the luminal cells on lactation, compared to the basal cells.
> ![Fu DE genes](../../images/rna-seq-counts-to-genes/fu_degenes.png "Fu et al, Nat Cell Biol 2015")
{: .hands_on}

The tables of differentially expressed genes are output as links in the `Report` (`limma-voom_basalpregnant-basallactate.tsv` and `limma-voom_luminalpregnant-luminallactate.tsv`), see below, and also as datasets in the history (`DE tables`). With multiple contrasts, a plot for each contrast is generated for relevant plots, as shown below. This enables a quick and easy visual comparison of the contrasts.

![Multiple contrasts](../../images/rna-seq-counts-to-genes/multiple_contrasts.png "Multiple contrasts output")

> <details-title>R code in detail</details-title>
>
> 1. The `Report` with all the plots and tables can be downloaded by clicking on the floppy disk icon on the dataset in the history.
> 2. If you want to see the R code used, or to explore some of the outputs in R, you can select to output the Rscript and/or RData files in the tool form.
{: .details}

To see some methods for identifying differentially expressed pathways in this dataset, see the follow-on tutorial [RNA-seq genes to pathways]({% link topics/transcriptomics/tutorials/rna-seq-genes-to-pathways/tutorial.md %}). To see how to create a heatmap of custom genes using this dataset, see the tutorial [Visualization of RNA-Seq results with heatmap2]({% link topics/transcriptomics/tutorials/rna-seq-viz-with-heatmap2/tutorial.md %})

# Conclusion


In this tutorial we have seen how counts files can be converted into differentially expressed genes with limma-voom. This follows on from the accompanying tutorial, [RNA-seq reads to counts]({% link topics/transcriptomics/tutorials/rna-seq-reads-to-counts/tutorial.md %}), that showed how to generate counts from the raw reads (FASTQs) for this dataset. In this part we have seen ways to visualise the count data, and QC checks that can be performed to help assess the quality and results. We have also reproduced results similar to what the authors found in the original paper with this dataset. For further reading on analysis of RNA-seq count data and the methods used here, see the articles; *RNA-seq analysis is easy as 1-2-3 with limma, Glimma and edgeR ({% cite Law2018 %})*  and *From reads to genes to pathways: differential expression analysis of RNA-Seq experiments using Rsubread and the edgeR quasi-likelihood pipeline ({% cite Chen2016 %})*.


topic: transcriptomics
tutorial: clipseq
title: CLIP-Seq data analysis from pre-processing to motif detection
difficulty: intermediate
time_estimation: 6h
tools_json: []
requirements_json: []
tags_json: []
content_hash: e79eef45042d3cb02f4fc544ab7f1ba5
last_modified: 2026-02-26T21:12:18.545053
zenodo_link: https://zenodo.org/record/1327423
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/clipseq/tutorial.html

content:
The eCLIP data provided here is a subset of the eCLIP data of RBFOX2 from a study published by *Nostrand et al.* ({% cite VanNostrand2016 %}). The dataset contains the first biological replicate of RBFOX2 CLIP-seq and the input control experiment (FASTQ files). The data was changed and downsampled to reduce data processing time, consequently the data does not correspond to the original source pulled from *Nostrand et al.* ({% cite VanNostrand2016 %}). Also included is a text file (.txt) encompassing the chromosome sizes of hg38 and a genome annotation (.gtf) file taken from [Ensembl](http://ftp.ensemblorg.ebi.ac.uk/pub/release-74/gtf/homo_sapiens/).

**Table 1**: Data from RBFOX2 CLIP-Seq and control experiment which we are going to use in this tutorial from a study published by *Nostrand et al.* ([2016]({% cite VanNostrand2016 %}). PE stands for a paired-end read library. Hep G2 is a human liver cancer cell line. Replicate is the number of the biological replicate. Stranded means, if the read-library is strand specific, i.e., in a specific forward-reverse direction.

| Cellular state | Datatype | Description | Replicate | ENCODE Accession | Library type | Read length | Stranded? |
| ---            | ---      | :-:     | :-:       | ---           | :-:          | :-:         | :-:       |
| Hep G2            | eCLIP | RBFOX2   | 1         | ENCSR987FTF     | PE           | 175-300          | Yes        |
| Hep G2            | eCLIP | input   | 1         | ENCSR799EKA     | PE           | 175-300          | Yes        |


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Finding Binding Motifs for RBFOX2

RBFOX2 is a relevant development and tissue-specific splicing factor with the conserved motif `TGCATG`. It binds mainly in introns. We therefore want to process and validate the data from human liver cancer cells (HepG2) to find this conserved motif and in the process identify the function of RBFOX2 as well as describe the function of the targeted RNA.

## Step 1: Get data

> <hands-on-title>Data upload</hands-on-title>
> 1. Create and name a new history for this tutorial.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the following files from [Zenodo](https://zenodo.org/record/2579279).
>
>    ```
>    https://zenodo.org/record/2579279/files/hg38_chr_sizes.txt
>    https://zenodo.org/record/2579279/files/Homo_sapiens.GRCh38.87.gtf
>    https://zenodo.org/record/2579279/files/RBFOX2-204-CLIP_S1_R1_RBFOX2.fastq
>    https://zenodo.org/record/2579279/files/RBFOX2-204-CLIP_S1_R2_RBFOX2.fastq
>    https://zenodo.org/record/2579279/files/RBFOX2-204-INPUT_S2_R1.fastq
>    https://zenodo.org/record/2579279/files/RBFOX2-204-INPUT_S2_R2.fastq
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    ![upload](../../images/transcriptomics_images/upload_data_page.png "Data can be imported directly with links.")
>
>   ![data](../../images/transcriptomics_images/clipseq_data_uploaded.png "Imported datasets will appear in the history panel.")
>
{: .hands_on}

# Step 2: Quality Control

As for any NGS data analysis, CLIP-seq data must be quality controlled before being aligned to a reference genome. For more detailed information on NGS quality control, check out the [tutorial for NGS]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}). One crucial thing to check for CLIP-Seq data is the sequence duplication levels. PCR duplications occur naturally in any NGS experiment during the PCR amplification of the genetic material. CLIP-Seq is prone to many PCR duplicates because of the sparse material that is obtained during a CLIP-Seq experiment resulting in high PCR cycles in many occasions. More information about the problem of PCR duplicates can be found [in this guide](http://www.cureffi.org/2012/12/11/how-pcr-duplicates-arise-in-next-generation-sequencing/).

## Report with **FastQC**

> <hands-on-title>Quality control with FastQC</hands-on-title>
>
> 1. **FastQC** {% icon tool %}: Run the tool **FastQC** on each FASTQ file to assess the quality of the raw data. An explanation of the results can be found on the [FastQC web page](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/).
>
>    > <tip-title>Running a tool on multiple data files</tip-title>
>    >
>    > You can run this tool - and many other tools - on all the FASTQ files at once!
>    > To do this, first select the "Multiple datasets" icon (two stacked pages) under the "Input FASTQ file" heading in the **FASTQC** Tool Form, then shift+click to select multiple FASTQ files.
>    {: .tip}
>
> 2. Check the **Sequence Duplication Levels** plot.
>
>   ![fastqbefore](../../images/transcriptomics_images/clipseq_duplication_level_1.png "Sequence duplication levels <b>before</b> de-duplication; The y-axis represents the portion of reads with the specific duplication level. An exact sequence match is needed to detect a duplicated read. The blue line shows the duplication level distribution of the full sequence set. The red line depicts an ideal curve after a de-duplication step (duplicates are filtered out).")
>
>    > <question-title></question-title>
>    >
>    > 1. Has our data high sequence duplication levels?
>    > 2. Why is there still a small spike in the red line?
>    > 3. What does the headline tell you?
>    >
>    > > <solution-title></solution-title>
>    > > 1. Yes, ideally we would see a blue line with one high peak in a duplication of one, everything else should be zero.
>    > > 2. Spikes in the red line come from different duplication levels in the original data (blue line).
>    > > 3. The headline states an expected value of reads that would remain after a de-duplication step. A high percentage suggest, that no de-duplication step is needed. This should also correspond with the red line.
>    > {: .solution }
>    {: .question}
{: .hands_on}

# Step 3: Removal of Adapters, Barcodes and Unique Molecular Identifiers (UMIs)

It is often necessary to remove adapter and barcodes sequences as well as UMIs. <br/>
**Adapters** (or primers) are needed for PCR amplification and sequencing in a standard NGS protocol. Unfortunately, it might happen during the sequencing that the machine does not stop at the read end and sequences through the adapter as well. That is why, we need to check if our reads contain those sequences which we are then cutting out.<br/>
**Barcodes** on the other hand are especially designed for a read library and intentionally sequenced. Sometimes experiments are sequenced at the same time which is called **multiplexing**. Multiplexing allows for a better data normalisation and comparison. The barcodes are then used to divide the un-multiplexed data set into the individual read libraries. (**Note: Our data is already de-multiplexed, i.e., we do not have to take barcode sequences into account.**)<br/>
**UMIs** are similar to barcodes but these sequences are unique for each read. UMIs were introduced since iCLIP to deal with the high duplication levels of a CLIP experiment. Because each read contain an UMI, PCR duplicates of that read also contain the same UMI, which makes it possible to fuse all reads with the same UMI.

## Removal of adapter sequences with **Cutadapt**

In this task we are going to remove two 3' and two 5' adapters from the reads (Note: The eCLIP protocol uses more adapter sequences, for more information take a look in the original publication {% cite VanNostrand2016 %}).

> <hands-on-title>Adapter Removal</hands-on-title>
>
> **Cutadapt (v. 4.0)** {% icon tool %} with the following parameters:
>    - *"Single-end or Paired-end reads?"*: `Paired-end`
>    - {% icon param-file %} *"FASTQ/A file #1"*: `FASTQ R1`
>    - {% icon param-file %} *"FASTQ/A file #2"*: `FASTQ R2`
>    - *"Read 1 Options"*:
>       - In *"3' Adapters"*:
>            - Click on *"Insert 3' Adapters"*:
>            - In *"1: 3' Adapters"*:
>                - *"Source"*: `Enter custom sequence`
>                    - *"Enter custom 3' adapter sequence"*: `AACTTGTAGATCGGA`
>            - Click on *"Insert 3' Adapters"*:
>            - In *"2: 3' Adapters"*:
>                - *"Source"*: `Enter custom sequence`
>                    - *"Enter custom 3' adapter sequence"*: `AGGACCAAGATCGGA`
>        - In *"5' (Front) Adapters"*:
>            - Click on *"Insert 5' (Front) Adapters"*:
>            - In *"1: 5' (Front) Adapters"*:
>                - *"Source"*: `Enter custom sequence`
>                    - *"Enter custom 5' adapter sequence"*: `CTTCCGATCTACAAGTT`
>            - Click on *"Insert 5' (Front) Adapters"*:
>            - In *"2: 5' (Front) Adapters"*:
>                - *"Source"*: `Enter custom sequence`
>                    - *"Enter custom 5' adapter sequence"*: `CTTCCGATCTTGGTCCT`
>       - *"Cut bases from reads before adapter trimming"*: `-5`
>    - *"Read 2 Options"*:
>       - In *"3' Adapters"*:
>            - Click on *"Insert 3' Adapters"*:
>            - In *"1: 3' Adapters"*:
>                - *"Source"*: `Enter custom sequence`
>                    - *"Enter custom 3' adapter sequence"*: `AACTTGTAGATCGGA`
>            - Click on *"Insert 3' Adapters"*:
>            - In *"2: 3' Adapters"*:
>                - *"Source"*: `Enter custom sequence`
>                    - *"Enter custom 3' adapter sequence"*: `AGGACCAAGATCGGA`
>        - In *"5' (Front) Adapters"*:
>            - Click on *"Insert 5' (Front) Adapters"*:
>            - In *"1: 5' (Front) Adapters"*:
>                - *"Source"*: `Enter custom sequence`
>                    - *"Enter custom 5' adapter sequence"*: `CTTCCGATCTACAAGTT`
>            - Click on *"Insert 5' (Front) Adapters"*:
>            - In *"2: 5' (Front) Adapters"*:
>                - *"Source"*: `Enter custom sequence`
>                    - *"Enter custom 5' adapter sequence"*: `CTTCCGATCTTGGTCCT`
>    - *"Adapter Options"*:
>        - *"Minimum overlap length"*: `5`
>    - *"Filter Options"*:
>        - *"Minimum length (R1)"*: `10`
>        - *"Specify a minimum/maximum length for reverse reads (R2)"*: `Disabled`
>
>   > <comment-title>Why do we remove 5 bp from the first read?</comment-title>
>   > In eCLIP it can happen that the sequencing goes over the first read into the UMI, which is at the 3' end of the first read. The UMI is 5 bp long in our data. To make sure our first read in the read pair does not contain the UMI, we simply remove the last 5 bp from it. The UMI that we actually need for the de-duplication is located on the 5' end of our second read in the read pair.
>   {: .comment}
>
>    > <comment-title>Do the same thing for the input control data set.</comment-title>
>    >
>    > If you processed the RBFOX2 FASTQ dataset then do the same thing for the input control dataset or *vice verca*.
>   {: .comment}
>
{: .hands_on}

## Removal of UMIs with **UMI-tools extract**

In this task we are going to remove the UMI at the 5' end of the second read. We are then going to use the removed UMIs later on to de-duplicate our read library and account for PCR duplicates.

> <hands-on-title>UMI Removal</hands-on-title>
>
> 1. **UMI-tools extract** {% icon tool %} with the following parameters:
>    - *"Library type"*: `Paired-end`
>     - {% icon param-file %} *"Reads in FASTQ format"*: `FASTQ R2` (output from the **Cutadapt** call {% icon tool %})
>     - {% icon param-file %} *"Reads in FASTQ format"*: `FASTQ R1`  (output from the **Cutadapt** call {% icon tool %})
>     - *"Barcode on both reads?"*: `Barcode on first read only`
>    - *"Use Known Barcodes?"*: `No`
>    - *"Method to extract barcodes"*: `String`
>    - *"Barcode pattern for first read"*: `NNNNN`
>    - *"Is the barcode at the 5' end?"*: `Yes`
>    - *"Output log?"*: `Yes`
>    - *"Enable quality filter?"*: `No`
>
>   > <comment-title>Do the same thing for the input control data set.</comment-title>
>   >
>   > If you processed the RBFOX2 FASTQ dataset then do the same thing for the input control dataset or *vice verca*.
>   {: .comment}
>
>   > <question-title></question-title>
>   > 1. What is the meaning of the barcode pattern?
>   >
>   >   > <solution-title></solution-title>
>   >   > 1. The symbol "N" defines our UMI. The symbol "X" are nucleotides that we would reattach to our read. It is important to note, that the UMI can be encompassed by an in-line barcode, that is why we need two distinct symbols "N" and "X".
>   > {: .solution }
>   {: .question}
>
{: .hands_on}

# Step 4: Aligning Reads to a Reference Genome

To determine where DNA fragments originated in the genome, the sequenced reads must be aligned to a reference genome. This is equivalent to solving a jigsaw puzzle, but unfortunately, not all pieces are unique. In principle, you could do a BLAST analysis to figure out where the sequenced pieces fit best in the known genome. However, this can take a couple of weeks. Nowadays, there are many read alignment programs. STAR {% cite Dobin2012%} is one of them that works well with CLIP-Seq data. STAR is able to use genome as well as transcriptome data. This ability is handy, since CLIP-Seq generates transcriptome data, consequently we have to take RNA processing steps like splicing events into account.

## Aligning with **RNA STAR**

> <hands-on-title>Alignment</hands-on-title>
>
> 1. **RNA STAR** {% icon tool %} with the following parameters:
>    - *"Single-end or paired-end reads"*: `Paired-end (as individual datasets)`
>     - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file, forward reads"*: `FASTQ R2` (output from **UMI-tools extract** {% icon tool %})
>     - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file, reverse reads"*: `FASTQ R1` (output from **UMI-tools extract** {% icon tool %})
>    - *"Custom or built-in reference genome"*: `Use a built-in index`
>        - *"Reference genome with or without an annotation"*: `use genome reference with builtin gene-model`
>            - *"Select reference genome"*: `Homo sapiens (hg38+GRCh38.87)`
>    - *"Count number of reads per gene"*: `No`
>    - *"Would you like to set output parameters (formatting and filtering)?"*: `Yes`
>        - *"Extra SAM attributes to include"*: `All`
>        - *"Include strand field flag XS"*: `Yes -- and reads with inconsistent and/or non-canonical introns are filtered out`
>        - *"Would you like to set additional output parameters (formatting and filtering)?"*: `Yes`
>    - *"Other parameters (seed, alignment, limits and chimeric alignment)"*: `Extended parameter list`
>        - In *"Alignment parameters"*:
>            - *"Use end-to-end read alignments, with no soft-clipping?"*: `Yes`
>        - *"Would you like to set chimeric alignment parameters?"*: `No`
>
>   > <comment-title>Note: We have switched R1 and R2 as forward and reverse reads!</comment-title>
>   >
>   > We need to do that because the eCLIP read library is organised in a way such that the first mate (R1) is in reverse and the second mate (R2) in forward orientation.
>   {: .comment}
>
>   > <comment-title>Soft-Clipping vs Hard-Clipping</comment-title>
>   >
>   > Clipping is a way to deal with low quality bases during the alignment step. In **Soft-Clipping** the bases at the 5' and 3' end of the read are not part of the alignment. In **Hard-Clipping** the bases at the 5' and 3' end of the read are not part of the alignment **and** will be completely removed from the read sequence in the BAM file.
>   {: .comment}
>
>   > <comment-title>Do the same thing for the input control data set.</comment-title>
>   >
>   > If you processed the RBFOX2 FASTQ dataset then do the same thing for the input control dataset or *vice verca*.
>   {: .comment}
>
>   > <question-title></question-title>
>   >
>   > 1. Can you think of a reason why we disabled the soft-clipping?
>   >
>   >   > <solution-title></solution-title>
>   >   >
>   >   > 1. In eCLIP the cross linking position should be at the beginning of the second read. If we would enable soft-clipping, we would add potential bases with low quality at the end of our second reads that would blur our cross linking position and we would lose precision to detect potential binding regions of RBFOX2.
>   > {: .solution}
> {: .question}
>
{: .hands_on}

# Step 5: De-Duplication

Lets return to the UMIs which we extracted in step three. Since we have mapped the reads to our reference genome, we can now identify which reads might be duplicated or not. The UMI-tools {% cite Smith2017 %} will help us again in this matter.

## De-duplication with **UMI-tools deduplicate**

> <hands-on-title>De-Duplication</hands-on-title>
>
> 1. **UMI-tools deduplicate** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Reads to deduplicate in SAM or BAM format"*: `bam` (output of **RNA STAR** {% icon tool %})
>    - *"extract_umi_method"*: `Read ID`
>    - *"Separator between read id and UMI."*: `_`
>    - *"Method used to identify PCR duplicates within reads."*: `Identify clusters based on hamming distance and resolve networks by using the node counts`
>    - *"Edit distance threshold"*: `1`
>    - *"BAM is paired end"*: `Yes`
>
>   > <comment-title>What is the purpose of the method we have chosen for the de-duplication?</comment-title>
>   >
>   > `UMI-tools deduplication` has several methods. The method we have picked is called the  **adjacency** method. For detailed information have a look at method descriptions of UMI-tools {% cite Smith2017 %}. For a brief explanation: the method fuses reads together when they have the same coordinates and the same UMI. However, sequencing errors can occur in the UMI. Thus, in the **adjacency** method we fuse also UMIs that differ in a maximal number of characters and where we identify a lot of copies, i.e., the method creates clusters of nodes, a node for each individual UMI, and fuses these nodes based on the [hamming distance](https://en.wikipedia.org/wiki/Hamming_distance) and read counts.
>   {: .comment}
>
>   > <comment-title>Do the same thing for the input control data set.</comment-title>
>   >
>   > If you processed the RBFOX2 bam file then do the same thing for the input control bam or *vice verca*.
>   {: .comment}
>
>   > <question-title></question-title>
>   >
>   > 1. Why are we doing the de-duplication after the read mapping?
>   > 2. What is the edit distance threshold?
>   >
>   >   > <solution-title></solution-title>
>   >   >
>   >   > 1. Because we need the coordinates of the reads to see if some reads have the same coordinates and the same UMI.
>   >   > 2. `UMI-tools dedulication` uses the hamming distance between UMIs to detect sequencing errors in the UMIs. A hamming distance of 1 means, that two UMIs are the same if they differ in maximal one character. If two characters are different, then these are two disparate UMIs.
>   > {: .solution}
> {: .question}
>
{: .hands_on}

## Quality check of the de-duplication with **FastQC**

> <hands-on-title>De-duplication quality control</hands-on-title>
>
> 1. **FastQC** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Short read data from your current history"*: `bam` (output of **UMI-tools deduplicate** {% icon tool %})
> 2. View the **Sequence Duplication Levels** plot.
> ![fastqafter](../../images/transcriptomics_images/clipseq_duplication_level_2.png "Sequence duplication levels <b>after</b> de-duplication.")
>
>   > <question-title></question-title>
>   >
>   > 1. What can you see, if you compare the duplication levels to our first quality control in step two?
>   > 2. Why are the duplication levels still high?
>   >
>   >   > <solution-title></solution-title>
>   >   >
>   >   > 1. We have reduced the duplication levels.
>   >   > 2. The duplication levels are still high, because we don't have the full data sets (reads are missing).
>   > {: .solution}
> {: .question}
>
{: .hands_on}

# Step 6: Second Quality Control

In this section we check the quality of our mapped reads and see if our samples are correlated or not. It is actually not really necessary for our data sets, because of the small sample size. However, for bigger analyses it is wise to check if some samples may encompass major quality problems.

## Check the read coverage with **plotFingerprint**

> <hands-on-title>Check the read coverage</hands-on-title>
>
> 1. **plotFingerprint** {% icon tool %} with the following parameters:
>    - *"Sample order matters"*: `No`
>    - *"Bam file"*: `both bam files` (output of **UMI-tools deduplicate** {% icon tool %})
>    - *"Show advanced options"*: `yes`
>     - *"Bin size in bases"*: `100`
>    - *"Show advanced output settings"*: `no`
> 2. View the output image.
> ![fingerprint](../../images/transcriptomics_images/clipseq_fingerprint.png "Graph of IP strength from <b>plotFingerprint</b>. The y-axis represents the fraction of the total number of reads. `plotFingerprint` randomly samples genome regions (bins) of a specified length and sums the per-base coverage that overlap with those regions. The x-axis therefore orders the bins from lowest coverage to highest coverage. The plot here is too idealistic which is because of our small data set.")
>
>   > <comment-title>What does this graph represent, especially for CLIP-Seq data?</comment-title>
>   >
>   > It shows us how good the CLIP Signal compared to the control signal is. Now be careful, CLIP-Seq experiments involve either a total RNA control or a negative control with another protein that unspecifically binds RNA (e.g., IgG). An ideal total RNA control (input control) like ours with perfect uniform distribution of reads along the genome/transcriptome (i.e. without enrichments) and infinite sequencing coverage should generate a straighter, diagonal line. On the other hand, a very specific and strong CLIP enrichment will be indicated by a prominent and steep rise of the cumulative sum towards the highest rank. Yet, a negative control often has the same sharp slope at the end as a CLIP experiment but often depicts a straighter, diagonal line in the beginning like the input control.
>   {: .comment}
>
>   > <question-title></question-title>
>   >
>   > 1. How do (or should) input datasets differ from CLIP datasets?
>   > 2. What do you think about the quality of the CLIP experiment?
>   >
>   >   > <solution-title></solution-title>
>   >   >
>   >   > 1. We expect that the control (input) has a different, more straighter, diagonal slope than the CLIP experiment.
>   >   > 2. Both the CLIP experiment and the control are closely related. It is maybe wise to check the correlation of the two samples.
>   > {: .solution}
> {: .question}
>
{: .hands_on}

For additional information on how to interpret **plotFingerprint** plots, read the information [the DeepTools documentation](https://deeptools.readthedocs.io/en/latest/content/tools/plotFingerprint.html#background).

## Correlation between Samples

We have seen that our input control and our CLIP experiment might be strongly correlated, meaning, that our potential RBFOX2 binding regions are not truly enrichment when comparing to a our control.
We are therefore going to further check the correlation between our control and CLIP experiment.

> <hands-on-title>Assessing correlation between samples</hands-on-title>
>
> 1. **multiBamSummary** {% icon tool %} with the following parameters:
>    - *"Sample order matters"*: `No`
>    - *"Bam file"*: `both bam files` (output of **UMI-tools deduplicate** {% icon tool %})
>    - *"Choose computation mode"*: `Bins`
>     - *"Bin size in bp"*: `1000`
>    - *"Show advanced options"*: `no`
> 2. **plotCorrelation** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Matrix file from the multiBamSummary tool"*: `outFile` (output of **multiBamSummary** {% icon tool %})
>    - *"Correlation method"*: `Spearman`
>    - *"Plotting type"*: `Heatmap`
>    - *"Skip zeros"*: `Yes`
>    - *"Image file format"*: `png`
> 3. View the output image.
> ![correlation](../../images/transcriptomics_images/clipseq_correlation.png "Heatmap of correlation matrix generated by <b>plotCorrelation</b>. The y-axis shows you on the left site the histogram of the clustering. The colored scale depicts the correlation value. Like  `plotFingerprint`, `multiBamSummary` randomly samples genome regions (bins) of a specified length and sums the per-base coverage that overlap with those regions. The correlation between these bins is then calculated for each pair of samples. An ideal plot would result in two disparate clusters: one for the biological replicates of your CLIP-Seq experiment and one for the replicates of your control. A bad plot would happen if your control and CLIP-Seq experiment end up in one cluster, which means that they are closely related and something went wrong in the laboratory.")
>
>   > <question-title></question-title>
>   >
>   > 1. Why do we want to skip zeros in plotCorrelation?
>   > 2. What happens if the Spearman correlation method is replaced by the Pearson method?
>   > 3. What does the correlation tell you about our data?
>   >
>   >   > <solution-title></solution-title>
>   >   >
>   >   > 1. Large areas of zeros would lead to a correlation of these areas. The information we would get out of this computation would be meaningless.
>   >   > 2. The clusters would be different.
>   >   > 3. Despite the result of `plotFingerprint`, we see that the correlation between our experiment and control is not nearly as significant as we thought.
>   > {: .solution}
> {: .question}
>
{: .hands_on}

# Step 7: Peak Calling

Peak calling is one of the most important steps in the data analysis of CLIP-Seq, next to mapping. There exist a variety of peak callers like [PEAKachu](https://github.com/tbischler/PEAKachu), PureCLIP ({% cite Krakau2017 %}) or Piranha ({% cite Uren2012 %}) each with different underlying assumptions and parameters. It is difficult (nearly impossible) to find a generic solution for every data set. We are going to use `PEAKachu` to find possible binding motifs for our data. `PEAKachu` is able to incorporate control data in contrast to other peak callers like `Piranha`, thus allowing to find binding regions that are significantly enriched in comparison to our control (input) data.

## Peak calling with **PEAKachu**

> <hands-on-title>Peak calling</hands-on-title>
>
> 1. **PEAKachu** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Experiment Libraries"*: `bam RBFOX2` (output of **UMI-tools deduplicate** {% icon tool %})
>    - {% icon param-file %} *"Control Libraries"*: `bam Input` (output of **UMI-tools deduplicate** {% icon tool %})
>    - *"Pairwise Replicates"*: `No`
>    - *"Paired End"*: `Yes`
>    - *"Maximum Insert Size"*: `"200"`
>    - *"Select Mode"*: `Adaptive`
>        - *"Normalisation Method."*: `DESeq2`
>    - *"Mad Multiplier"*: `"0.0"`
>    - *"Fold Change Threshold"*: `"2.0"`
>    - *"Adjusted p-value Threshold"*: `"0.05"`
> 2. Take a look at the MA plot of **PEAKachu**.
> ![ma](../../images/transcriptomics_images/clipseq_ma_plot.png "MA plot of <b>PEAKachu</b>.  MA stands for M (log ratio) and A (mean average). It shows you the general trend of the log2 fold-change in dependence of the average mean of the expression rate of the peaks. Points in a straight, diagonal line symbolise the behavior of the logarithm which is often visible for low expressed peaks, because one read can change the fold-change quite drastically. The blue lines depict the normalisation constants. Read dots are significant peaks. It is hard to tell what a good MA plot would look like. Even a small number of significant peaks can be enough, depending on the biological question you ask.")
>
>   > <comment-title>Adjusted p-value Threshold</comment-title>
>   >
>   > P-values are calculated by DESeq2. A low p-value represent a significantly enriched binding region. Since we are doing thousands of independent hypothesis testings, we have to correct for significant p-values that happen just by chance. DESeq2 uses the Benjamini-Hochberg procedure as a correction. For more information read [this handbook](http://www.biostathandbook.com/multiplecomparisons.html).
>   {: .comment}
>
>   > <comment-title>Fold-Change Threshold</comment-title>
>   >
>   > The fold-change is calculated by DESeq2 and is actually a log2 fold-change. Because we want significantly enriched binding regions, we basically search for peaks that are at least four times higher in our CLIP experiment than in our control.
>   {: .comment}
>
>   > <question-title></question-title>
>   >
>   > 1. What happens if we decrease or increase the log2 fold-change threshold of 2.0?
>   > 2. Take a look into the peak output file of PEAKachu. Why aren't there any p-values?
>   > 3. Is the log2 fold-change alone a good measurement to validate our peaks? Maybe the MA plot helps you to figure out this question?
>   >
>   > > <solution-title></solution-title>
>   > >
>   > > 1. If we would **decrease** the log2 fold-change threshold then we would get more peaks. However, we would also increase the number of false positives, i.e., peaks that are **not** significantly enriched.<br/>
>   > >    If we would **increase** the log2 fold-change threshold, then we would decrease the number of peaks and possible false positives. However, we would also increase the number of false negatives, i.e., we would lose potentially interesting binding sites.
>   > > 2. PEAKachu works mainly with replicated data, because DESeq2 calculates only p-values for data sets where the experiment and control have at least two replicates. This feature makes statistically sense, because data sets without replicates are not representative enough to make general assumptions, but for the purpose of demonstration we are going to ignore this fact.
>   > > 3. Yes and no. A significant fold-change with high read counts (read coverage) might be enough to validate a peak. However, some peaks can also be covered by very few reads and still have a high fold-change, e.g., log2(4/1). It is for you to decide if a fold-change like log2(4/1) is enough evidence that the binding region is significantly enriched.
>   > {: .solution}
>   {: .question}
>
{: .hands_on}

# Step 8: Peak Analysis

In this last step, we are going to analyse the peaks that we obtained from the previous step. This should give you a small glimpse of what someone might be interested in. Therefore we want to answer in this step the following questions:
1) Which sequential motifs are potentially conserved in our binding regions, i.e., what are potential binding motifs of RBFOX2? 2) Which RNAs does RBFOX2 preferentially bind and what are their biological function?

## Motif detection with **MEME-ChIP**

Lets first find out which sequence motifs RBFOX2 might preferentially bind to.

> <hands-on-title>Motif detection</hands-on-title>
>
> 1. **Text reformatting** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `tabular` (output of **PEAKachu** {% icon tool %})
>    - *"AWK Program"*: `NR>1{if($3 < $4) {print $1,$3,$4,"clip_peak_"NR-1,$9,$5;}else {print $1,$4,$3,"clip_peak_"NR-1,$9,$5;}}`
> 2. Set the datatype of the output generated by **Text reformatting** as `bed`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="bed" %}
>
> 3. **SlopBed** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"BED/VCF/GFF file"*: `bed` (output of **Text reformatting** {% icon tool %})
>    - *"Genome file"*: `hg38_chr_sizes.txt` (file from your history)
>    - *"Choose what you want to do"*: `Increase the BED/GFF/VCF entry by the same number base pairs in each direction.`
>        - *"Number of base pairs"*: `20`
> 4. **Extract Genomic DNA** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Fetch sequences for intervals in"*: `bed` (output of **SlopBed** {% icon tool %})
>    - *"Interpret features when possible"*: `Yes`
>    - *"Choose the source for the reference genome"*: `locally cached`
>        - *"Using reference genome"*: `hg38`
>    - *"Select output format"*: `fasta`
> 5. **MEME-ChIP** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Primary sequences"*: `FASTA` (output of **Extract Genomic DNA** {% icon tool %})
>    - *"Sequence alphabet"*: `DNA`
>    - *"Options Configuration"*: `Advanced`
>        - *"Limit of sequences to pass to MEME"*: `100`
>        - *"Should subsampling be random?"*: `Yes`
>            - *"Seed for the randomized selection of sequences"*: `123`
>        - *"E-value threshold for including motifs"*: `0.05`
>        - *"Search given strand only"*: `Yes`
>        - *"What is the expected motif site distribution?"*: `Zero or one occurances per sequence`
>        - *"Minimum motif width"*: `5`
>        - *"Maximum motif width"*: `20`
>        - *"Maximum number of motifs to find"*: `20`
>        - *"Stop DREME searching after reaching this E-value threshold"*: `0.05`
>        - *"Stop DREME searching after finding this many motifs"*: `5`
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
> 6. View the **MEME-ChIP** html.
>
> ![motif](../../images/transcriptomics_images/clipseq_motif.png "Plot of a sequence motif. The x-axis of the sequence plots represents the nucleotide position of the motif. The y-axis stands for the total information (uncertainty) of each position and thus stands for the probability that the nucleotide at a certain position is the specific letter (for DNA: T,C,G,A). Bigger letters stand for a higher probability. For more information read [this Wikipedia article](https://en.wikipedia.org/wiki/Sequence_logo).")
>
>   > <comment-title>MEME and DREME</comment-title>
>   >
>   > **MEME** and **DREME** are two motif finding tools that MEME-ChIP uses. More information about theses tools can be found on [meme-suite.org](http://meme-suite.org/).
>   {: .comment}
>
>   > <comment-title>What is the purpose of SlopBED?</comment-title>
>   >
>   > **PEAKachu** might underestimate the length of the binding regions, because the actually protein-RNA-binding concentrates on the cross linking site that is one nucleotide long. Thus, we extend the peaks from **PEAKachu** by 20 bp at each site. A true conserved binding motif will not be affected by it, if we make the region to wide.
>   {: .comment}
>
>   > <comment-title>What is the meaning of the E-value?</comment-title>
>   >
>   > The E-value represents the expected number of times we would find our sequence motif in a database (peak file), just by chance. This means, that a small E-value correspond to a very significant motif, because the expected number we would find that motif in our peak file just by chance is very low. Other sequences like repeats will have a high E-value on the other hand.
>   {: .comment}
>
>   > <question-title></question-title>
>   >
>   > 1. Click on the link under the column **Discovery/​Enrichment Program** for two of the motifs one from **MEME** and one from **DREME**. What is the site-count for the most significant motif? How many sequences have we checked for **MEME** and for **DREME**? What is the fraction of sequences that covered that motif? (MEME-ChIP version 4.11.2)
>   > 2. Have we found the motif `TGCATG`?
>   >
>   >   > <solution-title></solution-title>
>   >   >
>   >   > 1. For the result with MEME-ChIP version 4.11.2: **MEME** site count = 37, total sequences = 100, fraction = 37%; **DREME** site count = 103, total sequences = 247, fraction = 41.7%.
>   >   > 2. Yes.
>   > {: .solution}
> {: .question}
>
{: .hands_on}

## Functional analysis with **RNA Centric Annotation System**

Lets move to the next step and find out where these motifs might come from, i.e., which kind of RNAs RBFOX2 preferentially bind to. For that we will use the `RNA Centric Annotation System`. Run the tool and then try to answer the subsequent questions.

> <hands-on-title>Functional analysis</hands-on-title>
>
> 1. **RNA Centric Annotation System (v. 1.5.4)** {% icon tool %} with the following parameters:
>    - *"Genome Version"*: `hg38`
>    - {% icon param-file %} *"Target regions in BED format"*: `bed` (output of **SlopBed** {% icon tool %})
>    - {% icon param-file %} *"Reference annotation in ENSEMBL GTF format"*: `Homo_sapiens.GRCh37.74.gtf` (file from your history)
>    - *"Run annotation."*: `Yes`
>    - *"Run GO term enrichment"*: `Yes`
>    - *"Run gene set enrichment"*: `No`
>    - *"Run motif search"*: `No`
>    - *"Downsampling (N)"*: `"0"`
> 2. View the output html of **RNA Centric Annotation System**
>
>   > <question-title></question-title>
>   >
>   > 1. What is the main target region of RBFOX2 (e.g., cds, exon, 5' UTR)? Does this confirm the literature.
>   > 2. From which cellular compartments do the targets come from?
>   > 3. Where does RBFOX2 mainly bind around the intron-exon boundaries and total transcript?
>   >
>   >   > <solution-title></solution-title>
>   >   >
>   >   > 1. Introns and transcripts. Yes.
>   >   > 2. Actin filaments, actin cytoskeleton, cell projection part etc.
>   >   > 3. Around the exon-intron boundaries more at the 3' ends of transcripts.
>   > {: .solution}
> {: .question}
>
{: .hands_on}

# Additional Step

Sometimes it is wise to take a look at individual peaks; maybe to check the peak calling algorithm or the general quality of the read coverage of the called peaks. We will cover the motivation behind this step in this section and explain why it would be wise to further inspect your results. For this we take the de-duplicated reads in step 5 and the called peaks in step 7 to take a deeper look inside our data.

## Generating bigWig files of the alignment ends

We will first generate coverage files that gives us a better visualization for our read coverage along the genome. SAM or BAM files can be used too, but they are often to big and harder to handle, if you try to inspect the read coverage. Therefore we are going to generate a bigWig (binary form of a BedGraph) file.

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Extract alignment ends** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Alignments in SAM or BAM format"*: `bam` (output of **UMI-tools deduplicate** {% icon tool %})
> 2. **SortBED** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Sort the following BED file"*: `bed` (output of **Extract alignment ends** {% icon tool %})
>    - *"Sort by"*: `chromosome, then by start position (asc)`
> 3. **Create a BedGraph of genome coverage** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"The BAM or BED file from which coverage should be computed"*: `bed` (output of **SortBED** {% icon tool %})
>    - *"Report regions with zero coverage"*: `No`
>    - *"Treat split/spliced BAM or BED12 entries as distinct BED intervals when computing coverage."*: `No`
>    - *"Calculate coverage based on"*: `both strands combined`
> 4. **Wig/BedGraph-to-bigWig** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Convert"*: `bedgraph` (output of **Create a BedGraph of genome coverage** {% icon tool %})
>    - *"Converter settings to use"*: `Default`
>
>   > <comment-title>Do the Same thing for the input control data set.</comment-title>
>   >
>   > If you processed the RBFOX2 bam dataset then do the same thing for the input control dataset or *vice verca*.
> {: .comment}
>
{: .hands_on}

## Inspection of peaks and aligned data with IGV

Next we are going to use the generated bigWig files of our alignment ends and the peaks we obtained from `PEAKachu` and investigate the read coverage of our called peaks with [IGV](http://software.broadinstitute.org/software/igv/). This is an alternative to Trackster. For more information about IGV follow the  [tutorial]({{site.baseurl}}/topics/introduction/tutorials/igv-introduction/tutorial.html).<br/>

**Try to answer the questions for this hands-on.** It will give you an idea why it would be wise to inspect the read coverage of the peaks.

> <hands-on-title>IGV</hands-on-title>
>
> 1. Open IGV on your local computer.
> 2. Choose as a reference genome **hg38**.
> 3. Download and drag-and-drop the following files from galaxy into IGV:
>     - **peaks_annotations.gff** of PEAKachu {% icon tool %}
>     - **CLIP.bigWig** of Wig/BedGraph-to-bigWig {% icon tool %}
>     - **INPUT.bigWig** of Wig/BedGraph-to-bigWig {% icon tool %}
> 4. Inspect the peaks (e.g., <b>chr11:peak_2</b> chr11:2152602-2152742)
> ![igv_peak](../../images/transcriptomics_images/clipseq_peak_igv.png "Inspection of a peak in IGV.")
>
>   > <question-title></question-title>
>   >
>   > 1. What are the ideal conditions that PEAKachu {% icon tool %} will call a peak.
>   > 2. What are the challenges a peak caller has to face? What defines a peak (significantly enriched region)?
>   > 3. Imagine you find a region that should be definitely be a peak but your peak caller does not spot it. What can you do?
>   >
>   >   > <solution-title></solution-title>
>   >   >
>   >   > 1. High read coverage in the region for the CLIP experiment and low read coverage in the region for the input control.
>   >   > 2. As simply explained in the previous answer, a peak (significantly binding region) should be significantly enriched in the CLIP experiment and not in the input control. However, the difficulty is to find out if two adjacent peaks are two distinct peaks or just one big peak. Furthermore, very small bumps can also be just an artifact of a way bigger peak. In addition, a high read coverage in the CLIP experiment and not in the input control does not necessarily mean that we have an enriched region. The significance depends on the background (noise) of our data. A good normalisation is therefore crucial.
>   >   > 3. From question two you might know now, that this is not an easy question. Every peak caller has different parameters that can be optimised for your type of problem. If you find a spot that you think should be certainly a peak, than play around with the parameters and options of the peak caller, or switch to another algorithm.
>   > {: .solution}
> {: .question}
>
{: .hands_on}

# Conclusion


In this tutorial you imported raw eCLIP data, evaluated the quality of the read library before the read processing, removed adapter sequences and unique molecular identifier from the reads, aligned the reads to a reference genome, de-duplicated the read library, analysed the quality of the de-duplication and read mapping, found significantly enriched regions (peaks) for the protein RBFOX2 and finally scrutinised the peaks on potential conserved motifs and target RNAs. In addition, you took a short glimpse on the quality of the peak calling. Keep in mind, the read library for CLIP-Seq data might have different formats for different protocols (e.g, PAR-CLIP). However, the overall analysis you learned in this tutorial stays nearly the same.

### Remark about Workflow


In order to use the workflow linked to this tutorial, you have to create two lists of dataset pairs; one for you control and one for your CLIP experiment.

> <tip-title>Creating a list of dataset pairs.</tip-title>
>
> * Click on the check box icon in the history.
> ![builddatasetpairs1](../../images/transcriptomics_images/clipseq_build_list_data_set_pairs_1.png "Select data sets for your list of data set pairs.")
> * Select the data files for your experiment (or control).
> * Select **n of N selected** right to the check box icon.
> * Select **Build List of Dataset Pairs**
> * In the new window, if you see no files and a warning, then select **Clear filters**. After that, organise the files into forward-reverse pairs.
> ![builddatasetpairs2](../../images/transcriptomics_images/clipseq_build_list_data_set_pairs_2.png "Organise the data sets into pairs.")
>
>  ![builddatasetpairs3](../../images/transcriptomics_images/clipseq_build_list_data_set_pairs_3.png "Result after organising the data sets into pairs.")
> * Give the list a name and click on the button **Create list**.
{: .tip}


topic: transcriptomics
tutorial: rb-rnaseq
title: Reference-based RNAseq data analysis (long)
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: b82a17460da9dbceb523e279234f6063
last_modified: 2026-02-26T21:12:18.560828
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/rb-rnaseq/tutorial.html

content:
# RNAseq: Reference-based

This tutorial is inspired by an exceptional [RNAseq course](http://chagall.med.cornell.edu/RNASEQcourse/) at the Weill Cornell Medical College compiled by Friederike Dündar, Luce Skrabanek, and Paul Zumbo and by tutorials produced by Björn Grüning (@bgruening) for Freiburg Galaxy instance. Much of Galaxy-related features described in this section have been developed by Björn Grüning (@bgruening) and configured by Dave Bouvier (@davebx).

RNAseq can be roughly divided into two "types":

 * **Reference genome-based** - an assembled genome exists for a species for which an RNAseq experiment is performed. It allows reads to be aligned against the reference genome and significantly improves our ability to reconstruct transcripts. This category would obviously include humans and most model organisms but excludes the majority of truly biologically intereting species (e.g., [Hyacinth macaw](https://en.wikipedia.org/wiki/Hyacinth_macaw));
 * **Reference genome-free** - no genome assembly for the species of interest is available. In this case one would need to assemble the reads into transcripts using *de novo* approaches. This type of RNAseq is as much of an art as well as science because assembly is heavily parameter-dependent and difficult to do well.

In this lesson we will focus on the **Reference genome-based** type of RNA seq.

## Experimental procedures affect downstream analyses

The *Everything's connected* slide by Dündar et al. (2015) explains the overall idea:

![Everything's connected](../../images/transcriptomics_images/everything_connected.png)

There is a variety of ways in which RNA is treated during its conversion to cDNA and eventual preparation of sequencing libraries. In general the experimental workflow includes the following steps:

 * RNA purification;
 * Reverse transcription using Reverse Transcriptase (RT), which produces the first strand of cDNA ("c" stands for *complimentary*);
 * Second strand synthesis using DNA polymerase;
 * Library preparation for sequencing.

In listing these basic steps we are ignoring a vast amount of details such as, for example, normalization strategies and procedures needed to deal with rare RNAs or degraded samples (see [Adiconis:2013](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3821180/)). Yet, there are two important experimental considerations that would effect the ways in which one analyses data and interprets the results. These are:

 * Priming for the first cDNA strand synthesis;
 * Stranded versus Non-stranded libraries.

### Priming for the first strand synthesis

Reverse Transcriptase (RT) requires a primer. One can leverage the fact that the majority of processed mRNAs are polyadenylated and use oligo-dT primer to (mostly) restrict cDNA synthesis to fully processed mRNAs. Alternatively one can use a mix of random oligonucleotides to prime RT at a multitude of internal sites irrespective of RNA type and maturation status:


![Oligo-dT vs. random priming](../../images/transcriptomics_images/dT_random.png)

**Oligo-dT vs. random priming**<br>
Oligo-dT (**A**) and random priming (**B**)

Depending on the choice of the approach one would have different types of RNAs included in the final sequencing outcome. For example, if one attempts to study RNAs that are not polyadenylated or not fully processed, it would be unwise to use oligo-dT priming approach.

### Strand-specific RNAseq

RNAs that are typically targeted in RNAseq experiments are single stranded (e.g., mRNAs) and thus have polarity (5' and 3' ends that are functionally distinct):

![Relationship between DNA and RNA orientation](../../images/transcriptomics_images/dna_rna.png)

**Relationship between DNA and RNA orientation**

During a typical RNAseq experiment the information about strandedness is lost after both strands of cDNA are synthesized, size selected, and converted into sequencing library. However, this information can be quite useful for various aspects of RNAseq analysis such as transcript reconstruction and quantification. There is a number of methods for creating so called *stranded* RNAseq libraries that preserve the strand information (an excellent overview in [Levin et al, Nat Meth, 2010](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3005310/)):

![Generation of stranded RNAseq libraries](../../images/transcriptomics_images/stranded_protocols.png "Generation of stranded RNAseq libraries (Levin et al, Nat Meth, 2010)")

****<br>
Different types of stranded library generation protocols from [Levin:2010](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3005310/)

Depending on the approach and whether one performs single- or paired-end sequencing there are multiple possibilities on how to interpret the results of mapping of these reads onto genome/transcriptome:

[![Effects of RNAseq library types](../../images/transcriptomics_images/lib_type.png)](https://sailfish.readthedocs.org/en/master/library_type.html)

**Effects of RNAseq library types**<br>
Image and description below is from [Sailfish documentation](https://sailfish.readthedocs.org/en/master/library_type.html)

The relative orientation of the reads is only relevant if the library is pair-ended. The possible options are:

 * **I** = inward;
 * **O** = outward;
 * **M** = matching (co-directional).

Library can be stranded (**S**) or unstranded (**U**). If this library is stranded than depending on the protocols reads (single reads or forward reads in a paired-end run) may originate from:

 * **F** = read 1 in paired-end sequencing or single-end read is derived from the Forward strand;
 * **R** = read 1 in paired-end sequencing or single-end read is derived from the Reverse strand.

So by combining the relative orientation of reads is I, O, or M (if reads are paired), strandedness or the library (S or U), and whether the reads originate from forward and reverse strand (F or R) there can be quite a number of possibilities:

* IU  - (an unstranded paired-end library where the reads face each other)
* SF  - (a stranded single-end protocol where the reads come from the forward strand)
* OSR - (a stranded paired-end protocol where the reads face away from each other,
      read1 comes from reverse strand and read2 comes from the forward strand).
      and so on...

However, in practice, if you use Illumina paired-end RNAseq protocols you are unlikely to uncover many of these possibilities. You will either deal with:

 * unstranded RNAseq data (**IU** type from above. Also called **fr-unstranded** in TopHat/Cufflinks jargon);
 * stranded RNAseq data produced with Illumina TrueSeq RNAseq kits and [dUTP tagging](https://nar.oxfordjournals.org/content/37/18/e123) (**ISR** type from above or **fr-firststrand** in TopHat/Cufflinks nomenclature).

The implication of stranded RNAseq is that you can distinguish whether the reads are derived from forward- or reverse-encoded transcripts:

![Stranded RNAseq data look like this](../../images/transcriptomics_images/stranded_result.png)

**Stranded RNAseq data look like this**<br>
This example contrasts unstranded and stranded RNAseq experiments. <font color="red">Red transcripts</font> are from + strand and <font color="blue">blue</font> are from - strand. In stranded example reads are clearly stratified between the two strands.  A small number of reads from opposite strand may represent anti-sense transcription. The image from GATC Biotech.

### Replicates: Biological or Technical and how many?

An RNAseq experiment without a sufficient number of replicates will be a waste of money. Replicates are essential to be able to correct for variation due to differences within/among organisms, cells, sequencing machines, library preparation protocols and numerous other potential factors. There are two types of replicates (as described by [Dündar:2015](http://chagall.med.cornell.edu/RNASEQcourse/Intro2RNAseq.pdf)):


 * **Technical replicates** can be defined as *different library preparations from the same RNA sample*. They should account for batch effects from the library preparation such as reverse transcription and PCR amplification. To avoid possible lane effects (e.g., differences in the sample loading, cluster amplification, and efficiency of the sequencing reaction), it is good practice to multiplex the same sample over different lanes of the same flowcell. In most cases, technical variability introduced by the sequencing protocol is quite low and well controlled.

* **Biological replicates**. There is an on-going debate over what kinds of samples represent true biological replicates. Obviously, the variability between different samples will be greater between RNA extracted from two unrelated humans than between RNA extracted from two different batches of the same cell line. In the latter case, most of the variation that will eventually be detected was probably introduced by the experimenter (e.g., slightly differing media and plating conditions). Nevertheless, this is variation the researcher is typically not interested in assessing, therefore the ENCODE consortium defines biological replicates as RNA from an independent growth of cells/tissue (ENCODE [2011](https://genome.ucsc.edu/ENCODE/protocols/dataStandards/ENCODE_RNAseq_Standards_V1.0.pdf)).

The number of replicates should be as high as practically possible. Most RNAseq experiments include three replicates and some have as many as 12 (see Schurch et al. [2015](https://arxiv.org/abs/1505.02017)).

## Read mapping

After sequencing is performed you have a collection of sequencing reads for each sample/replicate. In a reference-based RNAseq experiment these need to be mapped against the genome. Because in the case of eukaryotic transcriptome most reads originate from processed mRNAs lacking exons, they cannot be simply mapped back to the genome. Instead they can be separated into two categories:

 * Reads that map entirely within exons
 * Reads that cannot be mapped within an exon across their entire length because they span two or more exons

*Spliced* mappers have been developed to efficiently map transcript-derived reads against genome.

### TopHat, TopHat2, and HiSat

[Tophat](https://bioinformatics.oxfordjournals.org/content/25/9/1105.abstract) was one of the first tools designed specifically to address this problem by identifying potential exons using reads that do map to the genome, generating possible splices between neighboring exons, and comparing reads that did not initially map to the genome agaisnt these *in silico* created junctions:

[![TopHat and TopHat2: Mapping RNAseq regions to genome](../../images/transcriptomics_images/tophat.png)](https://bioinformatics.oxfordjournals.org/content/25/9/1105/F1.expansion.html)

**TopHat and TopHat2: Mapping RNAseq regions to genome**<br>
In TopHat reads are mapped against the genome and are separated into two categories: (1) those that map, and (2) those that initially unmapped (IUM). "Piles" of reads representing potential exons are extended in search of potential donor/acceptor splice sites and potential splice junctions are reconstructed. IUMs are then mapped to these junctions. Image from [Trapnell:2009](https://bioinformatics.oxfordjournals.org/content/25/9/1105.full).


[![TopHat has been subsequently improved with the development of TopHat2](../../images/transcriptomics_images/tophat2.png)](https://genomebiology.biomedcentral.com/articles/10.1186/gb-2013-14-4-r36)

**TopHat has been subsequently improved with the development of TopHat2**<br>
Image from [Kim:2012](https://genomebiology.biomedcentral.com/articles/10.1186/gb-2013-14-4-r36) summarizes steps involved in aligning of RNAseq reads with TopHat2

To further optimize and speed up spliced read alignment Kim at al. [2015](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4655817/) developed [HISAT](https://ccb.jhu.edu/software/hisat2/index.shtml). It uses a set of [FM-indices](https://en.wikipedia.org/wiki/FM-index) consisting one global genome-wide index and a collection of ~48,000 local overlapping 42 kb indices (~55,000 56 kb indices in HiSat2). This allows to find initial seed locations for potential read alignments in the genome using global index and to rapidly refine these alignments using a corresponding local index:


![Hierarchical Graph FM index in HiSat/HiSat2](../../images/transcriptomics_images/hisat.png)

**Hierarchical Graph FM index in HiSat/HiSat2**<br>
A part of the read (blue arrow) is first mapped to the genome using the global FM index. The HiSat then tries to extend the alignment directly utilizing the genome sequence (violet arrow). In (**a**) it succeeds and this read aligned as it completely resides within an exon. In (**b**) the extension hits a mismatch. Now HiSat takes advantage of the local FM index overlapping this location to find the appropriate matting for the remainder of this read (green arrow). The (**c**) shows a combination these two strategies: the beginning of the read is mapped using global FM index (blue arrow), extended until it reaches the end of the exon (violet arrow), mapped using local FM index (green arrow) and extended again (violet arrow). Image from [Kim:2015](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4655817/)

### STAR mapper

[STAR aligner](https://github.com/alexdobin/STAR) is a fast alternative for mapping RNAseq reads against genome utilizing uncompressed [suffix array](https://en.wikipedia.org/wiki/Suffix_array). It operates in [two stages](https://academic.oup.com/bioinformatics/article/29/1/15/272537/STAR-ultrafast-universal-RNA-seq-aligner). In the first stage it performs seed search:

![STAR's seed search](../../images/transcriptomics_images/star.png)

**STAR's seed search**<br>
Here a read is split between two consecutive exons. STAR starts to look for a *maximum mappable prefix* (MMP) from the beginning of the read until it can no longer match continuously. After this point it start to MMP for the unmatched portion of the read (**a**). In the case of mismatches (**b**) and unalignable regions (**c**) MMPs serve as anchors from which to extend alignments. Image from [Dobin:2013](https://bioinformatics.oxfordjournals.org/content/early/2012/10/25/bioinformatics.bts635.full.pdf+html).

At the second stage STAR stitches MMPs to generate read-level alignments that (contrary to MMPs) can contain mismatches and indels. A scoring scheme is used to evaluate and prioritize stitching combinations and to evaluate reads that map to multiple locations. STAR is extremely fast but requires a substantial amount of RAM to run efficiently.

## Transcript reconstruction

The previous step - mapping - assigns RNAseq reads to genomic locations and identifies splice junctions from reads that originate from different exons. At transcript reconstruction step this information is taken further in attempt to build transcript models. There is a number of tools for performing this task. A benchmarking paper by [Hayer:2015](https://bioinformatics.oxfordjournals.org/content/early/2015/09/03/bioinformatics.btv488.full.pdf+html) attempted to compare performance of existing approaches with one of the outcomes shown below:

[![Comparison of transcript reconsruction approaches](../../images/transcriptomics_images/rnaseq_comparison.png)](https://bioinformatics.oxfordjournals.org/content/early/2015/09/08/bioinformatics.btv488/transcriptomics_images/F5.large.jpg)

**Comparison of transcript reconsruction approaches**<br>
Here *recall* (the number of correctly constructed forms divided by the total number of real forms) versus *precision* (true positives divided by the sum of true positives and false positives) are plotted for seven transcript assemblers tested on two simulated datasets: *EnsemblPerfect* and *EnsemblRealistic*. The shaded region is indicating suboptimal performance (i.e., the white, unshaded region is "good"). The figure is from [Hayer:2015](https://bioinformatics.oxfordjournals.org/content/early/2015/09/03/bioinformatics.btv488.full.pdf+html).

Based on these results [Cufflinks](https://cole-trapnell-lab.github.io/cufflinks/) and [StringTie](https://ccb.jhu.edu/software/stringtie/) have satisfactory performence. The following discussion is based on inner workings of StringTie.

### Transcriptome assembly with StringTie

[StringTie](https://ccb.jhu.edu/software/stringtie/) assembles transcripts from spliced read alignemnts produced by tools such as STAR, TopHat, or HISAT and simultaneously estimates their abundances using counts of reads assigned to each transcript. The following images illustrates details of StringTie workflow:

![StringTie algorithm](../../images/transcriptomics_images/stringtie1.png)

**StringTie workflow**<br>
Image from [Pertea:2015](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4643835/)

In essence StringTie builds an alternative splice graph from overlapping reads in a given locus. In such a graph nodes correspond to exons (or, rather, contiguous regions of genome covered by reads; colored regions on the figure above), while edges are represented by reads connecting these exons. Next, it identifies a path within the splice graph that has the highest weight (largest number of reads on edges). Such path would correspond to an assembled transcript at this iteration of the algorithm. Because the edge weight is equal to the number of the reads StringTie estimates the coverage level for this transcript (see below) which can be used to estimate the transcript's abundance. Reads that are associated with the transcript that was just assembled are then removed and the graph is updated to perform the next iteration of the algorithm.

## Transcript quantification

Transcriptome quantification attempts to estimate expression levels of individuals transcripts. This is performed by assigning RNAseq reads to transcripts, counting, and normalization.

### Assigning reads to transcripts

To associate reads with transcripts they (the reads) need to be aligned to the transcriptome. Tools like Cufflinks and StringTie reconstruct transcripts from spliced read alignments generated by other programs (TopHat, HISAT, STAR), so they already have the information about which reads belong to each reconstructed transcript. Other tools such as [Sailfish](https://www.cs.cmu.edu/~ckingsf/software/sailfish/), [Kallisto](http://pachterlab.github.io/kallisto/), and [Salmon](http://combine-lab.github.io/salmon/) perform *lightweight* alignment of RNAseq reads against existing transcriptome sequences. The goal of lightweight alignment is to quickly distribute the reads across transcripts they likely originate from without worrying too much about producing high quality alignments. The upside of this is that the entire procedure can be performed very quickly. The downside is that these tools require high quality transcriptome as input, which is not a problem if you work with humans or mice, but is a problem if you are studying Hyacinth macaw or any other brilliantly colored creatures.

#### Lightweight alignment

[Sailfish](https://www.cs.cmu.edu/~ckingsf/software/sailfish/) has been initially designed to utilize [*k*-mer](https://en.wikipedia.org/wiki/K-mer) matching for finding association between reads and corresponding transcripts:

![Assigning reads to transcripts: Sailfish](../../images/transcriptomics_images/sailfish.png)

**Assigning reads to transcripts: Sailfish**<br>
Sailfish indexes input transcriptome for a fixed *k*-mer length and compares *k*-mers derived from RNAseq reads against this index. Image from [Patro:2014](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4077321/)

The current version of Sailfish uses [quasi-alignment](http://biorxiv.org/content/biorxiv/early/2015/10/22/029652.full.pdf) to extend exact matches found with *k*-mers:

![quasi-alignment](../../images/transcriptomics_images/quasi_aln.png)

**Quasi-alignment of reads in Sailfish**<br>
In Sailfish version [0.7.0](https://github.com/kingsfordgroup/sailfish/releases/tag/v0.7.0) and up transcriptome is concatenated into a single sequence using `$` separators from which a [suffix array](https://en.wikipedia.org/wiki/Suffix_array) and a [hash table](https://en.wikipedia.org/wiki/Hash_table) are constructed. A *k*-mer from an RNAseq read (green) is looked up in the hash table, which immediately gives its position in the suffix array allowing to extend the march as described in the legend and the [paper](http://biorxiv.org/content/biorxiv/early/2015/10/22/029652.full.pdf). Image from  [Srivastava:2015](http://biorxiv.org/content/biorxiv/early/2015/10/22/029652.full.pdf)

[Kallisto](https://pachterlab.github.io/kallisto/) also utilizes *k*-mer matching but uses a different data structure. It constructs a [De Bruijn graph](https://en.wikipedia.org/wiki/De_Bruijn_graph) from transcriptome input (pane **b** of the figure below). This graph is different from De Bruijn graphs used for genome assembly in that its nodes are *k*-mers and transcripts correspond to paths through the graph. To accommodate multiple transcripts that can lay along the same path (or sub-path) the paths are "colored" with each transcript given a distinct "color" (in genome assembly the graph is built from the reads and nodes usually correspond to overlaps between *k*-mers forming incoming and outgoing edges). Non-branching sections of the graph that have identical coloring are "glued" into contigs. Finally a [hash table](https://en.wikipedia.org/wiki/Hash_table) is built that stores the position of each transcriptome *k*-mer within the graph:

![Kallisto](../../images/transcriptomics_images/kallisto.png)

**Assigning reads to transcripts: Kallisto**<br>
Here a black read is being associated with a set consisting of red, blue, and green transcripts (**a**). First, a graph is built from transcriptome (**b**). Next, by finding common *k*-mers between the read and the graph the read is "threaded" along a path (**c** and **d**). The colors along that path would indicate which transcripts it is likely derived from. Specifically, this is done by taking intersection of "colors" (**c**). It this case the read is assigned to two transcripts: red and blue. Image from [Bray:2015](https://arxiv.org/pdf/1505.02710v2.pdf)

[Salmon](https://combine-lab.github.io/salmon/about/) does not use *k*-mer matching approach. Instead it creates [bwa](https://github.com/lh3/bwa)-like [FM-index](https://en.wikipedia.org/wiki/FM-index) and uses it to finds chains of *Maximal Exact Matches* (MEMs) and *Super Maximal Exact Matches* (SMEMs) between a read and the transcriptome.
[Patro:2015](http://biorxiv.org/content/biorxiv/early/2015/06/27/021592.full.pdf) define a MEM as "*a substring that is shared by the query (read) and reference (transcript) that cannot be extended in either direction without introducing a mismatch*". Similraly, a SMEM is defined as a "*MEM that is not contained within any other MEM on the query.*" One of the advantages of utilizing the FM-index is that a new index does not need to re-generated for a search with different set of parameters. In the case of Sailfish and Kallisto an index is dependent on *k*-mer length and has to be recomputed every time the *k* is changed. The overall schematics of Salmon operation is as follows:

![Salmon's method](../../images/transcriptomics_images/salmon.png)

**Assigning reads to transcripts: Salmon**<br>
Image from [Patro:2015](http://biorxiv.org/content/biorxiv/early/2015/06/27/021592.full.pdf)

### Estimating transcript levels

Once reads are apportioned across individual transcripts they can be quantified. There are several approaches for quantification.

#### Flow networks

StringTie, which performs assembly and quantification simultaneously converts splice graph into a flow network for which it solves [the maximum flow problem](https://en.wikipedia.org/wiki/Maximum_flow_problem). The maximum flow is such network represents the expression level for a given transcript:

![Maximum flow network](../../images/transcriptomics_images/stringtie2.png)

**StringTie flow network**<br>
Here each exon node from the splice graph is split into *in* and *out* nodes connected with an edge weighted by the number of reads corresponding to that exon. For example, the first exon is covered by seven reads and so the edge between 1-in and 1-out has a weight of 7. Expression level would correspond to the maximum flow through a path representing a given transcript. Image from [Pertea:2015](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4643835/)

#### Expectation Maximization

The Expectation/Maximization framework (EM) is utilized in a number of tools such as [eXpress](http://bio.math.berkeley.edu/eXpress/index.html) and more recently [Sailfish](http://www.cs.cmu.edu/~ckingsf/software/sailfish/), [Kallisto](http://pachterlab.github.io/kallisto/), and [Salmon](https://combine-lab.github.io/salmon/about/) (As an alternative strategy Salmon also utilizes [variational Bayesian method](http://arxiv.org/pdf/1308.5953v1.pdf). The principle of EM is nicely illustrated by [Lior Pachter](https://liorpachter.wordpress.com/) in his transcript quantification [review](http://arxiv.org/pdf/1104.3889v2.pdf). Suppose, as shown on the image below, there are three transcripts (green, red, and blue). There are five reads associated with these transcripts. One read (*d*) is unique to the red transcript, while others correspond to two (*b*, *c*, *e*) or three (*a*) transcripts. The EM is an iterative procedure. In the first round transcript abundances are initialized as equal (0.33 each as there are three transcripts) and during expectation reads are apportioned across transcripts based on these abundances. Next, during maximization step transcript abundances are re-calculated as follow. For red transcript we sum up fraction of each read as 0.33 + 0 + 0.5 + 1 + 0.5 for reads a, b, c, d, and e, respectively. We now divide this by the sum of read allocations for each transcript as 2.33 + 1.33 + 1.33 for red, green, and blue transcripts respectively. For all three transcript calculation will look like this:

<div>
	$$

\color{red}{red}   =  \frac{0.33 + 0.0 + 0.5 + 1.0 + 0.5}{2.33 + 1.33 + 1.33} = 0.47\\

\color{green}{green} =  \frac{0.33 + 0.5 + 0.0 + 0.0 + 0.5}{2.33 + 1.33 + 1.33} = 0.27\\

\color{blue}{blue}  =  \frac{0.33 + 0.5 + 0.5 + 0.0 + 0.0}{2.33 + 1.33 + 1.33} = 0.27

    $$
</div>

During next expectation stage read are re-apportioned across transcripts and the procedure is repeated until convergence:

![EM algorithm](../../images/transcriptomics_images/em.png)

**Expectation Maximization (EM)**<br>
Image from [Pacher:2011](https://arxiv.org/pdf/1104.3889v2.pdf)

### Understanding quantification metrics

As we've seen above quantification for a transcript is estimated using the number of associated reads. Yet the count is not a very good measure as it will be severely biased by multiple factors such as, for example, transcript length. Thus these counts need to be normalized. Normalization strategies can be roughly divided into two groups:

 * Normalization for comparison within a **single** sample;
 * Normalization for comparison among **multiple** samples/conditions.

In their [tutorial](http://chagall.med.cornell.edu/RNASEQcourse/) Dündar et al. have compiled a table summarizing various metrics. Below is description of normalization technique for within sample comparisons (between sample comparison can be found in the next section on differential expression analysis):

![RNAseq normalization metrics](../../images/transcriptomics_images/within_norm.png)

**RNAseq normalization metrics: Within sample comparisons**<br>
Table from Dündar et al. [2015](http://chagall.med.cornell.edu/RNASEQcourse/)

In addition, an ee this [xcellent overview of these metrics](https://haroldpimentel.wordpress.com/2014/05/08/what-the-fpkm-a-review-rna-seq-expression-units/).

#### <font color="red">&#9888; You should NEVER EVER use RPKM, FPKM, or TPM to compare expression levels <em>across</em> samples. These are RELATIVE measures! Consider yourself warned!</font>

## Finding expression differences

The goal of differential expression analysis (DE) is to find gene (DGE) or transcript (DTE) differences between conditions, developmental stages, treatments etc. In particular DE has two goals:

 * Estimate the *magnitude* of expression differences;
 * Estimate the *significance* of expression differences.

For this expression is estimated from read counts and attempts are made to correct for variability in measurements using replicates that are absolutely essential accurate results (see below). We begin our short discussion on DE by reproducing a figure from [Trapnell:2013](https://www.nature.com/nbt/journal/v31/n1/abs/nbt.2450.html) highlighting some of the challenges associated with judging expression differences from read counts:

![Differential expression: Read counts and Expression levels](../../images/transcriptomics_images/diff.png)

**Differential expression: Read counts and Expression levels**<br>
**Change in fragment count for a gene does not necessarily equal a change in expression**. (**a**) Simple read-counting schemes sum the fragments incident on a gene’s exons. The exon-union model counts reads falling on any of a gene’s exons, whereas the exon-intersection model counts only reads on constitutive exons. (**b**) Both of the exon-union and exon intersection counting schemes may incorrectly estimate a change in expression in genes with multiple isoforms. The true expression is estimated by the sum of the length-normalized isoform read counts. The discrepancy between a change in the union or intersection count and a change in gene expression is driven by a change in the abundance of the isoforms with respect to one another. In the top row, the gene generates the same number of reads in conditions A and B, but in condition B, all of the reads come from the shorter of the two isoforms, and thus the true expression for the gene is higher in condition B. The intersection count scheme underestimates the true change in gene expression, and the union scheme fails to detect the change entirely. In the middle row, the intersection count fails to detect a change driven by a shift in the dominant isoform for the gene. The union scheme detects a shift in the wrong direction. In the bottom row, the gene’s expression is constant, but the isoforms undergo a complete switch between conditions A and B. Both simplified counting schemes register a change in count that does not reflect a change in gene expression. Figure from [Trapnell:2013] (https://www.nature.com/nbt/journal/v31/n1/abs/nbt.2450.html)


The following discussion of DGE logic is reproduced from [Dündar:2015](http://chagall.med.cornell.edu/RNASEQcourse/).

To determine the genes whose read count differences between two conditions are greater than expected by chance, DGE tools must make assumptions about the distribution of read counts. The null hypothesis – that the mean read counts of the samples of condition **A** are equal to the mean read counts of the samples of condition **B** – is tested for each gene individually. One of the most popular choices to model the read counts is the [Poisson distribution](https://en.wikipedia.org/wiki/Poisson_distribution) because:

 * individual reads can be interpreted as binary data (Bernoulli trials): they either originate from gene *i* or not;
 * we are trying to model the discrete probability distribution of the number of successes (success = read is present in the sequenced library);
 * the pool of possible reads that could be present is large, while the proportion of reads belonging to
ene **i** is quite small.

The nice feature of a Poisson distribution is that variance = mean. Thus, if the RNA-seq experiment gives us a precise estimate of the mean read counts per condition, we implicitly know what kind of variance to expect for read counts that are not truly changing between two conditions. This, in turn, then allows us to identify those genes that show greater differences between the two conditions than expected by chance. While read counts of the same library preparation (*technical replicates*) can indeed be well approximated by the Poisson distribution, it has been shown that biological replicates have greater variance (noise) than expected. This [overdispersion](https://en.wikipedia.org/wiki/Overdispersion) can be captured with the [negative binomial distribution](https://en.wikipedia.org/wiki/Negative_binomial_distribution), which is a more general form of the Poisson distribution that allows the variance to exceed the mean. The square root of the dispersion is the coefficient of variation – SD/mean – after subtracting the variance we expect due to Poisson sampling.

In contrast to the Poisson distribution, we now need to estimate two parameters from the read counts: the mean as well as the dispersion. The precision of these estimates strongly depends on the number (and variation) of replicates – the more replicates, the better the grasp on the underlying mean expression values of unchanged genes and the variance that is due to biological variation rather than the experimental treatment. For most RNA-seq experiments, only two to three replicates are available, which is not enough for reliable mean and variance estimates. Some tools therefore compensate for the lack of replication by borrowing information across genes with similar expression values and shrink a given gene’s variance towards the regressed values. These fitted values of the mean and dispersion are then used instead of the raw estimates
o test for differential gene expression.

The best performing tools tend to be [edgeR](https://bioconductor.org/packages/release/bioc/html/edgeR.html), [DESeq/DESeq2](https://bioconductor.org/packages/release/bioc/html/DESeq2.html), and [limma-voom](https://www.bioconductor.org/packages/release/bioc/html/limma.html) (see Rapaport et al. ([2013](https://link.springer.com/article/10.1186/gb-2013-14-9-r95)); Soneson and Delorenzi ([2013](https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-14-91)); Schurch et al. ([2015](http://arxiv.org/abs/1505.02017)) for reviews of DGE tools). DESeq and limma-voom tend to be more conservative than edgeR (better control of false positives), but edgeR is recommended for experiments with fewer than 12 replicates (Schurch et al., [2015](http://arxiv.org/abs/1505.02017)).

## Let's try it

### The data

In this example we will use a downsampled version of simulated *Drosophila melanogaster* RNA-seq data used by Trapnell et al. [2012](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3334321/). These include two conditions (**C1** and **C2**), each containing three replicates (**R1**, **R2**, and **R3**) sequenced as a paired end library. Thus in total there are 12 fastq datasets.

Here is what to do to load the data:

#### Loading the data and create dataset collections

 Go to the data library (ask your instructor) and select all fastq files. Then Click `to History` button:

![Data libraries](../../images/transcriptomics_images/rnaseq_library.png)

The datasets will appear in your history:

![RNASeq data in history](../../images/transcriptomics_images/rnaseq_data_in_history.png)

Twelve datasets make a lot of clicking necessary. To avoid this annoyance we will combine them into two collections - **c1** and **c2** as shown in the video below. Also, see this [tutorial]({{site.baseurl}}/topics/galaxy-interface/tutorials/processing-many-samples-at-once/tutorial.html) for yet another explanation of dataset collections.
 <div class="embed-responsive embed-responsive-16by9"><iframe src="https://player.vimeo.com/video/163625221" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe></div>


### Mapping reads

We will map the reads with TopHat2. Select **TopHat** from **NGS: RNA Analysis** section of the tool menu (left pane of Galaxy's interface):

#### Mapping with TopHat2

In this case the input to TopHat is **not** individual datasets, but a collection instead. The [video](https://vimeo.com/163625221) above shows how to generate collection. Since we have created two collection as was described above, we will used them as inputs (note that **Is this single-end or paired-end data?** is set to `Paired-end (as collection)`). Make sure that the top part of TopHat interface looks like in the image below. Here the following parameters are set:

* **Mean Inner Distance between Mate Pairs** = `28` This is because paired reads are 100 bp and mean insert size is 228 bp so that 228 - (100 + 100) = 28
* **Use a built in reference genome or own from your history** = `Use a build-in genome` and `dm3` is selected. This is because the reads are from *D. melanogaster*.
* **TopHat settings to use** = `Full parameter list` This is done to be able to specify the strandedness of the library.
* **Library Type** = `FR First Strand`

![TopHat interface](../../images/transcriptomics_images/tophat_interface.png)

The same procedure is then repeated for collection **c2**. In the end it generates a lot of datasets in the history resulting in something resembling an image below. TopHat produces five types of output and because we started with dataset collections every one of the green boxes shown below is actually a collection of outputs for **c1** and **c2**, respectively.

![TopHat outputs for both collections](../../images/transcriptomics_images/tophat_output.png)

Let's now take a look at some of the alignments. We will use IGV for this purpose.

First, let's drill down to actual alignments produced by TopHat. For example, in figure shown above simply click on **TopHat on collection 14: accepted_hits** and you will see a list of datasets corresponding to alignments of reads derived from each conditions:

![Accepted hits on collection 14](../../images/transcriptomics_images/accepted_hits_1.png)

Now, click on **c2-r1x** and the following will appear:

![Accepted hits c2-r1x and D. melanogaster button](../../images/transcriptomics_images/accepted_hits_2.png)

Finally, use **D. melanogaster** link (highlighted above) and follow the on-screen instructions. By focusing IGV on genomic position `chrX:11,897,111-11,920,446` you will be able to see spliced alignments produced by TopHat:

![IGV TopHat](../../images/transcriptomics_images/igv_tophat.png)

and [sashimi plots](https://software.broadinstitute.org/software/igv/Sashimi) highlighting potential splice junctions:

![Sashimi plots](../../images/transcriptomics_images/sashimi.png)

### Performing differential expression analysis

Using mapped reads produced by TopHat we will perform analysis of differential gene expression using HTSeq/DESeq2 pipeline.

#### Gene-based read counting with HTseq-count

[`HTSeq-count`](https://www-huber.embl.de/users/anders/HTSeq/doc/count.html) is one of the most popular tools for gene quantification. `HTseq-count` gives you multiple choices on how to handle read mapping to multiple locations, reads overlapping introns, or reads that overlap more than one genomic feature:

[![HTseq-count read/feature overlap modes](../../images/transcriptomics_images/htseq_count.png)](https://www-huber.embl.de/users/anders/HTSeq/doc/count.html)

**`HTseq-count` read/feature overlap modes**<br>
The `htseq-count` script of the HTSeq suite offers three different modes to handle details of read–feature overlaps that are depicted here. The default of featureCounts is the behavior of the union option. Image is from [HTseq documentation](https://www-huber.embl.de/users/anders/HTSeq/doc/count.html); Caption by [Dündar:2015](http://chagall.med.cornell.edu/RNASEQcourse/)

Before we can use `HTseq-count` we need to download gene annotations for version `dm3` of the *Drosophila melanogaster* genome. We use version `dm3` because it is the same genome we have mapped reads against during the TopHat step.

#### Getting *Drosophila malanogaster* gene annotation from UCSC

Select **UCSC Main** from **Get Data** section of the menu. Within the UCSC Genome Browser interface set parameters as shown below. In particular make sure that **assembly** is set ti `dm3` and **output format** is set to `GTF`. Click **get output**.
[![UCSC table browser](../../images/transcriptomics_images/ucsc_dm3.png)](../../images/transcriptomics_images/ucsc_dm3.png)

This [GTF](https://www.ensembl.org/info/website/upload/gff.html) dataset will be used one of the input for HTseq-count.

`HTseq-count` takes two inputs: (1) mapped reads in BAM format and (2) a GTF dataset containing annotation of genes. Using these inputs it will compute the number of reads per gene.

#### Using HTseq-count in Galaxy

`htseq-count` needs strand information to proceed. The strand information is specified as `+`, `-`, or `.` (unknown) in a GTF dataset. `htseq-count` does not like `.` and will generate an error if such unstranded features appear in data. To prevent these errors from happening we will filter them out from GTF file using **Filter** tool from **Filter and Sort** section of tool menu. Here `c7 != "."` means that we need to filter all rows where strand column (column #7) contains a dot:

![Filter parameters](../../images/transcriptomics_images/filter_gtf.png)

Select **htseq-count** from **NGS: RNA analysis** section on the left side of the menu. Set parameters as shown below. The red arrow points that to enable `htseq-count` to see collections, you need to select the 'folder' button. In the case of this particular Galaxy history we will need to run `htseq-count` twice. Once on TopHat alignemnts for collection **c1** (dataset #37; shown below) and then on alignments for collection **c2** (dataset # 57).|
![htseq-count parameters](../../images/transcriptomics_images/htseq_count_interface.png)

This will generate read counts per gene.

#### Normalizing read counts and computing differential expression with `DESeq2`

`DESeq2` takes read counts produced by `HTseq-count` and apply size factor normalization. Specifically, `DESeq2` will:

 * For each gene, compute the geometric mean of read counts across all samples;
 * Every gene count in then divided by the geometric mean;
 * The median of these ratios is a sample's size factor used for normalization.

<font color="red">&#10148;</font> Note: For a comprehensive overview of differential gene expression with DESeq2 see [Love:2016](https://www.bioconductor.org/packages/3.3/bioc/vignettes/DESeq2/inst/doc/DESeq2.pdf).

#### DESeq2 in Galaxy

The `DESeq2` Galaxy's interface is shown below. `DESeq2` allows to incorporate multiple *factors* in the analysis. In our case we only have one factor, which we call **Conditions**. This is because we are trying to find genes that are differentially expressed between two conditions. The first condition will the first **factor level**, while condition 2 will be the second **factor level**. Here the input for this first factor level is set to a collection `84: htseq-count on collection 37` and the input for the second input is set to `92: htseq-count on collection 57`. Make sure that **Visualising the analysis results** is set to `Yes`:

![DESeq2 parameters](../../images/transcriptomics_images/deseq2_interface.png)

This will produce output as shown below. The columns are: (**1**) gene identifier, (**2**) mean normalised counts, averaged over all samples from both conditions, (**3**) logarithm (base 2) of the fold change, (**4**) the standard error estimate for the log2 fold change estimate, (**5**) [Wald test](https://en.wikipedia.org/wiki/Wald_test) statistic, (**6**) p value for the statistical significance of this change, and (**7**) *p*-value adjusted for multiple testing with the Benjamini-Hochberg procedure which controls false discovery rate ([FDR](https://en.wikipedia.org/wiki/False_discovery_rate)). There is only one gene with significant change in gene expression between conditions: `CG1803-RC` with *p*-value = 1.6x10<sup>-05</sup>

![DESeq2 output](../../images/transcriptomics_images/deseq2_output.png)

In addition to the list of genes DESeq2 outputs a graphical summary of the result. It includes a number of plots that should be used to evaluate the quality of the experiment. The histogram of *p*-values below shows that in our sample there is in fact just one instance of a significant *p*-value:

![p values histogram](../../images/transcriptomics_images/p_val_hist.png)

The [MA plot](https://en.wikipedia.org/wiki/MA_plot) below shows the relationship between the expression change (M) and average expression strength (A). Genes with adjusted *p*-value < 0.1 are in red (there is only one such gene in thi sample at the bottom of the graph):

![MA plot](../../images/transcriptomics_images/MA_plot.png)

The Principal Component Analysis ([PCA](https://en.wikipedia.org/wiki/Principal_component_analysis)) shows the separation between Condition 1 and 2. This type of plot is useful for visualizing the overall effect of experimental covariates and batch effects (each replicate is plotted as an individual data point):

![The Principal Component Analysis output](../../images/transcriptomics_images/pca.png)

A heatmap of sample-to-sample distance matrix gives us an overview over similarities and dissimilarities between samples:

![A heatmap of sample-to-sample distance matrix](../../images/transcriptomics_images/euc_dist.png)


topic: assembly
tutorial: assembly-decontamination
title: Decontamination of a genome assembly
difficulty: intermediate
time_estimation: 1H30M
tools_json: []
requirements_json: []
tags_json: ["biodiversity"]
content_hash: 36b766b7f53b5c5e91999d0ee0e3924d
last_modified: 2026-02-26T21:12:14.398239
zenodo_link: https://zenodo.org/records/13367433
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/assembly-decontamination/tutorial.html

content:
When sequencing a genome, it is common that contamination from a foreign organism get mixed with the genomic material of our species of interest. For instance, if you are processing a whole body sample of an insect then you will sequence not only the insect, but everything on and inside of it. When building a reference genome, it is important to separate these contaminants from the genome of our species. Foreign DNA sequences could cause false positive identification when running BLAST analyses, the misidentification of genes that don't actually belong to the species, or they can be incorporated as 'reference' sequence for that species in the public archives when those sequences did not actually belong to that species.


In this tutorial, we will learn how to separate these contaminants from a genome assembly of a vertebrate. It will also include the removal of mitogenome contigs so that the final genome assembly contains only nuclear DNA. (A separate tutorial exists for assembling the mitogenome.)



> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get data

For this tutorial, we are using the genome assembly of the zebra finch (*Taeniopygia guttata*) generated by the [Vertebrate Genome Project](https://vertebrategenomesproject.org/). We downsized the assembly to 50 of the largest scaffolds to keep the runtime reasonable, and added some contaminants for the sake of this tutorial.


> <comment-title>Run this analysis on a "real" assembly</comment-title>
>
> If you want to run this analysis on a real assembly generated by the [Vertebrate Genome Project](https://vertebrategenomesproject.org/), you can find the scaffolded assembly on [Genome Ark](https://www.genomeark.org/) as a remote repository and upload it to Galaxy (available on the three main Public Galaxy instances: .org, .eu, .org.au).
>
> {% snippet faqs/galaxy/datasets_import_from_remote_files.md location="Genome Ark -> species -> Taeniopygia_guttata -> bTaeGut2 -> assembly_vgp_hic_2.0 -> bTaeGut2.hic.hap1.s2.fasta" %}
>
{: .comment}


The analysis described in this tutorial will work with any assembly in FASTA format. If your initial dataset is compressed, use the tool `Convert compressed file to uncompressed.` first.


> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}):
>
>    ```
>    https://zenodo.org/records/13367433/files/Contaminated_assembly.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the dataset : `Contaminated Assembly`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Check that the datatype is `fasta`
>
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="fasta" %}
>
>
{: .hands_on}

# Genome Masking


Genome masking refers to hiding regions of low complexity in the genome, such as repeats. Masking a genome reduces the noise and improves the efficacy of the decontamination by emphasizing the regions that contribute the most to the classification ({% cite saini2016gene %}, {% cite maskedref-website %}).

The first step of our analysis is therefore to mask our genome assembly. There are two types of masking, depending on the requirements of the tools we are using:

- **Soft Masking** The masked regions are replaced by lower case letters. This is the prefered method of masking when the tools understand it, because it does not cause any loss of information.
- **Hard Masking** The masked regions are replaced by a wildcard letter: `N`s for nucleotide sequences, `X`s for proteins. Use this technique if you are unsure whether the tools you will be using knows to ignore lower case letters.


> <details-title> Genome masking for everyone </details-title>
>
> Repeat masking is important for a lot of analyses like sequence alignments, classification, gene annotation. Repetitive and low complexity regions cause problems for search and clustering algorithms based on patterns. For gene annotation, repeats can cause non-specific gene hits.
>
{: .details}

In this tutorial we will use `Kraken2` to classify our contaminant scaffolds, `Blastn` to identify the mitochondrial sequences, and `DustMasker` to mask our genome. `Dustmasker` generates soft masked sequences, while `Kraken2` and `Blast` needs hard masked sequences, so we will need to do some file manipulations along the way.

Before running the analysis, ensure the sequences in your assembly are uppercase. This will ensure the soft masking can be done properly. *If* your sequences are not uppercase, then run the optional step "Convert the assembly file to upper case". We will then convert the lower case letter to `N`s to hard mask the sequences for `Kraken2` and `Blast`.

> <details-title> Why do the sequences need to be in uppercase? </details-title>
>
> Because soft masking uses lowercase to denote masked base pairs, the input sequence should not have lowercase sequences because then you cannot tell afterwards whether a sequence is lowercase because it started that way or because it was soft masked.
>
{: .details}


> <details-title> Optional : Convert the assembly file to upper case </details-title>
>
> 1. {% tool [Text transformation](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sed_tool/1.1.1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `Contaminated assembly`
>    - *"SED Program"*: `s/^[^>].+/\U&/g` (Convert letters in the sequence to uppercase)
>
>
{: .details}


> <hands-on-title> Mask the Genome with Dustmasker </hands-on-title>
>
> 1. {% tool [NCBI BLAST+ dustmasker](toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_dustmasker_wrapper/2.14.1+galaxy1) %} with the following parameters:
>    - *"Subject database/sequences"*: `FASTA file from your history`
>        - {% icon param-file %} *"Nucleotide FASTA subject file to use instead of a database"*: `Contaminated assembly` (or the output of **Text transformation** {% icon tool %} if you converted your sequences to upper case)
>    - *"DUST level"*: `40`
>    - *"Output format"*: `FASTA`
>
> 2. Rename your file `Soft Masked assembly`
>
{: .hands_on}

Take a peek at your genome assembly. You can see that the beginning of the first scaffold has been converted to lower case, which masks repeats corresponding to the telomeres in the sequences.

> <hands-on-title> Soft Masking to Hard Masking </hands-on-title>
>
> 1. {% tool [Text transformation](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sed_tool/1.1.1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `Soft Masked assembly` (output of **NCBI BLAST+ dustmasker** {% icon tool %})
>    - *"SED Program"*: `/^>/!y/atcgn/NNNNN/`: In sequence lines (not starting with ">"), replace each instance of lower case bases (a t c or g), including undefined ones (n), to `N`s).
>
> 2. Rename your file `Hard Masked assembly`
>
>    {% snippet  faqs/galaxy/analysis_regular_expressions.md %}
> 
>    > <comment-title> Unsure about your regex? </comment-title>
>    >
>    > Ask ChatGPT about it. E.g. [What is the sed command to convert a text to upper case?](https://chatgpt.com/share/de569538-0e4d-41ac-8380-2d52af62b3d7) or test with a [regex tester](https://sed.js.org/) website (Warning, some of these testers do not understand the lowercase and uppercase transformations)
>    > 
>    {: .comment}
>
> 
{: .hands_on}

Take a look at your dataset and see that the lower case bases have been converted to `N`s.

# Identify Non-Target Contaminants

[Kraken2](https://ccb.jhu.edu/software/kraken2/) is a tool for taxonomic classification. It matches the kmers found in our assemblies to the ones stored in its databases. For the purpose of decontamination, we only need to identify possible contaminants, not the sequences belonging to our sampled organisms. Therefore, we only need to use a database containing likely contaminants such as bacteria, virus, and human (always wear gloves, folks!).
The best option at the time of this tutorial's writing is the PlusPF database which contains the RefSeq Standard (archaea, bacteria, viral, plasmid, human, UniVec_Core) plus protozoa and fungi data.



> <hands-on-title> Identify the taxonomy of our sequences </hands-on-title>
>
> 1. {% tool [Kraken2](toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.1.1+galaxy1) %} with the following parameters:
>    - *"Single or paired reads"*: `Single`
>        - {% icon param-file %} *"Input sequences"*: `Hard Masked assembly` (output of **Text transformation** {% icon tool %})
>    - *"Print scientific names instead of just taxids"*: `Yes`
>    - *"Confidence"*: `0.3`
>    - *"Split classified and unclassified outputs?"*: `Yes`
>    - *"Select a Kraken2 database"*: `PlusPF (2021-05-17)`
>
>
{: .hands_on}


Kraken provides us with three outputs:
- A table containing the classification information for each sequence
- A fasta file containing the classified sequences (our contaminants)
- A fasta file containing the unclassified sequences (our precious zebra finch)

We could decide to use the fasta file of unclassified sequences as our assembly, but remember that our genome is hard masked. That means that we lost information. Because we don't use it now doesn't mean we never will. So instead of using the masked genome, we will remove the classified sequences from our original assembly.


But first, we are naturally curious people, and we want to know what type of contaminants we had in our sample.


## Investigate our contaminants

Take a look at the classification output of Kraken2. You can see five columns containing:

1. `C` / `U`: a one letter code indicating that the sequence was either classified or unclassified.
2. The sequence ID.
3. The taxonomy ID Kraken2 used to label the sequence; this is 0 if the sequence is unclassified.
4. The length of the sequence in bp.
5. A space-delimited list indicating the LCA mapping of each k-mer in the sequence(s). For example, "562:13 561:4 A:31 0:1" would indicate that:
    - the first 13 k-mers mapped to taxonomy ID #562
    - the next 4 k-mers mapped to taxonomy ID #561
    - the next 31 k-mers contained an ambiguous nucleotide
    - the next k-mer was not in the database

To identify our contaminants, we only need the information from the first three columns, and only for the sequences that are classified. So we will start by extracting our three columns and then filtering out unclassified sequences.

> <hands-on-title> Extract classification information </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1,c2,c3`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: `Kraken2 on data X: Classification` (output of **Kraken2** {% icon tool %})
>
>
>    > <comment-title> Columns definitions </comment-title>
>    >
>    > The three columns that we extracted contain:
>    > 1. Whether the sequence is classified `C` or unclassified `U`
>    > 2. The sequence name
>    > 3. The sequence classification
>    >
>    {: .comment}
>
>    > <comment-title> What if my organism is in the database? </comment-title>
>    >
>    > The filtering of contaminants is based on their presence in the database. Any sequence classified by Kraken2 is considered as a contaminant. If you are working on an organism that is present in the database, this filtering will not work.
>    >
>    > *If you can't find a database that include contaminants but exclude your organism, you will need to process the outputs of Kraken differently.* Instead of filtering the sequences on classified/unclassified, select the sequences that do not match the taxonomic ID of your species. That is true for the rest of the section.
>    {: .comment}
>
{: .hands_on}


> <hands-on-title> Remove unclassified sequences </hands-on-title>
>
> 1. {% tool [Filter](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `Cut on data X` (output of **Cut** {% icon tool %})
>    - *"With following condition"*: `c1!='U'`
>
> 2. Rename the output `Contaminants informations`
>
{: .hands_on}


> <question-title></question-title>
>
>  How many different contaminants are in our sample?
>
> > <solution-title></solution-title>
> >
> >  We found two contaminants in our sample: Salmonella and Yersinia phage phiR1-37.
> >
> {: .solution}
>
{: .question}


## Get the list of contaminants to remove from our assembly

In order to remove the contaminant sequences from our assembly, we need a list of the contaminants names, that we will get by extracting the contaminant IDs from the Kraken output.


> <hands-on-title> Get contaminant sequence names </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c2`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: `Contaminants informations` (output of **Filter** {% icon tool %})
>
> 2. Rename the output `List of Contaminants`
>
{: .hands_on}


Now that we have our list of contaminants, we will go through a similar process to extract mitochondrial sequences.


# Identify mitochondrial DNA

We start with running Blastn against a database containing mitochondrial sequences.

> <hands-on-title> Blastn against RefSeq Mitochondrion </hands-on-title>
>
> 1. {% tool [NCBI BLAST+ blastn](toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastn_wrapper/2.14.1+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Nucleotide query sequence(s)"*: `Hard Masked assembly` (output of **Text transformation** {% icon tool %})
>    - *"Subject database/sequences"*: `Locally installed BLAST database`
>        - *"Nucleotide BLAST database"*: `RefSeq Mitochondrion`
>    - *"Type of BLAST"*: `blastn - Traditional BLASTN requiring an exact match of 11, for somewhat similar sequences`
>    - *"Output format"*: `Tabular (select which columns)`
>        - *"Standard columns"*: `qseqid, sseqid, length, qstart, qend, evalue`
>        - *"Extended columns"*: `qlen`
>        - *"Miscellaneous columns"*: `qcovs, qcovhsp`
>    - *"Advanced Options"*: `Hide Advanced Options`
>
>
{: .hands_on}

This step may take a little while, so you have time to make yourself a cup of coffee.

Next, we parse the result of the Blastn to extract a report on the alignements between our assemblies and the database, and a list of the scaffolds aligning with mitochondrial sequences.


> <hands-on-title> Extract alignement informations from the Blast output</hands-on-title>
>
> 1. {% tool [Parse mitochondrial blast](toolshed.g2.bx.psu.edu/repos/iuc/parse_mito_blast/parse_mito_blast/1.0.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Tabular file generated by mito-blast"*: `blastn Text transformation on data X vs 'refseq_mitochondrion'` (output of **NCBI BLAST+ blastn** {% icon tool %})
>
> 2. Rename the output `List of Mitochondrial sequences`
>
{: .hands_on}



> <question-title></question-title>
>
>  How many scaffolds belonging to the mitochondria are in our sample?
>
> > <solution-title></solution-title>
> >
> >  We found two mitochondrial scaffolds in our assembly.
> >
> {: .solution}
>
{: .question}



# Remove contaminants and mitochondrial DNA from the assembly

From the previous sections, we obtain two files containing a list of sequences belonging to contaminants and one belonging to the mitochondria. We will merge these two lists to clean up our original assembly.


> <hands-on-title> Concatenate lists of scaffolds to remove </hands-on-title>
>
> 1. {% tool [Concatenate datasets](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/0.1.1) %} with the following parameters:
>    - {% icon param-files %} *"Datasets to concatenate"*: `List of Contaminants`, `List of Mitochondrial sequences`
>
> 2. Rename the output `Sequences to remove`
>
{: .hands_on}

Now we will use gfastats, a tool box for genome assemblies with capability for manipulations and statistics generation.

> <hands-on-title> Remove scaffolds from the original genome assembly </hands-on-title>
>
> 1. {% tool [gfastats](toolshed.g2.bx.psu.edu/repos/bgruening/gfastats/gfastats/1.3.6+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: `output` (Input dataset)
>    - *"Specify target sequences"*: `Enabled`
>        - {% icon param-file %} *"Exclude specific intervals"*: `Sequences to remove` (output of **Concatenate datasets** {% icon tool %}). This input can be a bed file with sequence coordinates, or, in our case, a simple list of sequence IDs to remove from the fasta file. 
>    - *"Tool mode"*: `Genome assembly manipulation`
>        - *"Output format"*: `FASTA.gz`
>
>
{: .hands_on}

# Conclusion

In this tutorial we used taxonomic classification to identified contaminants and Blast to identify mitochondrial sequences in a genome assembly. We then removed these sequences to keep only the nuclear DNA of our species of interest.

Now that we have removed the contaminants and mitochondrial DNA from our assembly, it is ready for manual curation!


topic: assembly
tutorial: hybrid_denovo_assembly
title: Hybrid genome assembly - Nanopore and Illumina
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: ["assembly", "nanopore", "illumina", "denovo", "genomics"]
content_hash: f7fba721d7cbaa2c01ee001e5719f13c
last_modified: 2026-02-26T21:12:14.465534
zenodo_link: https://doi.org/10.5281/zenodo.15756327
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/hybrid_denovo_assembly/tutorial.html

content:
*Assemble a genome! Learn how to create and assess genome assemblies using the powerful combination of Nanopore and Illumina reads.*

This tutorial explores how long and short read data can be combined to produce a high-quality 'finished' bacterial genome sequence. Termed 'hybrid assembly', we will use read data produced from two different sequencing platforms, Illumina (short read) and Oxford Nanopore Technologies (long read), to reconstruct a bacterial genome sequence.

In this tutorial, we will perform 'de novo assembly'. De novo assembly is the process of assembling a genome from scratch using only the sequenced reads as input - no reference genome is used.  This approach is common practise when working with microorganisms, and has seen increasing use for eukaryotes (including humans) in recent times.  

Using short read data (Illumina) alone for de novo assembly will produce a complete genome, but in pieces (commonly called a 'draft genome'). For the genome to be assembled into a single chromosome (plus a sequence for each plasmid), reads would need to be longer than the longest repeated element on the genome (usually ~7,000 base pairs, Note: Illumina reads are 350 base maximum). Draft bacterial genome sequences are cheap to produce (less than AUD\$60) and useful (>300,000 draft *Salmonella enterica* genome sequences published at NCBI [https://www.ncbi.nlm.nih.gov/pathogens/organisms/](https://www.ncbi.nlm.nih.gov/pathogens/organisms/)), but sometimes you need a high-quality 'finished' bacterial genome sequence.  There are <1,000 'finished' or 'closed' *Salmonella enterica* genome sequences.

In these cases, long reads can be used together with short reads to produce a high-quality assembly. Nanopore long reads (commonly >40,000 bases) can fully span repeats, and reveal how all of the genome fragments should be arranged. Long reads currently have higher error rates than short reads, so the combination of technologies is particularly powerful. Long reads provide information on the genome structure, and short reads provide high base-level accuracy.  

Combining read data from the long and short read sequencing platforms allows the production of a complete genome sequence with very few sequence errors, but the cost of the read data is about AUD$1,000 to produce the sequence. Understandably, we usually produce a draft genome sequence with very few sequence errors using the Illumina sequencing platform.

Nanopore sequencing technology is rapidly improving, expect the cost difference to reduce!!

- **Data:** Nanopore reads, Illumina reads, bacterial organism (*Bacillus subtilis*) reference genome
- **Tools:** Flye, Pilon, Unicycler, Quast, Busco
- **Pipeline:** Hybrid de novo genome assembly - Nanopore draft Illumina polishing
- **Pipeline:** Hybrid de novo genome assembly - Unicycler

This Galaxy tutorial based on material from the [Hybrid genome assembly - Nanopore and Illumina](https://www.melbournebioinformatics.org.au/tutorials/tutorials/hybrid_assembly/nanopore_assembly/) tutorial and workshop created by [Melbourne Bioinformatics](https://mdhs.unimelb.edu.au/melbournebioinformatics) at the University of Melbourne.


> <agenda-title></agenda-title>
> 
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Background

## How do we produce the genomic DNA for a bacterial isolate?

Traditional *in vitro* culture techniques are important. Take a sample (e.g. a swab specimen from an infected sore) and streak a 'loopful' on to solid growth medium that suppoprts the growth of the bacteria. **Technology from the time of Louis Pasteur!**

Mixtures of bacterial types can be sequenced e.g. prepare genomic DNA from environmental samples containing bacteria - water, soil, faecal samples etc. (Whole Metagenome Sequencing)

![image of bacterial growth of a sample](../../images/denovo_assembly/sample_bacterial_growth.png)

One colony contains $$10^7 – 10^8$$ cells. The genomic DNA extracted from one colony is enough for Illumina sequencing. Larger amounts of genomic DNA are required for Nanopore sequencing.

## Shotgun sequencing - Illumina Sequencing Library

Genomic DNA is prepared for sequencing by fragmenting/shearing: multiple copies of Chromosome + plasmid  --> ~500 bp fragments

Note: Nanopore sequencing - there is usually no need to shear the genomic DNA as specialist methods are used to minimise shearing during DNA preparation. For Nanopore sequencing, the longer the DNA fragments, the better!

# Nanopore draft assembly, Illumina polishing

In this section, you will use `Flye` to create a draft genome assembly from Nanopore reads. We will perform assembly, then assess the quality of our assembly using two tools: `Quast` and `Busco`.

![workflow diagram for hybrid assembly starting with nanopore read](../../images/denovo_assembly/nanopore_illumina_hybrid_assembly.png)

## Upload data

Let's start with uploading the data.

> <hands-on-title>Import the data</hands-on-title>
> 
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import from [Zenodo](https://zenodo.org/uploads/15756328):
>   
>    - FASTQ file with illumina forward reads: `illumina_reads_1.fastq`
>    - FASTQ file with illumina reverse reads: `illumina_reads_2.fastq`
>    - FASTQ file with nanopore reads: `nanopore_reads.fastq`
>    - FASTA file with reference genome: `reference_genome.fasta`
> 
>    ```
>    https://zenodo.org/records/15756328/files/illumina_reads_1.fastq
>    https://zenodo.org/records/15756328/files/illumina_reads_2.fastq
>    https://zenodo.org/records/15756328/files/nanopore_reads.fastq
>    https://zenodo.org/records/15756328/files/reference_genome.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
> 
{: .hands_on}


## A baseline for "high-quality" assemblies

To begin, we will identify what a high-quality assembly looks like.

When running assembly tools, we want to check the quality of assemblies we produce.
It is paramount that genome assemblies are high-quality for them to be useful. To get a baseline for what is considered a "high-quality" assembly, we will first run a common assembly QC tool - `Busco` - on a published genome similar to the organism we are working with today.

In your imported files, you should see a `reference_genome.fasta` item.
This is the published genome we will compare against.

### QC with Busco

BUSCO analysis uses the presence, absence, or fragmentation of key genes in an assembly to determine its quality.

BUSCO genes are specifically selected for each taxonomic clade, and represent a group of genes that each organism in the clade is expected to possess. At higher clades, 'housekeeping genes' are the only members, while at more refined taxa such as order or family, lineage-specific genes can also be used.

We expect the reference genome to have all of these genes. When running `Busco`, we expect it to find most (if not all) of these in the assembly.

Find and select the `Busco` tool in the tools panel using the search bar.
In this tutorial, we know our organism is within the 'Bacillales' order.

> <hands-on-title>Run Busco</hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.8.0+galaxy1) %}:
>    - *"Sequences to analyse"*: `reference_genome.fasta`
>    - *"Mode"*:
>      - *"Select a gene predictor"*: `Augustus`
>    - *"Auto-detect or select lineage?"*: `Select lineage`
>      -  *"Lineage"*: `Bacillales`
>    - *"Which outputs should be generated"*: `Short summary text`
>   
> 2. ***Rename*** the `short summary` output to `Busco on Reference`.
> 
> 3. View the `Busco on Reference` output:
>    - It may look something like this:
>      ![output busco report for reference assembly](../../images/denovo_assembly/busco_reference_assembly.png)
> 
{: .hands_on}

> <question-title></question-title>
> 
> What can you conclude from the BUSCO report?
> 
> > <solution-title></solution-title>
> > 
> > - It seems that `Busco` could find almost all expected genes in the reference genome assembly.
> > - By looking at the results, we see that we have 449 / 450 Complete BUSCOs, and one Fragmented BUSCO.
> > - This will form the baseline for the `Busco` QC results expected of a high-quality genome assembly.
> > - From here, we will use our input DNA sequence data to assemble the genome of the sequenced organism, and will compare the QC results to that of the published `reference_genome.fasta` assembly.
> > 
> {: .solution}
> 
{: .question}


## Draft assembly with Flye + Nanopore reads

Our first assembly will use the long-read data to create a draft genome, then the short-read data to "polish" (improve) the draft into a better assembly.

We will start by using a long-read assembly tool called `Flye` to create an assembly using the Nanopore long-read data.

Search for `Flye` in the tools panel search bar and select the tool.

> <hands-on-title>Run Flye</hands-on-title>
>
> 1. {% tool [Flye](toolshed.g2.bx.psu.edu/repos/bgruening/flye/flye/2.9.6+galaxy0) %}:
>    - *"Input reads"*: `nanopore_reads.fastq`
>    - Leave all other parameters as default
>    - Scroll down and run Flye by clicking the blue `Run tool` button at the bottom of the page.
> 
> 2. View output:
>    - `Flye` produces a number of outputs. We only need the 'consensus' fasta file. You can delete the other outputs.
> 
> 3. ***Rename*** the `Flye on data XX: consensus` output to `Flye: Assembly`
> 
{: .hands_on}
 

## Assessing the Flye draft assembly quality

### Busco

We need to check if our assembly is good quality or not. It is paramount that genome assemblies are high-quality for them to be useful.

> <hands-on-title>Run Busco</hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.8.0+galaxy1) %}:
>    - *"Sequences to analyse"*: `Flye: Assembly`
>    - *"Mode"*:
>      - *"Select a gene predictor"*: `Augustus`
>    - *"Auto-detect or select lineage?"*: `Select lineage`
>      -  *"Lineage"*: `Bacillales`
>    - *"Which outputs should be generated"*: `Short summary text`
> 
> 2. ***Rename*** the `short summary` output to `Busco on Flye: Assembly`.
> 
> 3. View the `Busco on Flye: Assembly` output:
>    - It may look something like this:
>      ![output busco report for draft assembly](../../images/denovo_assembly/busco_draft_assembly.png)
>    - The `full table` is also useful. It gives a detailed list of the genes we are searching for, and information about whether they were missing, fragmented, or complete in our assembly.
>      ![output busco table for draft assembly](../../images/denovo_assembly/busco_table_draft_assembly.png)
> 
{: .hands_on}


> <question-title></question-title>
> 
> How does this compare with the reference genome BUSCO report?
> 
> > <solution-title></solution-title>
> > 
> > We see that many genes are fragmented or missing. Our draft genome assembly isn't as good as the reference genome ***yet***. This is because we have so far only used the Nanopore long-read sequences, which have higher base-level error rates than short reads. In the next steps we will use Illumina short reads to correct for errors in the assembled Nanopore reads.
> > 
> {: .solution}
> 
{: .question}


### Quast

Aside from `Busco`, we can use another method to perform assembly QC. `Quast` allows us to compare two assemblies to determine their similarity.

Although the organism we sequenced may be different, we can use `Quast` to compare our assembly with the provided reference genome to see how similar they are on the individual base level.

Search for the `Quast` tool in the tools panel.

> <hands-on-title>Run Quast</hands-on-title>
>
> 1. {% tool [Quast](toolshed.g2.bx.psu.edu/repos/iuc/quast/quast/5.3.0+galaxy0) %}:
>    - *"Assembly mode"*: `Individual assembly`
>    - *"Use customized names for the input files?"*: `No, use dataset names`
>    - *"Contigs/scaffolds file"*: `Flye: Assembly`
>    - *"Type of assembly?"*: `Genome`
>    - *"Use a reference genome?"*: `Yes`
>      - *"Reference genome"*: `reference_genome.fasta`
>    - *"Output files"*: Select `HTML reports`, `Tabular reports`
> 
> 2. ***Rename*** the `tabular report` output to `Quast on Flye: Assembly`.
> 
> 3. View output:
>    - `Quast` will produce a HTML report summarising it's results.
>    - Open the report. It may look something like this:
>      ![output quast report for reference assembly](../../images/denovo_assembly/quast_draft_assembly.png)
>    - Note the:
>      - Genome fraction (%)
>      - \# mismatches per 100 kbp
>      - \# indels per 100 kbp
>      - \# contigs information
> 
{: .hands_on}

> <question-title></question-title>
> 
> What can you conclude about our draft assembly?
> 
> > <solution-title></solution-title>
> > 
> > - From the output of `Quast`, our draft assembly seems to have good coverage and not too many contigs.
> > - Unfortunately, the mismatch / indel rate is quite high.
> > - Although we don't expect our organism to be ***identical*** to the supplied reference, we would expect fewer mismatches and indels, as the provided reference genome is ***very*** similar to organism which was sequenced.
> > 
> {: .solution}
> 
{: .question}


## Assembly Polishing with Pilon

We should be able improve our draft assembly with the Illumina reads available and correct some of these errors.

This process involves two steps. We will first align the Illumina reads to our draft assembly, then supply the mapping information to `Pilon`, which will use this alignment information to error-correct our assembly.

Illumina reads have much higher per-base accuracy than Nanopore reads. We will map the Illumina reads to our draft assembly using a short-read aligner called `BWA-MEM`. Then we can give `Pilon` this alignment file to polish our draft assembly.

### Map Illumina reads to draft assembly

Search for `Map with BWA-MEM` in the tools panel.

> <hands-on-title>Map with BWA-MEM</hands-on-title>
>
> 1. {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.19) %}:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>    - *"Use the following dataset as the reference sequence"*: `Flye: Assembly`
>    - *"Single or Paired-end reads"*: `Paired `
>    - *"Select first set of reads"*: `illumina_reads_1.fastq`
>    - *"Select second set of reads"*: `illumina_reads_2.fastq`
> 
> 2. View output:
>    - The output will be a BAM file (Binary Alignment Map). This is tabular data recording information about how reads were aligned to the draft assembly.
>    - We can now use this output BAM file as an input to `Pilon`.
> 
> 3. ***Rename*** the output to `Flye: Short read alignments`
> 
{: .hands_on}


### Polish assembly with Pilon

Search for `Pilon` in the tools panel.

> <hands-on-title>Run Pilon</hands-on-title>
>
> 1. {% tool [Pilon](toolshed.g2.bx.psu.edu/repos/iuc/pilon/pilon/1.20.1) %}:
>    - *"Source for reference genome used for BAM alignments"*: `Use a genome from History`
>      - *"Select a reference genome"*: `Flye: Assembly`
>    - *"Type automatically determined by pilon"*: 
>      - *"Input BAM file"*: `Flye: Short read alignments`
>    - *"Variant calling mode"*: `No`
> 
> 2. View output:
>    - `Pilon` gives a single output file - the polished assembly.
> 
> 3. ***Rename*** the output to `Flye: Polished assembly`
> 
{: .hands_on}



### Compare draft and polished assemblies

We are now interested to see how much `Pilon` improved our draft assembly.

> <hands-on-title>Run Quast</hands-on-title>
>
> Run `Quast` as before with the new `Flye: Polished assembly` data
> 
> 1. Select the history item for our initial `Quast` job, then click the rerun {% icon dataset-rerun %} button. This will load the settings used for the previous `Quast` job.
> 
> 2. Change the *"Contigs/scaffolds file"* input to `Flye: Polished assembly`.
> 
> 3. Click the `Run Tool` button to submit the job.
>
> 4. ***Rename*** the `tabular report` output to `Quast on Flye: Polished assembly`.
> 
> 5. After `Quast` has finished, open the HTML report. 
>    - Make note of `# mismatches per 100 kbp` and `# indels per 100 kbp`.
> 
{: .hands_on}

> <question-title></question-title>
> 
> Has our assembly improved?
>
> > <solution-title></solution-title>
> > 
> > Yes, we now have lower mismatches and indels per 100kbp.
> > 
> {: .solution}
> 
{: .question}

> <hands-on-title>Run Busco</hands-on-title>
>
> Run `Busco` as before with the new `Flye: Polished assembly` data
> 
> 1. Select the history item for our initial `Busco` job, then click the rerun {% icon dataset-rerun %} button. This will load the settings used for the previous `Busco` job.
> 
> 2. Change the *"Sequences to analyse"* input to `Flye: Polished assembly`.
> 
> 3. Click the `Run Tool` button to submit the job.
>
> 4. ***Rename*** the `short summary` output to `Busco on Flye: Polished assembly`.
> 
> 5. After `Busco` has finished, open the `Busco on Flye: Polished assembly` output.
> 
{: .hands_on}

> <question-title></question-title>
> 
> Have we identified more expected genes?
>
> > <solution-title></solution-title>
> > 
> > Yes, we now identified 423 complete BUSCO genes.
> > 
> {: .solution}
> 
{: .question}


## Section Summary

All going well, the polished assembly should be much higher quality than our draft.

The per-base accuracy of our assembly contigs should have markedly improved. This is reflected in the lower mismatches and indels per 100kbp reported by `Quast`, and the higher number of complete BUSCO genes. Our contiguity and coverage (as measured by the genome fraction (%) statistic reported by `Quast`) may not show the same level of improvement, as the polishing step is mainly aimed at improving per-base contig accuracy.

Our next step is to use a purpose-built hybrid de novo assembly tool, and compare its performance with our sequential draft + polishing approach.


## Section Questions

> <question-title></question-title>
> 
> Which read set - short or long - was used to create our draft?
>
> > <solution-title></solution-title>
> > 
> > Long reads (Nanopore) were used to create the draft. Long reads allow excellent re-creation of the proper structure of the genome, and adequately handle repeat regions. The drawback of long reads is a higher error rate of the technology compared to short reads. This results in more mismatches and indels.
> > 
> {: .solution}
> 
{: .question}

> <question-title></question-title>
> 
> How was the draft polished?
>
> > <solution-title></solution-title>
> > 
> > Illumina reads have higher per-base accuracy than Nanopore. Illumina reads were aligned to the draft assembly, then `Pilon` used this alignment information to improve locations with errors in the assembly.
> > 
> {: .solution}
> 
{: .question}

> <question-title></question-title>
> 
> How does `Quast` inform on assembly quality?
>
> > <solution-title></solution-title>
> > 
> > `Quast` shows summary information about the assembly contigs. If a reference genome is given, it informs the genome fraction (how much of the reference is covered by the assembly), if any genomic regions appear duplicated, and error information including the rate of mismatches and indels.
> > 
> {: .solution}
> 
{: .question}

> <question-title></question-title>
> 
> How does `Busco` inform on assembly quality?
>
> > <solution-title></solution-title>
> > 
> > `Busco` does not use a reference genome to compare. It attempts to locate key genes which should be present in the assembly, and reports whether it could/could not find those genes. If a key gene is found, it reports whether the gene was fragmented (errors) or complete.
> > 
> {: .solution}
> 
{: .question}



## Purpose-built hybrid assembly tool - Unicycler

In this section, we will use a purpose-built tool called `Unicycler` to perform hybrid assembly.

`Unicycler` uses our Nanopore and Illumina read sets together as input, and returns an assembly. Once we have created the assembly, we will assess its quality using `Quast` and `Busco` and compare with our previous polished assembly. We will also perform BUSCO analysis on the supplied reference genome itself, to record a baseline for our theoretical best `Busco` report.

![workflow diagram for hybrid assembly using unicycler](../../images/denovo_assembly/unicycler_hybrid_assembly.png)

### Hybrid de novo assembly with Unicycler

`Unicycler` performs assembly in the opposite manner to our previous approach. Illumina reads are used to create an assembly graph, then Nanopore reads are used to disentangle problems in the graph. The Nanopore reads serve to bridge Illumina contigs, and to reveal how the contigs are arranged sequentially in the genome.

### Run Unicycler

Find `Unicycler` in the tools panel. It is listed as `Create assemblies with Unicycler`

Run `Unicycler` using the Nanopore and Illumina read sets.

> <hands-on-title>Run Unicycler</hands-on-title>
>
> 1. {% tool [Unicycler](toolshed.g2.bx.psu.edu/repos/iuc/unicycler/unicycler/0.5.1+galaxy0) %}:
>    - *"Paired or Single end data?"*: `Paired `
>    - *"Select first set of reads"*: `illumina_reads_1.fastq`
>    - *"Select second set of reads"*: `illumina_reads_2.fastq`
>    - *"Select long reads. If there are no long reads, leave this empty"*: `nanopore_reads.fastq`
> 
> 2. View Output
>    - `Unicycler` will output three files 
>       - The assembly 
>       - An assembly graph 
>       - SPAdes graphs 
>    - We are interested in the `Final Assembly` output, which is the assembly as a `fasta` file. 
> 
> 3. ***Rename*** the `Final Assembly` output to `Unicycler: Assembly`.
> 
> > <comment-title></comment-title>
> >
> > If `nanopore_reads.fastq` does not appear in the dropdown list, its datatype needs to be changed. 
> >
> > - Click the pencil {% icon galaxy-pencil %} icon next to `nanopore_reads.fastq` in the history panel.
> > - Then select the `Datatypes` tab.
> > - For the parameter `New Type`: `fastqsanger`
> > - Leave all else default and execute the program.
> > 
> {: .comment}
> 
{: .hands_on}


### Comparing Unicycler assembly to Nanopore + Illumina polished assembly

`Busco` and `Quast` can be used again to assess our `Unicycler` assembly. As a purpose-built tool, it generally produces much better assemblies than our sequential approach. This is reflected as (`Quast`) a lower number of contigs, lower mismatches and indels per 100kb, and (`Busco`) greater number of BUSCO genes complete.

> <hands-on-title>Run Quast</hands-on-title>
>
> Run `Quast` as before with the new `Unicycler: Assembly` data
> 
> 1. Select the history item for our initial `Quast` job, then click the rerun {% icon dataset-rerun %} button. This will load the settings used for the previous `Quast` job.
> 
> 2. Change the *"Contigs/scaffolds file"* input to `Unicycler: Assembly`.
> 
> 3. Click the `Run Tool` button to submit the job.
>   
> 4. ***Rename*** the `tabular report` output to `Quast on Unicycler: Assembly`.
> 
> 5. At time of writing, these were the `Quast` results:
> 
> ![output quast report for unicycler assembly](../../images/denovo_assembly/quast_unicycler_assembly.png)
> 
{: .hands_on}

> <hands-on-title>Run Busco</hands-on-title>
>
> Run `Busco` as before with the new `Unicycler: Assembly` data
> 
> 1. Select the history item for our initial `Busco` job, then click the rerun {% icon dataset-rerun %} button. This will load the settings used for the previous `Busco` job.
> 
> 2. Change the *"Sequences to analyse"* input to `Unicycler: Assembly`.
> 
> 3. Click the `Run Tool` button to submit the job.
>
> 4. ***Rename*** the `short summary` output to `Busco on Unicycler: Assembly`.
> 
> 5. At time of writing, these were the `Busco` results:
> 
> ![output busco report for unicycler assembly](../../images/denovo_assembly/busco_unicycler_assembly.png)
> 
{: .hands_on}

We can now create an aggregate summary report from all of our `Quast` and `Busco` outputs using `MultiQC`, which enables us to easily compare the quality of our genome assemblies.

> <hands-on-title>Run MultiQC</hands-on-title>
>
> 1. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy4) %}:
>    - *"1: Results"*:
>      - *"Which tool was used generate logs?"*: `BUSCO`
>      - *"Output of BUSCO"*: `Busco on Reference`, `Busco on Fly: Assembly`, `Busco on Fly: Polished assembly`, `Busco on Unicycler: Assembly`, 
>    - Click `+ Insert Results` to add a second results field
>    - *"2: Results"*:
>      - *"Which tool was used generate logs?"*: `QUAST`
>      - *"Output of Quast"*: `Quast on Fly: Assembly`, `Quast on Fly: Polished assembly`, `Quast on Unicycler: Assembly`
>
> 2. View the `Webpage` output from `MultiQC`:
>    - At time of writing, these were the results.
>      - The number of BUSCOs identified in each assembly by `Busco`:
>        ![buscos identified in each assembly](../../images/denovo_assembly/busco_plot_bacillales_odb10.png)
>      - A table of `Quast` statistics for each assembly:
>        ![table of quast statistics](../../images/denovo_assembly/quast_table_statistics.png)
>      - The number of contigs identified in each assembly by `Quast`:
>        ![number of contigs in each assembly](../../images/denovo_assembly/quast_num_contigs.png) 
> 
{: .hands_on}


> <question-title></question-title>
> 
> How does the `Unicycler` assembly compare with the previous assembly produced using `Flye` + `Pilon`?
> 
> > <solution-title></solution-title>
> > 
> > It seems that the `Unicycler` assembly is much better than the `Flye` + `Pilon` polishing assembly, and produces the same report as the reference genome. Awesome! Looks like this is pretty good.
> > 
> > The `Unicycler` assembly has:
> > 
> > - A very high Genome fraction (coverage of the reference genome)
> > - Very few mismatches and indels per 100 kbp
> > - 15 contigs (compared with the reference genome only having 1 contig)
> > 
> {: .solution}
> 
{: .question}


These results indicate that our `Unicycler` assembly is high quality, and the organism is extremely similar to the organism of the reference genome.

To improve our assembly further to be publishable "complete" quality, we would want to reduce the number of contigs (ideally to a single contig). If the organism has plasmids, we would expect a handfull of contigs, likely below 10.

To do this, the best course of action would be to generate more long-read data.


### Section Questions

> <question-title></question-title>
> 
> Why did we select `Paired` for our Illumina reads in the `Unicycler` tool?
> 
> > <solution-title></solution-title>
> > 
> > Our short read set was `paired-end`. Short read technology can only sequence a few hundred base-pairs in a single read. To provide better structural information, paired-end sequencing was created, where longer fragments (fixed length) are used. A few hundred bp is sequenced at both ends of the fragment, leaving the middle section unsequenced. The reads produced (the mate-pair) from a single fragment are separated by a fixed length, so we know they are nearby in the genome.
> > 
> {: .solution}
> 
{: .question}

> <question-title></question-title>
> 
> Does `Unicycler` begin by using the long or short reads?
> 
> > <solution-title></solution-title>
> > 
> > Unicycler uses short reads first. It creates an assembly graph from short reads, then uses the long reads to provide better structural information of the genome.
> > 
> {: .solution}
> 
{: .question}

> <question-title></question-title>
> 
> How does `Unicycler` use long reads to improve its assembly graph?
> 
> > <solution-title></solution-title>
> > 
> > The assembly graph produced by short reads has tangled regions. When we don't know how sections of the genome are arranged, tangled regions appear in the graph. `Unicycler` uses Nanopore reads which overlap these tangled regions to resolve the proper structure of the genome.
> > 
> {: .solution}
> 
{: .question}


# Conclusion

We have covered two methods for hybrid de novo assembly. The combination of long- and short-read technology is clearly powerful, represented by our ability to create a good assembly with only 25x coverage (100Mb) of Nanopore and 50x coverage (200Mb) of Illumina reads.

To further improve our assembly, extra Nanopore read data may provide the most benefit. At 50x coverage (200Mb), we may achieve a single, or few contig assembly with high per-base accuracy.

In this tutorial, we had a reference genome for a closely related organism, which we could use when assessing the quality of our draft genome assemblies. Without access to a reference genome, `Quast` will produce fewer metrics for assessing the assembled genome quality. In this situation, you can look at the number of contigs in the assembled genome, the minimum number of contigs that produce 50\% and 90\% of the bases in the assembly and the GC\% content. For good quality assemblies, the total and minimum number (50\% and 90\%) of contigs should be small and the GC\% content should be near 50\%.

The development of new purpose-built tools for hybrid de novo assembly like `Unicycler` have improved the quality of assemblies we can produce. These tools are of great importance and while they already produce great results, they will continue to improve over time.

-------------------------------
## Additional reading
Links to additional recommended reading and suggestions for related tutorials.

- Flye: [https://github.com/fenderglass/Flye/blob/flye/docs/USAGE.md#algorithm](https://github.com/fenderglass/Flye/blob/flye/docs/USAGE.md#algorithm)
- Pilon: [https://github.com/broadinstitute/pilon/wiki/Methods-of-Operation](https://github.com/broadinstitute/pilon/wiki/Methods-of-Operation)
- Unicycler: [https://github.com/rrwick/Unicycler](https://github.com/rrwick/Unicycler)
- Quast: [https://academic.oup.com/bioinformatics/article/29/8/1072/228832](https://academic.oup.com/bioinformatics/article/29/8/1072/228832)
- BUSCO analysis: [https://academic.oup.com/bioinformatics/article/31/19/3210/211866](https://academic.oup.com/bioinformatics/article/31/19/3210/211866)


topic: assembly
tutorial: mrsa-nanopore
title: Genome Assembly of MRSA from Oxford Nanopore MinION data (and optionally Illumina data)
difficulty: introductory
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: ["nanopore", "assembly", "amr", "microgalaxy"]
content_hash: 1dbab14f5188d339ab6e979fc19960ed
last_modified: 2026-02-26T21:12:14.492247
zenodo_link: https://zenodo.org/record/10669812
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/mrsa-nanopore/tutorial.html

content:
Sequencing (determining of DNA/RNA nucleotide sequence) is used all over
the world for all kinds of analysis. The product of these sequencers are
reads, which are sequences of detected nucleotides. Depending on the
technique these have specific lengths (30-500bp) or using Oxford
Nanopore Technologies sequencing have much longer variable lengths.

{% snippet faqs/galaxy/sequencing_nanopore.md %}

In this training we will build an assembly of a bacterial genome, from data produced in
"Complete Genome Sequences of Eight Methicillin-Resistant
*Staphylococcus aureus* Strains Isolated from Patients in
Japan" {% cite Hikichi_2019 %}:

> Methicillin-resistant *Staphylococcus aureus* (MRSA) is a major pathogen
> causing nosocomial infections, and the clinical manifestations of MRSA
> range from asymptomatic colonization of the nasal mucosa to soft tissue
> infection to fulminant invasive disease. Here, we report the complete
> genome sequences of eight MRSA strains isolated from patients in Japan.
{: .quote cite="{% cite_url Hikichi_2019 %}"}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Galaxy and data preparation

Any analysis should get its own Galaxy history. So let's start by creating a new one and get the data into it.

> <hands-on-title>History creation</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

Now, we need to import the data: 1 FASTQ file containing the reads from the sequencer.

> <hands-on-title>Data upload</hands-on-title>
> 1. {% tool [Import](upload1) %} the files from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>
>    ```
>    {{ page.zenodo_link }}/files/DRR187567.fastq.bz2
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Rename the dataset to keep only the sequence run ID (`DRR187567`)
>
>    {% snippet faqs/galaxy/datasets_rename.md name="DRR187567" %}
>
> 3. Tag the dataset `#unfiltered`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
> 4. **View** {% icon galaxy-eye %} the renamed file
>
{: .hands_on}

The dataset is a FASTQ file.

> <question-title></question-title>
>
> 1. What are the 4 main features of each read in a fastq file.
> 2. What is the name of your first read?
>
> > <solution-title></solution-title>
> > 1. The following:
> >
> >    -   A `@` followed by a name and sometimes information of the read
> >    -   A nucleotide sequence
> >    -   A `+` (optional followed by the name)
> >    -   The quality score per base of nucleotide sequence (Each symbol
> >        represents a quality score, which will be explained later)
> >
> > 2. `DRR187567.1`
> {: .solution}
{: .question}

{% include _includes/cyoa-choices.html option1="Without Illumina MiSeq data" option2="With Illumina MiSeq data" default="Without Illumina MiSeq data" text="Do you have associated Illumina MiSeq data?" disambiguation="miseq"%}

<span class="Without-Illumina-MiSeq-data"></span>

<div class="With-Illumina-MiSeq-data" markdown="1">

> <hands-on-title>Illumina Data upload</hands-on-title>
> 1. {% tool [Import](upload1) %} the files from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>
>    ```
>    {{ page.zenodo_link }}/files/DRR187559_1.fastqsanger.bz2
>    {{ page.zenodo_link }}/files/DRR187559_2.fastqsanger.bz2
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Rename the datasets to remove `.fastqsanger.bz2` and keep only the sequence run ID (`DRR187559_1` and `DRR187559_2`)
>
>    {% snippet faqs/galaxy/datasets_rename.md name="DRR187567" %}
>
> 3. Create a paired collection named `Paired Reads`
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
{: .hands_on}

</div>

# Quality Control

During sequencing, errors are introduced, such as incorrect nucleotides being called. These are due to the technical limitations of each sequencing platform. Sequencing errors might bias the analysis and can lead to a misinterpretation of the data. **Sequence quality control is therefore an essential first step in any analysis.**

When assessing the fastq files all bases had their own quality (or Phred score) represented by symbols. You can read more in our dedicated [Quality Control Tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).

To assess the quality by hand would be too much work. That's why tools like [NanoPlot](https://github.com/wdecoster/NanoPlot) or
[FastQC](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/) are made, which will generate a summary and plots of the data statistics. NanoPlot is mainly used for long-read data, like ONT and PACBIO and FastQC for any read.

> <hands-on-title>Quality Control</hands-on-title>
>
> 1. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy0) %} with the following parameters:
>    - {% icon param-files %} *"Short read data from your current history"*: `DRR187567`
>
> 2. Inspect the webpage output
>
{: .hands_on}

FastQC combines quality statistics from all separate reads and combines them in plots. An important plot is the Per base sequence quality. 

![FastQC plot showing reads that mostly stay in the read](./images/fastqc.png)

<div class="With-Illumina-MiSeq-data" markdown="1">

Here, we are going to trim the Illumina data using **fastp** ({% cite Chen2018 %}):

- Trim the start and end of the reads if those fall below a quality score of 20
- Filter for reads to keep only reads with at least 30 bases: Anything shorter will complicate the assembly

> <hands-on-title>Quality improvement of the Illumina reads</hands-on-title>
>
> 1. {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/1.0.1+galaxy3) %} with the following parameters:
>    - *"Single-end or paired reads"*: `Paired Collection`
>        - *"Select paired collection(s)"*: `Paired Reads`
>    - In *"Filter Options"*:
>        - In *"Length filtering Options"*:
>            - *Length required*: `30`
>    - In *"Read Modification Options"*:
>        - In *"Per read cuitting by quality options"*:
>            - *Cut by quality in front (5')*: `Yes`
>            - *Cut by quality in tail (3')*: `Yes`
>            - *Cutting window size*: `4`
>            - *Cutting mean quality*: `20`
>    - In *"Output Options"*:
>        - *"Output JSON report"*: `Yes`
>     
> We next need to extract the **fastp** filtered forward and reverse reads from the paired collection as the next tool we are going to run, **filtlong**, requires the forward and reverse short reads to be provided as separate datasets.
>
> 2. {% tool [Flatten collection](__FLATTEN__) %} with the following parameters:
>    - *"Input Collection"*: **fastp** `Paired-end output`
>      
> 3. **Rename** {% icon galaxy-pencil %} the output to `Flattened fastp output`
>      
> 4. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - *"Input List"*: `Flattened fastp output`
>    - *"How should a dataset be selected?"*: `Select by element identifier`
>    - *"Element identifier"*: `DRR187559_forward` (filtered forward reads)
> 
> 5. **Rename** {% icon galaxy-pencil %} `DRR187559_forward` to `Trimmed DRR187559_forward`
>   
> 6. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - *"Input List"*: `Flattened fastp output`
>    - *"How should a dataset be selected?"*: `Select by element identifier`
>    - *"Element identifier"*: `DRR187559_reverse` (filtered forward reads)
>   
> 7. **Rename** {% icon galaxy-pencil %} `DRR187559_reverse` to `Trimmed DRR187559_reverse`
> 
{: .hands_on}

</div>

Depending on the analysis it could be possible that a certain quality or length is needed. The reads can be filtered using the [Filtlong](https://github.com/rrwick/Filtlong) tool. In this training all reads below 1000bp will be filtered.

<div class="With-Illumina-MiSeq-data" markdown="1">

When Illumina reads are available, we can use them **if they are good Illumina reads (high depth and complete coverage)**  as external reference. In this case, Filtlong ignores the Phred quality scores and instead judges read quality using k-mer matches to the reference (a more accurate gauge of quality).

</div>

> <hands-on-title>Filtering</hands-on-title>
> 1. {% tool [filtlong](toolshed.g2.bx.psu.edu/repos/iuc/filtlong/filtlong/0.2.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input FASTQ"*: `DRR187567`
>    - In *"Output thresholds"*:
>        - *"Min. length"*: `1000`
>
>    <div class="With-Illumina-MiSeq-data" markdown="1">
>    - In *"External references"*:
>        - {% icon param-file %} *"Reference Illumina read"*: `Trimmed DRR187559_forward`
>        - {% icon param-file %} *"Reference Illumina read"*: `Trimmed DRR187559_reverse`
>    </div>
>
> 2. Rename the dataset to `DRR187567-filtered`
>
>    {% snippet faqs/galaxy/datasets_rename.md name="DRR187567-filtered" %}
>
{: .hands_on}

The output can be evaluated using **NanoPlot** for plotting long read sequencing data and alignments

> <hands-on-title>Filtering</hands-on-title>
> 1. Convert the datatype of `DRR187567` to uncompress it
>
>    {% snippet faqs/galaxy/datasets_convert_datatype.md conversion="Convert compressed to uncompressed" %}
>
> 2. {% tool [NanoPlot](toolshed.g2.bx.psu.edu/repos/iuc/nanoplot/nanoplot/1.41.0+galaxy0) %} with the following parameters:
>    - *"Select multifile mode"*: `batch`
>        - *"Type of the file(s) to work on"*: `fastq`
>            - {% icon param-files %} *"files"*: **both** `DRR187567 uncompressed` and `DRR187567-filtered`
>    - In *"Options for customizing the plots created"*:
>        - *"Show the N50 mark in the read length histogram."*: `Yes`
>
{: .hands_on}

We ran the **NanoPlot** two times: one for the raw reads (`DRR187567`) and one for the reads after filtering (`DRR187567-filtered`). 

For each run, **NanoPlot** generates 5 outputs:

- 2 plots: 
   - Histogram Read Length
   - Histogram Read Length after log transformation

      ![before filtering, showing a large mostly gaussian curve](./images/nanoplot_before.png "Before filtering")
      
      ![after filtering, same curve as before but everything below 1000bp is gone](./images/nanoplot_after.png "After filtering")


- 2 tabular files with statistics: one general and one after filtering.

   The second one is empty because we did not used **NanoPlot** filtering options.

- A HTML report summarizing above information

   We can compare the two generated reports. Galaxy allows to view several datasets side-by-side using the Window Manager function

   > <hands-on-title>Inspect NanoPlot reports</hands-on-title>
   >
   > 1. Enable Window Manager
   >
   >    {% snippet faqs/galaxy/features_scratchbook.md %}
   >
   > 2. Open both NanoPlot HTML Reports
   > 3. Check the **Summary statistics** section of each to compare the results
   >
   {: .hands_on}

   Summary statistics  | Not Filtered   | Filtered (Filtlong) | Change (%)
   ------------------- | -------------- | ------------------- | ----------
   Number of reads     | 91,288         | 69,906              | -23.4%
   Number of bases     | 621,945,741.0  | 609,657,642.0       | -2.0%
   Median read length  | 3,400.0        | 5,451.0             | 60.3%
   Mean read length    | 6,813.0        | 8,721.1             | 28.0%
   Read length N50     | 14,810.0       | 15,102.0            | 2.0%
   Mean read quality   | 9              | 9                   | 0.0%
   Median read quality | 8.9            | 9.0                 | 1.1%

   > <question-title></question-title>
   >
   > 1. What is the increase of your median read length?
   > 2. What is the decrease in the number of bases?
   > 3. What is coverage?
   > 4. What would be the coverage before and after trimming, based on a genome size of 2.9 Mbp?
   >
   > > <solution-title></solution-title>
   > > 1. 3,400 bp to 5,451 bp, a 60.3% increase
   > > 2. A -2.0% decrease is not a very significant decrease. Our data was quite good to start with and didn't have many short reads which were removed (23.4%)
   > > 3. The coverage is a measure of how many reads 'cover' on average a single base in the genome. If you divide the total reads by the genome size, you will get a number how many times your genomes could theoretically be "covered" by reads.
   > > 4. Before $$ \frac{621,945,741}{2,900,000} = 214.4$$ and after $$ \frac{609,657,642}{2,900,000} = 210.2$$. This is *not* a very big decrease in coverage, so no cause for concern. Generally in sequencing experiments you have some minimum coverage you expect to see based on how much of your sample you sequenced. If it falls below that threshold it might be cause for concern.
   > >
   > {: .solution}
   {: .question}

While there is currently no community consensus over the best trimming or filtering practices with long read data, there are still some steps that can be beneficial to do for the assembly process. {% tool [Porechop](toolshed.g2.bx.psu.edu/repos/iuc/porechop/porechop/0.2.3) %} is a commonly used tool for removing adapter sequences, and we used {% tool [filtlong](toolshed.g2.bx.psu.edu/repos/iuc/filtlong/filtlong/0.2.0) %} for removing shorter reads which might make the assembly process more difficult.

Many people do not do any trimming of their NanoPore data based on the quality as it is expected that the quality is low, and often the focus is on assembling large {SVs} rather than having high quality reads and base-level variation analyses.

# Assembly

When the quality of the reads is determined and the data is filtered (like we
did with filtlong) and/or trimmed (like is more often done with short read data)
an assembly can be made.

There are many tools that create assembly for long-read data, e.g. Canu ({% cite koren2017canu %}), Raven ({% cite vaser2021time %}), Miniasm  ({% cite Li_2016 %}). In this tutorial, we use [**Flye**](https://github.com/fenderglass/Flye) ({% cite lin2016assembly %}). **Flye** is a de
novo assembler for single molecule sequencing reads. It can be used from bacteria to human assemblies. The Flye assembly is based on finding overlapping reads with variable length with high error tolerance. 

{% snippet faqs/galaxy/analysis_results_may_vary.md %}

> <hands-on-title>Assembly using Flye</hands-on-title>
>
> 1. {% tool [Flye assembly](toolshed.g2.bx.psu.edu/repos/bgruening/flye/flye/2.9.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input reads"*: `DRR187567-filtered` (output of **filtlong** {% icon tool %})
>    - *"Mode"*: `Nanopore corrected (--nano-corr)`
>    - *"Reduced contig assembly coverage"*: `Disable reduced coverage for initial disjointing assembly`
>
{: .hands_on}

**Flye** generates 4 outputs:

- A FASTA file called `consensus` with contigs, i.e. the contiguous sequences made by combining separate reads in the assembly, and possibly scaffolds built by **Flye**

   > <question-title></question-title>
   > 
   > How many contigs are there?
   > > <solution-title></solution-title>
   > >
   > > There are 2 sequences in the `consensus` dataset
   > {: .solution}
   {: .question}

- 2 assembly graph files: `assembly graph` and `graphical fragment assembly`

   We can visualize the assembly graph using the `graphical fragment assembly` with **Bandage** ({% cite wick2015bandage %}), a package for exploring assembly graphs through summary reports and visualizations of their contents.

   > <hands-on-title>Assembly inspection</hands-on-title>
   > 1. {% tool [Bandage Image](toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_image/2022.09+galaxy4) %} with the following parameters:
   >    - {% icon param-file %} *"Graphical Fragment Assembly"*: `graphical fragment assembly` (output of **Flye**)
   >
   {: .hands_on}

   ![Bandage output showing 2 contigs](./images/bandage_before_polishing.jpg)

- A table `assembly info` with assembly information

   It should be something similar but probably sligtly different because Flye can differ a bit per assembly:

   #seq_name | length  | cov. | circ. | repeat | mult. | alt_group | graph_path
   --------- | ------  | ---- | ----- | ------ | ----- | --------- | ----------
   contig_1  | 2927029 | 183  | Y     | N      | 2     | *         | 1
   contig_2  | 60115   | 379  | Y     | Y      | 2     | *         | 2

   > <question-title></question-title>
   > 1. What is the coverage of your longest contig?
   > 2. What is the length of your longest contig?
   > 3. Could this contig potentially be a MRSA genome?
   >
   > > <solution-title></solution-title>
   > > While results may vary due to randomness in the assembly process, in our case we had:
   > > 1. 183
   > > 2. 2.9Mb
   > > 3. 2.9Mb is approximately the size of a MRSA genome. So contig 1 could be the genome. Contig 2 could be one small potential plasmid genome.
   > {: .solution}
   {: .question}

## Assembly Evaluation

To evaluate the assembly, we use also [Quast](http://quast.bioinf.spbau.ru/) ({% cite Gurevich2013 %}) (QUality ASsessment Tool), a tool providing quality metrics for assemblies. This tool can be used to compare multiple assemblies, can take an optional reference file as input to provide complementary metrics, etc

> <hands-on-title>Quality Control of assembly using Quast</hands-on-title>
>
> 1. {% tool [Quast](toolshed.g2.bx.psu.edu/repos/iuc/quast/quast/5.2.0+galaxy1) %} with the following parameters:
>    - *Assembly mode?*: `Co-assembly`
>       - *"Use customized names for the input files?"*: `No, use dataset names`
>         - {% icon param-file %} *"Contigs/scaffolds file"*: `consensus` output of **Flye**
>
{: .hands_on}

QUAST outputs assembly metrics as an HTML file with metrics and graphs. 

> <question-title></question-title>
>
> 1. How many contigs is there?
> 2. How long is the largest contig?
> 3. What is the total length of all contigs?
> 4. What is you GC content?
> 5. How does it compare to results for KUN1163 in [Table 1](https://journals.asm.org/doi/10.1128/mra.01212-19#tab1) in {% cite Hikichi_2019 %}?
>
> > <solution-title></solution-title>
> >
> > 1. 2 contigs
> > 2. The largest contig is 2,907,099 bp.
> > 3. 2,967,214 (*Total length (>= 0 bp)*). 
> > 4. The GC content for our assembly is 32.91%. 
> > 5. KUN1163 has a genome size of 2,914,567 bp (not far from the length of the largest contig), with a GC content of 32.91% (as in our assembly)
> >
> {: .solution}
{: .question}

<div class="With-Illumina-MiSeq-data" markdown="1">

## Assembly Polishing

We can now polish the assembly using both the short reads and/or long reads. This process aligns the reads to the assembly contigs, and makes corrections to the contigs where warranted. 

Several tools exist for polishing, e.g. Racon ({% cite Vaser_2017 %}). Here we will use **Polypolish** ({% cite Wick_2022 %}), a tool for polishing genome assemblies with short reads.

**Polypolish** needs as input the assembly but also SAM/BAM files where each read has been aligned to *all* possible locations (not just a single best location). Errors in repeats will be covered by short-read alignments, and **Polypolish** can therefore fix those errors. 

To get the SAM/BAM files, we need to map the short reads on the assembly. We will use **BWA-MEM** ({% cite Li_2009 %}, {% cite Li_2010 %}, {% cite Li2013Aligning %}).

We need to set up **BWA-MEM** so it aligns each read to all possible locations, not just the best location. This option does not work with paired-end alignment. We will then need to align forward and reverse read files separately, instead of aligning both read files with a single BWA-MEM run as usually recommended.

> <hands-on-title> Align short-reads on assembly </hands-on-title>
> 1. Change the datatype of both FASTQ outputs of **fastp** to `fastqsanger.gz`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="fastqsanger.gz" %}
>
> 2. {% tool [BWA-MEM2](toolshed.g2.bx.psu.edu/repos/iuc/bwa_mem2/bwa_mem2/2.2.1+galaxy1) %} with the following parameters:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>        - {% icon param-file %} *"Use the following dataset as the reference sequence"*: `consensus` output of **Flye**
>    - *"Single or Paired-end reads"*: `Single`
>        - {% icon param-files %} *"Select fastq dataset"*: `DRR187559_forward`, `DRR187559_reverse`
>    - *"Set read groups information?"*: `Do not set`
>    - *"Select analysis mode"*: `5.Full list of options`
>        - *"Set algorithmic options?"*: `Do not set`
>        - *"Set scoring options?"*: `Do not set`
>        - *"Set input/output options"*: `Set`
>            - *"Output all alignments for single-ends or unpaired paired-ends"*: `Yes`
>    - *"BAM sorting mode"*: `Not sorted (sorted as input)`
>
{: .hands_on}

We can now run **Polypolish**.

> <hands-on-title> Polish assembly </hands-on-title>
>
> 1. {% tool [Polypolish](toolshed.g2.bx.psu.edu/repos/iuc/polypolish/polypolish/0.5.0+galaxy2) %} with the following parameters:
>    - In *"Input sequences"*:
>        - {% icon param-file %} *"Select a draft genome for polishing"*: `consensus` output of **Flye**
>        - *"Select aligned data to polish"*: `Paired SAM/BAM files`
>            - {% icon param-file %} *"Select forward SAM/BAM file"*: output of **BWA-MEM2** on the `DRR187559_forward` output of **fastp**
>            - {% icon param-file %} *"Select reverse SAM/BAM file"*: output of **BWA-MEM2** on the `DRR187559_reverse` output of **fastp**
>
{: .hands_on}

To check the impact of the polishing, let's run **QUAST** on both **Flye** and **Polypolish** outputs.

> <hands-on-title>Quality Control of polished assembly</hands-on-title>
>
> 1. {% tool [Quast](toolshed.g2.bx.psu.edu/repos/iuc/quast/quast/5.2.0+galaxy1) %} with the following parameters:
>    - *Assembly mode?*: `Co-assembly`
>       - *"Use customized names for the input files?"*: `No, use dataset names`
>         - {% icon param-files %} *"Contigs/scaffolds file"*: `consensus` output of **Flye** and output of **Polypolish**
>
{: .hands_on}

The HTML report generated by **QUAST** gives metrics for both assembly side-by-side

Statistics | **Flye** output | **Polypolish** output
--- | --- | ---
Number of contigs | 2 | 2
Largest contig | 2,907,099 | 2,915,230
Total length (>= 0 bp) | 2,967,214 | 2,975,666
GC (%) | 32.91 | 32.84

> <question-title></question-title>
>
> Is the assembly after polishing better than before given the results for KUN1163 in [Table 1](https://journals.asm.org/doi/10.1128/mra.01212-19#tab1) in {% cite Hikichi_2019 %}?
>
> > <solution-title></solution-title>
> >
> > The largest contig after polishing has a length of 2,915,230 bp, which closr to the expected 2,914,567 bp. But the GC content (32.84% after polishing) is slightly worst given the  expected 32.91% also found in the assembly before polishing.
> >
> {: .solution}
{: .question}

</div>

# Conclusion

In this tutorial, we prepared long reads (using short reads if we had some) assembled them, inspect the produced assembly for its quality, and polished it (if short reads where provided). The assembly, even if uncomplete, is reasonable good to be used in downstream analysis, like [AMR gene detection]({% link topics/genome-annotation/tutorials/amr-gene-detection/tutorial.md %})


topic: assembly
tutorial: ERGA-post-assembly-QC
title: ERGA post-assembly QC
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["plants", "animals", "genome", "assembly", "QC", "biodiversity"]
content_hash: 5d09a80c34c1aa429fd4823bc8641491
last_modified: 2026-02-26T21:12:14.395698
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/ERGA-post-assembly-QC/tutorial.html

content:
![ERGA logo, 4 colorful letters on white background, with the European star circle](../../images/post-assembly-QC/ERGA.logo.jpg){: width="20px"}

The European Reference Genome Atlas ([ERGA](https://www.erga-biodiversity.eu/), {% cite Mazzoni2023 %}) is a large-scale network of researchers aiming to generate high-quality reference genomes for all eukaryotic life in Europe and build capacity to allow researchers anywhere to generate reference genomes and use them to answer questions regarding species conservation and biodiversity. ERGA uses state-of-the-art sequencing technologies and advanced bioinformatics tools to produce high-quality genome assemblies.

Reference genomes provide a baseline for understanding genetic diversity within and among populations, and can be used to identify populations at risk of genetic erosion. This information is crucial for developing effective conservation strategies and management plans for threatened and endangered species ({% cite Shafer2015 %}). Additionally, by better understanding the genetic basis of important traits, such as disease resistance and adaptation to changing environments, researchers can develop targeted interventions to mitigate the effects of environmental change and prevent the loss of genetic diversity ({% cite Frankham2011 %}). The ERGA project has the potential to greatly benefit biodiversity conservation efforts and advance our understanding of the genetic basis of biodiversity.

Genome post-assembly quality control is a crucial step for evaluating the accuracy and completeness of newly assembled genomes. This involves assessing the contiguity, completeness, accuracy, and consistency of the genome assembly using various bioinformatic tools and methods ({% cite Koren2017 %}, {% cite Hunt2015 %}, {% cite Mikheenko2015 %}, {% cite Vaser2017 %}, {% cite Zimin2017 %}). The QC process aims to ensure that genomic data is reliable and useful for downstream analyses such as annotation, comparative genomics, and functional studies.

In this tutorial you will learn how to implement the ERGA Genome Assembly QC pipeline, and how to interpretate the potential outcomes.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Study cases

In this tutorial we will evaluate three genome assemblies, belonging to three different taxonomic groups, in order to illustrate the different scenarios that can be identified. The characteristics of each of them are described briefly below.

<strong>Case 1: <em>Chondrosia reniformis</em></strong>

*Chondrosia reniformis* is a slow-growing marine sponge cosmopolitan species which can be found in the Mediterranean Sea and the eastern Atlantic Ocean in shallow waters; it is considered to playing an important ecological role in the marine ecosystem by filtering large volumes of water and providing habitat for other species ({% cite Voultsiadou2005 %}). The members of this species are gonochoristic and oviparous, whose physiology and behavious seems to highly influence the presence of endosymbiosis heterotrophic bacteria ({% cite sara1997 %}). 

![Figure 1: Chrondosia reniformis](../../images/post-assembly-QC/chondrosia.jpg "Example <em>Chrondosia reniformis</em> sponge.")

The assembly is based on 70x PacBio HiFi and Arima2 Hi-C data generated by the [Aquatic Symbiosis Genomics Project](https://www.aquaticsymbiosisgenomics.org/). The assembly process included the following sequence of steps: initial PacBio assembly generation with Hifiasm, retained haplotig separation with purge_dups, and Hi-C based scaffolding with YaHS. The mitochondrial genome was assembled using MitoHiFi. Finally, the primary assembly was analysed and manually improved using gEVAL.

<strong>Case 2: <em>Erythrolamprus reginae</em></strong>

*Erythrolamprus reginae* is a species of colubrid snake found in South America. This species has been reported to include triploid individuals with parthenogenic reproduction; this type of seems to be associated with higher mutation rates and more tandem duplications in the genome ({% cite Bogart1980 %}). 

![Figure 2: Erythrolamprus reginae](../../images/post-assembly-QC/erytrocampus.jpg "Example <em>Erythrolamprus reginae</em> snake.")

The assembly used in this tutorial correspond to the curated primary assembly generated by the [VGP project](https://vertebrategenomesproject.org/), based on 34x Pacbio HiFi and Arima2 Hi-C data, by using the [VGP assembly pipeline]({% link topics/assembly/tutorials/vgp_genome_assembly/tutorial.md %}).

<strong>Case 3: <em>Eschrichtius robustus</em></strong>

*Eschrichtius robustus*, commonly known as the gray whale, is a species of whale found primarily in the North Pacific Ocean. Adult gray whales can reach lengths of up to 14.9 meters and weights of up to 36,000 kilograms. It is a diploid species, genetically characterized by high homozygosity ({% cite BrnicheOlsen2018 %}).

![Figure 3: Eschrichtius robustus](../../images/post-assembly-QC/eschrichtius.jpg "Example <em>Eschrichtius robustus</em> whale.")

The assembly used in this tutorial correspond to the curated primary assembly generated by the [VGP project](https://vertebrategenomesproject.org/), based on 29x Pacbio HiFi data, by using the [VGP assembly pipeline]({% link topics/assembly/tutorials/vgp_genome_assembly/tutorial.md %}).


# Get data

As a first step we will get the data from Zenodo.

> <hands-on-title>Upload data</hands-on-title>
>
> 1. Create a new history for allocating temporally all the datasets. This history can be named as `Datasets history`
>
>
> 2. Import the Pacbio HiFi files from Zenodo:
>
>    - Open the file {% icon galaxy-upload %} __upload__ menu
>    - Click on **Rule-based** tab
>    - *"Upload type"*: `Collections`
>    - Copy the tabular data, paste it into the textbox and press <kbd>Build</kbd>
>
>       ```
>   dataset_01   https://zenodo.org/record/7786773/files/hifi.fastq.gz   fastqsanger.gz    HiFi    CReniformis_Pacbio
>   dataset_01   https://zenodo.org/record/7784764/files/m64055_210602_211608.hifi_reads.fastq.gz   fastqsanger.gz    HiFi    EReginae_PacBio
>   dataset_02   https://zenodo.org/record/7784764/files/m54306Ue_211020_191957.hifi_reads.fastq.gz   fastqsanger.gz    HiFi    EReginae_PacBio
>   dataset_03   https://zenodo.org/record/7784764/files/m54306Ue_211016_070813.hifi_reads.fastq.gz   fastqsanger.gz    HiFi    EReginae_PacBio
>   dataset_04   https://zenodo.org/record/7786773/files/m54306U_210529_225553.hifi_reads.fastq.gz   fastqsanger.gz    HiFi    EReginae_PacBio
>   dataset_01   https://zenodo.org/record/7781236/files/m54306Ue_220411_220734.demultiplex.bc1010--bc1010.hifi_reads.fastq.gz   fastqsanger.gz    HiFi    ERobustus_PacBio
>   dataset_02   https://zenodo.org/record/7781236/files/m54306Ue_220520_051743.demultiplex.bc1010--bc1010.hifi_reads.fastq.gz   fastqsanger.gz    HiFi    ERobustus_PacBio
>   dataset_03   https://zenodo.org/record/7781236/files/m64055e_220615_033108.demultiplex.bc1010--bc1010.hifi_reads.fastq.gz   fastqsanger.gz    HiFi    ERobustus_PacBio
>   dataset_04   https://zenodo.org/record/7786773/files/m64055e_220603_182128.demultiplex.bc1010--bc1010.hifi_reads.fastq.gz   fastqsanger.gz    HiFi    ERobustus_PacBio
>       ```
>    - Click on `Build`
>    - From **Rules** menu select `Add / Modify Column Definitions`
>       - Click `Add Definition` button and select `List Identifier(s)`: column `A`
>       - Click `Add Definition` button and select `URL`: column `B`
>       - Click `Add Definition` button and select `Type`: column `C`
>       - Click `Add Definition` button and select `Group Tag`: column `D`
>       - Click `Add Definition` button and select `Collection Name`: column `E`
>    - Click `Apply` and press <kbd>Upload</kbd>
>
> 3. Import the remainig datasets from Zenodo:
>
>    - Open the file {% icon galaxy-upload %} __upload__ menu
>    - Click on **Rule-based** tab
>    - *"Upload type"*: `Datasets`
>    - Copy the tabular data, paste it into the textbox and press <kbd>Build</kbd>
>
>       ```
>   CReformitis_assembly    https://zenodo.org/record/7831298/files/assembly_sponge.fasta.gz    fasta.gz    assembly
>   CReformitis_metadata    https://zenodo.org/record/7781236/files/metadata_chon.yaml  yaml    metadata
>   CReniformis_Hi-C_F   https://zenodo.org/record/7786773/files/hiC_1.fastq.gz   fastqsanger.gz    Hi-C
>   CReniformis_Hi-C_R   https://zenodo.org/record/7786773/files/hiC_2.fastq.gz   fastqsanger.gz    Hi-C
>   EReginae_Hi-C_F_01   https://zenodo.org/record/7831762/files/rEryReg1_Royal_Ground_Snake_R1.fastq.gz.gz.001   fastqsanger.gz    Hi-C
>   EReginae_Hi-C_F_02   https://zenodo.org/record/7831762/files/rEryReg1_Royal_Ground_Snake_R1.fastq.gz.gz.002   fastqsanger.gz    Hi-C
>   EReginae_Hi-C_F_03   https://zenodo.org/record/7831763/files/rEryReg1_Royal_Ground_Snake_R1.fastq.gz.gz.003   fastqsanger.gz    Hi-C
>   EReginae_Hi-C_R_01   https://zenodo.org/record/7833514/files/rEryReg1_Royal_Ground_Snake_R2.fastq.gz.gz.001   fastqsanger.gz    Hi-C
>   EReginae_Hi-C_R_02   https://zenodo.org/record/7833514/files/rEryReg1_Royal_Ground_Snake_R2.fastq.gz.gz.002   fastqsanger.gz    Hi-C
>   EReginae_Hi-C_R_03   https://zenodo.org/record/7831763/files/rEryReg1_Royal_Ground_Snake_R2.fastq.gz.gz.003   fastqsanger.gz    Hi-C
>   EReginae_assembly    https://zenodo.org/record/7788734/files/rEryReg1.pri.cur.20230105.fasta.gz    fasta.gz    assembly
>   EReginae_metadata    https://zenodo.org/record/7781236/files/metadata_eryth.yaml  yaml    metadata
>   ERobustus_assembly  https://zenodo.org/record/7781236/files/mEscRob2.pri.cur.20221201.fasta.gz  fasta.gz    assembly
>   ERobustus_metadata  https://zenodo.org/record/7781236/files/metadata_esch.yaml  yaml    metadata
>   Taxonomy_data   https://zenodo.org/record/7781236/files/new_taxdump.tar.gz  gz  taxonomy
>   Diamond_db   https://zenodo.org/record/7788734/files/Diamond_makedb.dmnd  dmnd  database
>       ```
>
>    - From **Rules** menu select `Add / Modify Column Definitions`
>       - Click `Add Definition` button and select `Name`: column `A`
>       - Click `Add Definition` button and select `URL`: column `B`
>       - Click `Add Definition` button and select `Type`: column `C`
>       - Click `Add Definition` button and select `Name Tag`: column `D`
>    - Click `Apply` and press <kbd>Upload</kbd>
>
{: .hands_on}

# What data have we imported?

For each species, we have uploaded the fasta file corresponding to the assembly, the raw sequencing data used to generate each assembly - in these cases PacBio HiFi longreads and Illumina Hi-C reads - and databases necessary to run the QC tools included in this tutorial. We will use the taxonomy database `new_taxdump.tar.gz` and the diamond database of protein sequences from the ncbi nt database to identify any sequence from contaminants or symbiont which are in our assembly using `BlobToolKit`.

Once we have imported all the datasets, we will move each one to its correspondent history.

> <hands-on-title>Upload data</hands-on-title>
>
> 1. Create three new empty histories, one for each specie.
> 2. Rename the histories as  `Case 1: *Chondrosia reniformis*`, `Case 2: *Erythrolamprus reginae*` and  `Case 3: *Eschrichtius robustus*`.
> 3. Click in **History options** and select `Show Histories Side-by-Side`
> 4. Click in `Select histories`, and include the histories corresponding to the three species.
> 5. Drag and drop to move the datasets to their correspondent histories.
>
>    > <comment-title>Non-unique datasets</comment-title>
>    > Both the **Taxonomic_data** and the **Diamond_database** should be included in all of them.
>    {: .comment}
>
> {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

Once all the datasets have been copied to their correspondent history, we should obtain something similar to this:

![Figure 4: Distribution of data](../../images/post-assembly-QC/histories_side_by_side.png "Histories corresponding to the three cases of study.")

# Genome assembly overview with BlobToolKit

**BlobToolKit** is a tool designed to assist researchers in analyzing and visualizing genome assembly data. The tool uses information from multiple data sources such as read coverage, sequence length, and taxonomic annotations to generate a comprehensive overview of genome assembly data ({% cite Challis2020 %}). One of the key characteristics of BlobToolKit is its ability to provide with a user-friendly interactive interface for analyzing complex genome assembly data.

In this tutorial, we will use BlobToolKit in order to integrate the following data:

- **Read coverage data**: BlobToolKit can use read coverage information to identify potential errors or gaps in the genome assembly. By comparing the depth of coverage across the genome, it can highlight regions that may be overrepresented or underrepresented, which can indicate potential issues with the assembly.
- **Taxonomic annotations**: BlobToolKit can use taxonomic information to identify potential contaminants or foreign DNA in the genome assembly. It does this by comparing the taxonomic profile of the genome assembly to a reference database of known organisms.
- **Sequence similarity data**: Sequence similarity data can be used to identify potential misassemblies or contaminants in the genome assembly. BlobToolKit can use BLAST/DIAMOND searches to compare the genome assembly to reference databases and identify regions that may be problematic.
- **BUSCO reports**: BlobToolKit can use BUSCO data to provide additional information about the quality of a genome assembly. It can generate plots of the number of complete and partial BUSCO genes in the genome assembly, as well as the number of missing and fragmented genes.

> <comment-title>Why should we evaluate contaminants?</comment-title>
>
> A significant proportion of the genome sequences in both GenBank and RefSeq (0.54% and 0.34% of entries, respectively) include sequences from contaminants; the contamination primarily exists in the form of short contigs, flanking regions on longer contigs, or areas of larger scaffolds that are flanked by Ns, although a few longer sequences with contamination were also detected ({% cite Steinegger2020 %}).
>
{: .comment}

In the next steps, we will generate the data required for generating the visualization plots with BlobToolKit.

## Generate read coverage data with **Minimap2**

Read coverage is an essential metric for evaluating the quality of genome assemblies, and it provides valuable information for identifying regions of high and low quality, detecting misassemblies, and identifying potential contaminants. Thus, for example, unexpected regions of low coverage suggests potential errors, such as misassemblies, gaps, or low complexity regions ({% cite Koren2017 %}).

In this tutorial we will use **Minimap2** for generation the coverage data. Minimap2 is a multi-purpose aligner which is particularly efficient at aligning long-reads produced by sequencing machines from PacBio or Oxford Nanopore ({% cite Li2018 %}).

> <comment-title>How is coverage information encoded in the BAM file?</comment-title>
>
> Coverage information is encoded in the BAM file format through the number of reads that align to each position in the reference genome. Specifically, the BAM file stores each read's start and end position in the genome, as well as the length and orientation of the read, among other information. To calculate coverage, the number of reads that overlap each position in the reference genome is counted.
>
{: .comment}

> <hands-on-title>Generate BAM file with Minimap2</hands-on-title>
>
> 1. {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0) %} with the following parameters:
>    - {% icon param-collection %} *"Collection of files to collapse into single dataset"*: `CReniformis_Pacbio`
>
>
> 2. {% tool [Map with minimap2](toolshed.g2.bx.psu.edu/repos/iuc/minimap2/minimap2/2.28+galaxy0) %} with the following parameters:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>        - {% icon param-file %} *"Select the reference genome"*: `CReformitis_assembly`
>    - *"Single or Paired-end reads"*: `Single`
>        - {% icon param-file %} *"FASTA/Q file"*: output of **Collapse Collection** {% icon tool %}
>    - *"Select a profile of preset options"*: `PacBio HiFi reads vs reference mapping (-k19 -w19 -U50,500 -g10k -A1 -B4 -O6,26 -E2,1 -s200 ) (map-hifi)`
>
> 3. Repeat the prevous steps with the datasets from the two remanining species.
>
{: .hands_on}


## Generate sequence similarity data with **DIAMOND**

**DIAMOND** is a sequence alignment tool that utilizes a more efficient algorithm compared to BLAST, allowing for much faster searches of large sequence databases. Specifically, DIAMOND uses a sensitive seed-extension approach that compares a set of small segments (seeds) from the query sequence to a database, and then extends the alignments based on the highest-scoring hits. This approach allows DIAMOND to perform up to 20,000 times faster than BLAST, with comparable or improved sensitivity and accuracy ({% cite Buchfink2014 %}).

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Diamond](toolshed.g2.bx.psu.edu/repos/bgruening/diamond/bg_diamond/2.0.15+galaxy0) %} with the following parameters:
>    - *"Alignment mode"*: `DNA query sequences (blastx)`
>        - *"Allow for frameshifts?"*: `yes`
>            - *"restrict hit culling to overlapping query ranges"*: `Yes`
>            - *"frame shift penalty"*: `15`
>    - {% icon param-file %} *"Input query file in FASTA or FASTQ format"*: `CReformitis_assembly`
>    - *"Will you select a reference database from your history or use a built-in index?"*: `Use one from the history`
>        - {% icon param-file %} *"Select the reference database"*: `Diamond_db`
>    - *"Restrict search taxonomically?"*: `No`
>    - *"Sensitivity Mode"*: `Fast (--fast)`
>        - *"Block size in billions of sequence letters to be processed at a time"*: `10.0`
>    - *"Method to filter?"*: `Maximum e-value to report alignments`
>    - *"Method to restrict the number of hits?"*: `Maximum number of target sequences`
>    - In *"Output options"*:
>        - *"Format of output file"*: `BLAST tabular`
>            - *"Tabular fields"*: `Query Seq-id`,`Subject Seq -id`, `Start of alignment in query`, `End of alignment in query`, `Expected value`, `Bit score` and `Unique Subject Taxonomy ID(s), separated by a ',' (in numerical order)`
>
> 2. Repeat the prevous steps with the datasets from the two remanining species.
>
{: .hands_on}


## Generate **BUSCO** report

**BUSCO** (Benchmarking Universal Single-Copy Orthologs) is a tool used for quantitative assessment of genome assembly, gene set, and transcriptome completeness. It is based on evolutionarily informed expectations of gene content derived from near-universal single-copy orthologs ({% cite Simo2015 %})

> <comment-title> Orthologs </comment-title>
>
> Orthologs are genes in different species which have usually the same function and have evolved from a common ancestral gene. They are important for new genome assemblies in order to predict gene functions and help with gene annotation ({% cite Gennarelli2010 %}).
>
{: .comment}

> <hands-on-title> Estimate single copy gene representation completeness </hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Sequences to analyse"*: `CReformitis_assembly`
>    - *"Lineage data source"*: `Use cached lineage Data`
>    - *"Cached database with lineage"*: `Busco v5 Lineage Datasets`
>    - *"Mode"*: `Genome assemblies (DNA)`
>        - *"Select a gene predictor"*: `Metaeuk`
>    - *"Auto-detect or select lineage?"*: `Auto-detect` (For *Erythrolamprus reginae*, select `Select lineage` then `Vertebrata` in *"Lineage"*)
>    - *"auto-lineage group"*: `All taxonomic groups (--auto-lineage)`
>    - *"Which outputs should be generated"*: `Short summary text`
>
>
>    > <details-title> Additional information </details-title>
>    >
>    > BUSCO sets represent 3,023 genes for vertebrates, 2,675 for arthropods, 843 for metazoans, 1,438 for fungi and 429 for eukaryotes. An intuitive metric is provided in BUSCO notation - C:complete[S:single, D:duplicated], F:fragmented, M:missing, n:number of genes used.
>    {: .details}
>
> 2. Repeat the prevous steps with the datasets from the two remanining species.
>
{: .hands_on}

> <question-title></question-title>
>
> What are the values of complete BUSCO genes for each species?
>
> > <solution-title></solution-title>
> >
> > 1. Eschrichtius robustus: 96.0%
> > 2. Erythrolamprus reginae: 91.3%
> > 3. Chondrosia reniformis: 82.8%
> >
> {: .solution}
>
{: .question}


## Generate interactive plots with **BlobToolKit**

**BlobToolKit** is a tool designed to assist researchers in analyzing and visualizing genome assembly data. The tool uses information from multiple data sources such as read coverage, gene expression, and taxonomic annotations to generate a comprehensive overview of genome assembly data ({% cite Challis2020 %}). One of the key characteristics of BlobToolKit is its ability to provide with a user-friendly interactive interface for analyzing complex genome assembly data.

To work with BlobToolKit we need to create a new dataset structure called **BlobDir**. Therefore the minimum requirement is a fasta file which contains the sequence of our assembly. A list of sequence identifiers and some statistics like length, GC proportion and undefined bases will then be generated.

To get a more meaningful analysis and therefore more useful information about our assembly, it is better to provide as much data as we can get. In our case we will also provide a metadata file if possible, NCBI taxonomy ID and the NCBI taxdump directory ({% cite Challis2020 %}).

> <hands-on-title> Creating the BlobDir dataset </hands-on-title>
>
> 1. {% tool [BlobToolKit](toolshed.g2.bx.psu.edu/repos/bgruening/blobtoolkit/blobtoolkit/3.4.0+galaxy0) %} with the following parameters:
>    - *"Select mode"*: `Create a BlobToolKit dataset`
>        - {% icon param-file %} *"Genome assembly file"*: `CReformitis_assembly`
>        - {% icon param-file %} *"Metadata file"*: `CReformitis_metadata`
>        - *"NCBI taxonomy ID"*: `121349`
>        - {% icon param-file %} *"NCBI taxdump directory"*: `Taxonomy_Data`
>
>
> 2. {% tool [BlobToolKit](toolshed.g2.bx.psu.edu/repos/bgruening/blobtoolkit/blobtoolkit/3.4.0+galaxy0) %} with the following parameters:
>    - *"Select mode"*: `Add data to a BlobToolKit dataset`
>        - {% icon param-file %} *"Blobdir.tgz file"*: output of **BlobToolKit** {% icon tool %}
>        - {% icon param-file %} *"BUSCO full table file"*: `Full table` (output of **Busco** {% icon tool %})
>        - *"BLAST/Diamond hits"*: `Enabled`
>           - *"BLAST/Diamond hits dataset"*: output of **Diamond** {% icon tool %}
>           - *"BLAST/Diamond file column order"*: `1=qseqid,7=staxids,6=bitscore,2=sseqid,3=qstart,4=qend,5=evalue` 
>        - {% icon param-file %} *"BAM/SAM/CRAM read alignment file"*: output of **Minimap2** {% icon tool %}
>
> 3. {% tool [Interactive BlobToolKit](interactive_tool_blobtoolkit) %} with the following parameters:
>    - {% icon param-file %} *"Blobdir file"*: output of **BlobToolKit** {% icon tool %}
>
> 4. Click on the {% icon galaxy-eye %} (eye) icon and inspect the output of **Interactive BlobToolKit**.
>
> 5. Repeat the prevous steps with the datasets from the two remanining species.
>
{: .hands_on}

Now, we will evaluate three of plots that we can find in the Blobtoolkit interactve interface. First, we will start with the **snailplot**, which provides a holistic view of the assembly.

![Figure 5: BlobToolKit snail plot](../../images/post-assembly-QC/Blobdir.snail.choRen.png  "Chondrosia reniformis snail plot summary. The grey line represents the assembly scaffolds, where the distance to the center of the circle indicates the length of the scaffolds. A scale line in the center of the circle helps to gauge the length. Longest contig: The red line in the plot indicates the longest contig in the assembly. N50 and N90: The dark orange and light orange lines represent the scaffolds contained in the N50 and N90 metrics, respectively. GC/AT content: The outer light and dark blue lines represent the GC and AT content, respectively. In an ideal assembly, the line between the two colors should be consistent, without much fluctuation. Gaps: Gaps in the assembly, if present, are denoted by the percent N in the bottom right of the image.")

The main plot is divided into 1,000 size-ordered bins around the circumference with each bin representing 0.1% of the 117,390,217 bp assembly. The **distribution of sequence lengths is shown in dark grey** with the plot radius scaled to the longest sequence present in the assembly (10,413,042 bp, shown in red).

**Orange and pale-orange arcs show the N50 and N90 sequence lengths** (8,459,200 and 6,903,244 bp), respectively. The **pale grey spiral shows the cumulative sequence count on a log scale** with white scale lines showing successive orders of magnitude. The **blue and pale-blue area around the outside of the plot shows the distribution of GC, AT and N percentages** in the same bins as the inner plot. A summary of complete, fragmented, duplicated and missing BUSCO genes in the eukaryota_odb10 set is shown in the top right.

> <question-title>Snailplot question</question-title>
>
> Which differences do you appreciate between the snailplots corresponding to *Erythrolamprus reginae* and *Eschrichtius robustus*?
>
> > <solution-title></solution-title>
> >
> > ![Figure 6: BlobToolKit snailplots comparison](../../images/post-assembly-QC/Blobdir.snail.png "Comparison between snailplots from E. reginae (A) and E. robustus (B).")
> >
> > - (A) The snake (*Erythrolamprus reginae*) does have a relatively large proportion of the genome assembled into some few and long scaffolds (reflected in the high "longest scaffold" value and the high "Log10 scaffold count" value).
> > - (B) The whale (*Eschrichtius robustus*) does have a relatively large proportion of the genome assembled into fragmented scaffolds (reflected in the low N90 value compared to the N50 value and the high "Log10 scaffold count" value)
> >
> {: .solution}
>
{: .question}

Now, we are going to analyze the blob plot corresponding to *Erythrolamprus reginae*, which is especially useful for gaining insights into the composition of your genomic data and identify potential contaminants or endosymbionts. 

![Figure 7: BlobToolKit circle plot](../../images/post-assembly-QC/Blobdir.blob.circle.EryReg.png "Blob plot of base coverage in input against GC proportion for sequences in *E. reginate assembly*. Sequences are coloured by phylum. Circles are sized in proportion to sequence length . Histograms show the distribution of sequence length sum along each axis")

The blob plot is a two-dimensional scatter plot that helps in visualizing and analyzing genomic data for quality control, contaminant detection, and filtering. In the circle plot, **each sequence is represented by a circle**, with its diameter proportional to the sequence length. Circles are colored based on their taxonomic affiliation, and their positions on the X and Y axes are determined by their GC content and coverage, respectively. GC content is the proportion of G and C bases in the sequence, which can differ substantially between genomes. Coverage, on the other hand, is a measure of the number of times a particular sequence has been read during the sequencing process. The plot also includes coverage and GC histograms for each taxonomic group, weighted by the total span (cumulative length) of sequences occupying each bin.

> <comment-title>Pre-curated assembly evaluation</comment-title>
>
> Usually, the evaluation of the pre-curated assembly yield quite different results when compared with the post-curation evaluation. A frequent difference is the abudance of contigs that belong to different organisms, as a result of contamination in the reads. It can be clearly appreciated in the next figure, which correspond to the *E. reginae* pre-curated assembly. A striking element is the presence of 38 contigs corresponding to the phylum Artverviricota (dark green), a group of viruses that encode a reverse transcriptase.
>
> ![Figure 8: BlobToolKit circle plot](../../images/post-assembly-QC/Blobdir.blob.circle_precuration.png "Blob plot of base coverage in input against GC proportion for sequences in pre-curated E. reginate assembly. Sequences are coloured by phylum. Circles are sized in proportion to sequence length . Histograms show the distribution of sequence length sum along each axis")
>
> In addition, as we can appreciate, the samples included contamination from fungi, arthropoda and bacteria.
>
{: .comment}

> <question-title>Blob plot question</question-title>
>
> Is there significant contamination in the *Eschrichtius robustus* and *Chondrosia reniformis* assemblies?
>
> > <solution-title></solution-title>
> >
> > ![Figure 9: BlobToolKit snailplots comparison](../../images/post-assembly-QC/Blobdir.blob.circle.EscRob_ChoRen.png "Comparison between blob plots from E. robustus (A) and C. reniformis (B).")
> >
> > - (A) The whale (*Eschrichtius robustus*) does have significant contamination. Approximately half of the assembled sequences of the whale does contain contaminants or cosymbionts (total count: 703, Chordata count: 316), however as the foreign sequences are smaller, they make up a small percentage of the total assembly.
> > - (B) The sponge (*Condrosia reniformis*) doesn't contain any foreign sequences or contaminatio (total count: 15, Chordata count: 15).
> >
> {: .solution}
>
{: .question}

Finally, let's have a look at the cumulative sequence length plot, which shows the curves for subsets of scaffolds assigned to each phylum relative to the overall assembly. It is useful for evaluating the **contribution of the contigs from contaminated reads to the final assembly**. In that case, first we will evaluate all sequences together, and then we will remove those correspoding to chordata or not classified (not-hit) in order to be able to evaluate in detail each of the contaminants. So, let's start with the accumulative plot corresponding to all sequences (fig. 10).

In this kind of plot, **the x-axis represent the number of contigs** (sorted by species and length), and **the y-axis correspond to the cumulative length in nucleotides**. The gray line shows the cumulative length of all sequences. As we can appreciate, most sequences corespond to the taxa chordata (1.9Gb, distributed along 151 contigs). In addition, we can see that the final assembly includes 174 contigs corresponding to different taxa, or not classified at all. In order to be able to analyze the contribution of those sequences, we will filter the contigs corresponding to the chordata.

![Figure 10: BlobToolKit cumulative plot](../../images/post-assembly-QC/Blobdir.cumulative.png "Cumulative sequence length for assembly Blobdir. The grey line shows cumulative length for all sequences. Coloured lines show cumulative lengths of sequences assigned to each phylum using the bestsum taxrule. Plot axes are scaled to the filtered assembly.")

> <comment-title> How do I apply filters to a dataset? </comment-title>
>
> Datasets can be filtered based on any category or variable using the "Filters" menu. To filter based on a category, click the tab above any bar of the preview histogram to hide/show records assigned to this category. To filter variables, use the sliders at either end of the preview histogram to change the maximum and minimum values or enter the numbers directly into the text boxes above the histogram.
>
{: .comment}

By hidding the contings corresponding to chordata, we can have a detailed view of the contributions of each contaminant (fig. 11).

![Figure 11: BlobToolKit cumulative plot](../../images/post-assembly-QC/Blobdir.cumulative_filtered.png "The grey line shows cumulative length for all sequences. Coloured lines show cumulative lengths of sequences assigned to each phylum using the bestsum taxrule and are stacked by cumulative value on the x-axis to show the proportion of each phylum in the overall assembly. The assembly has been filtered to exclude sequences with phylum matches Chordata. Plot axes are scaled to the filtered assembly.")

In figure 11 we can appreciate clearly the relative contribution of each contaminant with respect to the total contamination. 

# K-mer based genome profiling and evaluation

k-mers can be utilized to infer genome characteristics such as length and heterozygosity by analyzing their frequency distribution in raw sequencing reads ({% cite Manekar2018 %}). In this tutorial we will make use of three different tools in order to perform the k-mer based genome profiling and assembly evaluation.

> <comment-title> Genome profiling details </comment-title>
>
> You can find more details about genome profiling in the [VGP training]({% link topics/assembly/tutorials/vgp_genome_assembly/tutorial.md %}#genome-profile-analysis).
>
{: .comment}


## Generating k-mer profile with **Meryl**

DNA is double stranded and normally only one strand is sequenced. For our assembly we want to consider the other strand as well. Therefore canonical k-mers are used in most counting tools, exactly like in **Meryl**. A full k-mer pair is a sequence and the reverse complement of the sequence (e.g. ATG/CAT). The canonical sequence of a k-mer pair is the lexicographically smaller of the two reverse complementary sequences. So if CAT appears it will be counted as ATG. ({% cite Clavijo %})

Meryl is a powerful tool for counting k-mers in large-scale genomic datasets. It uses a sorting-based approach that sorts the k-mers in lexicographical order.

> <hands-on-title> Generating k-mer profile </hands-on-title>
>
> 1. {% tool [Meryl](toolshed.g2.bx.psu.edu/repos/iuc/meryl/meryl/1.3+galaxy6) %} with the following parameters:
>    - *"Operation type selector"*: `Count operations`
>        - {% icon param-collection %} *"Input sequences"*: `CReniformis_Pacbio` 
>        - *"K-mer size selector"*: `Set a k-mer size`
>            - *"K-mer size"*: `21`
>
>    > <comment-title> compute k </comment-title>
>    >
>    > In general k can be computed as k=log4 (G(1-p)/p), with G as genome size and p as tolerable collision rate.
>    > In our case we set the k-mer size to 21.
>    {: .comment}
>
>
> 2. Run {% tool [Meryl](toolshed.g2.bx.psu.edu/repos/iuc/meryl/meryl/1.3+galaxy6) %} again with the following parameters:
>    - *"Operation type selector"*: `Operations on sets of k-mers`
>        - *"Operations on sets of k-mers"*: `Union-sum: return k-mers that occur in any input, set the count to the sum of the counts`
>        - {% icon param-file %} *"Input meryldb"*: previous output of **Meryl** {% icon tool %}
>
> 3. Run {% tool [Meryl](toolshed.g2.bx.psu.edu/repos/iuc/meryl/meryl/1.3+galaxy6) %} a third time with the following parameters:
>    - *"Operation type selector"*: `Generate histogram dataset`
>        - {% icon param-file %} *"Input meryldb"*: previous output of **Meryl** {% icon tool %}
>
> 4. Rename the output as `Histogram dataset`
>
{: .hands_on}


## K-mer profile analysis with **GenomeScope**

**Genomescope** is used for analysing genomes with the help of k-mer profile analysis. It estimates the overall genome characteristics and the overall read characteristics. The tool will use a given k-mer profile which is calculated only from raw reads sequencing data. It then generates a plot with the calculated data giving us information about the completeness and quality of the to be assembled data. ({% cite Vurture2017 %})

> <hands-on-title> Generate plots for analysis </hands-on-title>
>
> 1. {% tool [GenomeScope](toolshed.g2.bx.psu.edu/repos/iuc/genomescope/genomescope/2.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Input histogram file"*: `Histogram dataset`
>    - *"Ploidy for model to use"*: `2` for *C. reniformis* and *E. robustus*, `3` for *E. reginae*.
>
>
>    > <comment-title> Plots </comment-title>
>    >
>    > The generated plots will have the coverage on the x-axis and the frequency on the y-axis. It shows the fitted model in the observed k-mer data as well as the k-mer peaks and other information like some estimated parameters of the assembly.
>    {: .comment}
>
{: .hands_on}

![Figure 12: Genomescope plot](../../images/post-assembly-QC/Chondrosia-reniformis-Linear_plot.png "Genomescope 21-mer profile (k:21) of Chondrosia reniformis, diploid (p:2). The plot includes estimations about the genome length (len:124,167,659bp), unique sequences (uniq:61.5%), homozygous portion (aa: 98.7%), heterozygous portions (ab:1.32%), mean k-mer coverage for heterozygous bases (kcov:35.4), read error rate (err:0.752%) and average rate of read duplications (dup:0.378).")

Figure 12 corresponds to the k-mer profile of the sponge (*Chondrosia reniformis*). Presumably, the large gene number in the sponge genome is due to regional gene duplication; so far evidence for a transposition in sponges has been presented. Data indicate that only 0.25 % of the total sponge genome comprises CA/TG microsatellites, and until now also no SINEs/transposable elements have been identified. The estimated genome size around 124,Mbp is relatively close to *Chondrosia reniformis* genome size (117,39Mbp).

> <question-title>Genome profile question</question-title>
>
> Which differences can you appreciate between the genome profile of *Erythrolamprus reginae* and *Eschrichtius robustus*?
>
> > <solution-title></solution-title>
> >
> > ![Figure 13: Genomescope plot ](../../images/post-assembly-QC/genome_profiling_comparison.png "Genomescope genome profile. Erythrolamprus reginae (A). Eschrichtius robustus (B). ")
> >
> > - (A) The genome profile of the snake (*Erythrolamprus reginae*) suggests a high level of heterozygosity (reflected by the first peak at 11.1x coverage). 51.6% unique sequences have been estimated. In this case the low value is attributed to the parthenogenesis and the polyploidy of this snake which causes tandem duplications and higher DNA replication as a result of the triploid-diploid cycles.
> > - (B) The genome profile of the whale (*Eschrichtius robustus*) suggests a high level of homozygosity (reflected by the second peak at 29x coverage). The homozygous k-mer distribution can be expected from the high grade of inbreeding.
> >
> {: .solution}
>
{: .question}



## K-mer based assembly evaluation with **Merqury**

Merqury is designed for evaluating the completeness and accuracy of long-read genome assemblies using short-read sequencing data. Thus the quality of assemblies which are generated by using third-generation sequencing technologies can be reviewed and assessed by the tool.
Merqury works by comparing k-mers of an assembly to those from unassembled high-accuracy reads of the raw sequencing data. K-mer-based methods are also used to identify errors and missing sequences. ({% cite Rhie2020 %})

> <hands-on-title> Generating stats and plots </hands-on-title>
>
> 1. {% tool [Merqury](toolshed.g2.bx.psu.edu/repos/iuc/merqury/merqury/1.3+galaxy2) %} with the following parameters:
>    - *"Evaluation mode"*: `Default mode`
>        - {% icon param-file %} *"K-mer counts database"*: `read_db` (output of **Meryl** {% icon tool %})
>        - *"Number of assemblies"*: `One assembly (pseudo-haplotype or mixed-haplotype)`
>            - {% icon param-file %} *"Genome assembly"*: `CReformitis_assembly` 
>
>    > <comment-title> Output </comment-title>
>    >
>    > Merqury will now generate following outputs:
>    >
>    >  1. stats with completeness statistics
>    >  2. QV stats with quality value statistics
>    >  3. plots
>    >
>    {: .comment}
>
{: .hands_on}

Let's have a closer look at the copy number plots for each of the three species. Merqury will generate k-mers from the raw sequencing data (in the following called the 'read set') and will compare them to the assembly, in contrast to genomescope. Despite being diploid or triploid the species only got plotted with the primary assembly. This is why only one-copy k-mers get evaluated.

![Figure 14: Merqury plot](../../images/post-assembly-QC/Eschrichtius-robustus.spectra-cn.fl.png "Merqury copy numbers (CN) plot of the whale (*Eschrichtius robustus*). The red area displays the k-mers of the assembly. The black area displays the k-mers only found in the read set. The black area can be indicative for sequencing error in the read set or missing sequences in the assembly.")

The small black area indicates that most of the k-mers found in the read set are also found in the assembly (but not all since only the primary assembly got plotted). It reflects a highly homozygous assembly. This plot indicates a sequencing coverage at ~30x as seen before with genomescope.

> <question-title>Genome profile question</question-title>
>
> Which differences can you appreciate between the genome profile of *Chondrosia reniformis* and *Erythrolamprus reginae*?
>
> > <solution-title></solution-title>
> >
> > ![Figure 15: Merqury plot](../../images/post-assembly-QC/merqury_comparison_plot.png "Merqury copy numbers (CN) plot of Chondrosia reniformis (A) and Erythrolamprus reginae (B). The red area displays the k-mers of the assembly. The black area displays the k-mers only found in the read set. The black area can be indicative for sequencing error in the read set or missing sequences in the assembly.")
> >
> > - (A) The sponge (*Chondrosia reniformis*) does have two peaks as seen before with genomescope which is typical for a diploid species. The large black area indicates that there is a high amount of k-mers in the read set which is not used in the assembly since only the primary assembly got plotted. The secondary assembly is missing.
> > - (B) The snake (*Erythrolamprus reginae*) does have three peaks as seen before with genomescope which is typical for a triploid species. The black area is even larger compared to the sponge since only the primary assembly got plotted. The secondary and third assembly are missing.
> >
> {: .solution}
>
{: .question}


## Providing assembly statistics with **gfastats**

gfastats is a tool for providing summary statistics and genome file manipulation. In our case it will generate genome assembly statistics in a tabular-format output. Metrics like N50/L50, GC-content and lengths of contigs, scaffolds and gaps as well as other statistical information are provided for assessing the contiguity of the assembly.

> <hands-on-title> Generate summary statistics </hands-on-title>
>
> 1. {% tool [gfastats](toolshed.g2.bx.psu.edu/repos/bgruening/gfastats/gfastats/1.2.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: `CReformitis_assembly` 
>    - *"Specify target sequences"*: `Disabled`
>    - *"Tool mode"*: `Summary statistics generation`
>        - *"Report mode"*: `Genome assembly statistics (--nstar-report)`
>
>
>    > <comment-title> N50/NG50 </comment-title>
>    >
>    > Consider taking all contigs and sorting them by size. Starting with the largest and ending with the smallest. Now add up the length of each contig beginning with the largest, then the second largest and so on. When reaching 50% of the total length of all contigs it's done. The length of the contig you stopped is the N50 value. ({% cite Videvall201703 %})
>    >
>    > NG50 or more general NG(X) is based on the same idea as N50. The difference is that in this case the whole genome size or estimated genome size is taken into account. Through this comparisons can be made over different assemblies and genome sizes.
>    >
>    {: .comment}
>
>    > <comment-title> L50 </comment-title>
>    >
>    > Remember adding up the length of each contig until reaching the 50%. The L50 value is the number of the contig you have stopped.
>    >
>    > Example: The sum of all contigs together is 2000 kbp. The contig at 50% has length 300 kbp and is the third one and thus the third largest.
>    > Then N50 = 300 kbp and L50 = 3.
>    >
>    {: .comment}
>
> In the best case a high quality assembly should consist of just a few and large contigs to represent the genome as a whole. Therefore a good assembly should lead to a high N50 value and in contrast a low quality assembly with tiny, fragmented contigs would lead to a low N50 value. ({% cite Videvall201704 %})
>
> The metric doesn't only rely on measuring the 50% mark. The general case is N(X) where X ranges from 0 - 100 mostly in ten steps. However the NX metric is not suitable for comparing different species with different genome lengths.
>
>    > <details-title> GC-content </details-title>
>    >
>    > The GC-content or guanine-cytosine ratio tells one about the occurrence of guanine and cytosine in a genome. It is stated in percent. The two nucleobases are held together by three hydrogen bonds. A high GC-content makes DNA more stable than a low GC-content. Because the ratio of most species and organisms has been found out by now, it is also a good metric to gauge completeness.({% cite Wikipedia %})
>    {: .details}
>
>    > <details-title> CC(contig/chromosome) ratio </details-title>
>    >
>    > Another good metric for gauging contiguity is the CC ratio. The value is calculated by dividing contig counts by the chromosome pair number. A perfect score would be 1. Therefore the lower the value the better the contiguity of the assembly.
>    {: .details}
{: .hands_on}


# Hi-C scaffolding

To understand how chromosomes are arranged in their three-dimensional structure in the nucleus, high resolution and high throughput imaging techniques have been developed. Hi-C is a high throughput method to measure pairwise contacts between pairs of genomic loci. 
It is based on cross-linking the DNA in the nucleus and its histones (proteins around which the DNA is wrapped). However the DNA gets cut and marked and afterwards the cut parts get ligated together. Then the DNA gets fragmented and sequenced forwards and backwards generating paired reads. The resulting information can be used to assemble the reads to the corresponding chromosome.
({% cite Pal2018 %})

## Pre-processing Hi-C data

The following analysis use Hi-C reads in single datasets: one for the forward reads and one for the reverse reads. If you have more than one fastq datasets, as we do for *Erythrolamprus reginae* you will need to merge them into single datasets. 
If you have only one pair of datasets, skip this step. 

> <hands-on-title> Merge  <em>Erythrolamprus reginae</em> Hi-C data into single datasets </hands-on-title>
> 1. Create a collection with the Hi-C forward reads name `EReginae Hi-C forward reads`.
>
> 2. Do the same with the reverse reads (R) and name it `EReginae Hi-C reverse reads`.
>
> 3. For species with more than one set of Hi-C reads, verify that the datasets are ordered the same way in both collections. If not, sort both collections with the tool  {% tool Sort collection %} 
>
> 4. {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0) %} with the following parameters:
>    - {% icon param-collection %} *"Collection of files to collapse into single dataset"*: `EReginae Hi-C forward Reads`
>
> 5. Rename the dataset `EReginae Hi-C forward Reads`
> 
> 6. {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0) %} with the following parameters:
>    - {% icon param-collection %} *"Collection of files to collapse into single dataset"*: `EReginae Hi-C reverse reads`
>
> 7. Rename the dataset `EReginae Hi-C reverse Reads`
>
> {% snippet faqs/galaxy/collections_build_list.md name="EReginae Hi-C forward Reads" datasets_description="the fastq.gz containing the Hi-C forward reads (F)" n="1" %}
>
> {% snippet faqs/galaxy/datasets_rename.md name="Hi-C forward Reads" %}
> 
{: .hands_on}

> <hands-on-title> Mapping Hi-C reads against a reference genome </hands-on-title>
> 
> 1. {% tool [BWA-MEM2](toolshed.g2.bx.psu.edu/repos/iuc/bwa_mem2/bwa_mem2/2.2.1+galaxy0) %} with the following parameters:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>        - {% icon param-file %} *"Use the following dataset as the reference sequence"*: `CReformitis_assembly` 
> 
>    - *"Single or Paired-end reads"*: `Single`
>        - {% icon param-file %} *"Select fastq dataset"*: `Hi-C forward Reads` (output of **Collapse Collection** {% icon tool %})
> 
>    - *"Set read groups information?"*: `Do not set`
>    - *"Select analysis mode"*: `1.Simple Illumina mode`
>    - *"BAM sorting mode"*: `Sort by read names  (i.e., the QNAME field) `
>
> 2. Rename the bam dataset `Mapped Hi-C forward Reads`
>
>
> 3. {% tool [BWA-MEM2](toolshed.g2.bx.psu.edu/repos/iuc/bwa_mem2/bwa_mem2/2.2.1+galaxy0) %} with the following parameters:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>        - {% icon param-file %} *"Use the following dataset as the reference sequence"*: `CReformitis_assembly`
>    - *"Single or Paired-end reads"*: `Single`
>        - {% icon param-file %} *"Select fastq dataset"*: `Hi-C reverse Reads` (output of **Collapse Collection** {% icon tool %})
>    - *"Set read groups information?"*: `Do not set`
>    - *"Select analysis mode"*: `1.Simple Illumina mode`
>    - *"BAM sorting mode"*: `Sort by read names  (i.e., the QNAME field) `
>
> 4. Rename the bam dataset `Mapp


topic: assembly
tutorial: hifi-assembly
title: Genome assembly using PacBio data
difficulty: intermediate
time_estimation: 3h
tools_json: []
requirements_json: []
tags_json: ["assembly", "pacbio", "biodiversity"]
content_hash: 0734d0671af244911fc6a855bd14c2db
last_modified: 2026-02-26T21:12:14.464774
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/hifi-assembly/tutorial.html

content:
In this tutorial, we will assemble a genome of a species of fungi in the family Aspergillus, *Aspergillus niger*, from PacBio sequencing data. These data will be downloaded from ENA. The quality of the assembly obtained will be analyzed, in particular by comparing it to a reference assembly, available on ENA.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get data

We will use long reads sequencing data: HiFi (High Fidelity long reads) from PacBio sequencing of *Aspergillus niger* genome. This data is available on ENA. We will also use later a reference genome assembly downloaded from ENA.

## Get data from ENA

> <hands-on-title>Data upload from ENA</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [ENA](https://www.ebi.ac.uk/ena/browser/)
>
>    ```
>    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR317/012/SRR31719412/SRR31719412_subreads.fastq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>
> 3. Rename the datasets
> 4. Check that the datatype is `fastq.gz`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
>
{: .hands_on}

## Get reference genome from ENA

> <hands-on-title>Data upload from ENA</hands-on-title>
>
> 1. Reference genome is available here: [ASM4765177v1 assembly for Aspergillus niger](https://www.ebi.ac.uk/ena/browser/view/GCA_047651775.1)
> 2. Download the `WGS Set FASTA (JBKZXA01.fasta.gz)` on your computer
> Make sure you download `WGS Set FASTA (JBKZXA01.fasta.gz)` and *NOT*  `ALL Set FASTA`
> 4. Upload this file on Galaxy (Upload --> Choose local file --> Select the file --> Start --> Close)
> 5. Check that the datatype is `fasta.gz`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

# Genome Assembly

{% include _includes/cyoa-choices.html option1="Hifiasm" option2="Flye" default="Hifiasm"
       text="[Hifiasm](https://github.com/chhylp123/hifiasm) and [Flye](https://github.com/mikolmogorov/Flye) are two well known assembler." %}
<div class="Hifiasm" markdown="1">

Hifiasm is a fast haplotype-resolved de novo assembler initially designed for PacBio HiFi reads. In general, hifiasm generates the assembly graphs in the GFA format, so a step of conversion to fasta is necessary. The GFA 1.0 format is a tab-delimited text format for describing a set of sequences and their overlap.
<br>
Hifiasm produces arguably the best single-sample telomere-to-telomere assemblies combing HiFi, ultralong and Hi-C reads, and it is one of the best haplotype-resolved assemblers for the trio-binning assembly given parental short reads. For a human genome, hifiasm can produce the telomere-to-telomere assembly in one day.

> <hands-on-title>Assembly with Hifiasm</hands-on-title>
>
> 1. {% tool [Hifiasm](toolshed.g2.bx.psu.edu/repos/bgruening/hifiasm/hifiasm/0.25.0+galaxy0) %} with the following parameters:
>    - *"Mode"*: `Standard`
>    - {% icon param-file %} *"Input reads"*: the raw data (fastq.gz)
>    - *"Output log file"*: Set to yes
>
>     The tool produces five datasets: Haplotype-resolved raw unitig graph, Haplotype-resolved processed unitig graph without small bubbles, Primary assembly contig graph, Alternate assembly contig graph, [hap1]/[hap2] contig graph.
>
> 2. {% tool [GFA to FASTA](toolshed.g2.bx.psu.edu/repos/iuc/gfa_to_fa/gfa_to_fa/0.1.2) %} with the following parameters:
>    - {% icon param-file %} *"Input GFA file"*: primary assembly contig graph
{: .hands_on}

> <question-title></question-title>
>
> What are the different output datasets from Hifiasm?
>
> > <solution-title></solution-title>
> >
> > - Haplotype-resolved raw unitig graph: This graph keeps all haplotype information, including somatic mutations and recurrent sequencing errors.
> > - Haplotype-resolved processed unitig graph without small bubbles: This graph 'pops' small bubbles in the raw unitig graph; small bubbles might be caused by somatic mutations or noise in data, which are not the real haplotype information.
> > - Primary assembly contig graph: This graph includes a complete assembly with long stretches of phased blocks, though there may be some haplotype collapse.
> > - Alternate assembly contig graph: This graph consists of all contigs that are discarded from the primary contig graph.
> > - [hap1]/[hap2] contig graph: Each graph consists of phased contigs (output only with Hi-C phasing enabled).
> >
> {: .solution}
>
{: .question}

</div>

<div class="Flye" markdown="1">

We will use *Flye*, a de novo assembler for single molecule sequencing reads, such as those produced by PacBio and Oxford Nanopore Technologies. It is designed for a wide range of datasets, from small bacterial projects to large mammalian-scale assemblies. The package represents a complete pipeline: it takes raw PacBio / ONT reads as input and outputs polished contigs. Flye also has a special mode for metagenome assembly. All informations about Flye assembler are here: [Flye](https://github.com/fenderglass/Flye/).

> <hands-on-title>Assembly with Flye</hands-on-title>
>
> 1. {% tool [Flye](toolshed.g2.bx.psu.edu/repos/bgruening/flye/flye/2.9.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input reads"*: the raw data (fastq.gz)
>    - *"Mode"*: `PacBio HiFi`
>    - *"Number of polishing iterations"*: `1`
>    - *"Reduced contig assembly coverage"*: `Disable reduced coverage for initial disjointing assembly`
>    - *"Generate a log file"*: Set to yes
>
>     The tool produces four datasets: consensus, assembly graph, graphical fragment assembly and assembly info
{: .hands_on}

> <question-title></question-title>
>
> What are the different output datasets?
>
> > <solution-title></solution-title>
> >
> > - The first dataset (consensus) is a fasta file containing the final assembly (1461 contigs). You may notice that the result (contigs number) you obtained is sligthy different from the one presented here. This is due to the Flye assembly algorithm which doesn't always give the eact same results.
> > - The second and third dataset are assembly graph files. These graphs are used to represent the final assembly of a genome, they are based on reads and their overlap information. Some tools such as [Bandage](http://rrwick.github.io/Bandage/) allow to visualize the assembly graph.
> > - The fourth dataset is a tabular file (assembly_info) containing extra information about contigs/scaffolds.
> >
> {: .solution}
>
{: .question}
  
</div>

# Quality assessment

## Genome assemblies comparison with **Quast**

A way to calculate metrics assembly is to use ***QUAST = QUality ASsessment Tool***. Quast is a tool to evaluate genome assemblies by computing various metrics. The manual of Quast is here: [Quast](http://quast.sourceforge.net/docs/manual.html#sec3)
Quast allows to compare the assembly carried out with the reference genome and produces statistics such as the genome fraction.
However, when comparing to a reference genome, Quast results do not display the assembly N50, therefore, in this tutorial, you will generate two quast reports : One specifying a reference genome and one not specifying a reference genome.

> <hands-on-title>Quast</hands-on-title>
>
> 1. {% tool [Quast](toolshed.g2.bx.psu.edu/repos/iuc/quast/quast/5.0.2+galaxy3) %} without specifying a reference genome with the following parameters:
>    - *"Assembly mode?"*: `Individual assembly (1 contig file per sample)`
>    - *"Use customized names for the input files?"*: `No, use dataset names`
>        - {% icon param-collection %} *"Contigs/scaffolds file"*: `JBKZXA01.fasta.gz` (reference assembly), `fasta file` (output of **GFA to FASTA** {% icon tool %}) and/or `consensus` (output of **Flye** {% icon tool %})
>    - *"Type of assembly"*: `Genome`
>        - *"Use a reference genome?"*: `No`
>        - *"Type of organism"*: `Fungus: use of GeneMark-ES for gene finding, ...`
>
> 2. {% tool [Quast](toolshed.g2.bx.psu.edu/repos/iuc/quast/quast/5.0.2+galaxy3) %} specifying a reference genome with the following parameters:
>    - *"Assembly mode?"*: `Individual assembly (1 contig file per sample)`
>    - *"Use customized names for the input files?"*: `No, use dataset names`
>        - {% icon param-collection %} *"Contigs/scaffolds file"*: `fasta file` (output of **GFA to FASTA** {% icon tool %}) and/or `consensus` (output of **Flye** {% icon tool %})
>    - *"Type of assembly"*: `Genome`
>        - *"Use a reference genome?"*: `Yes`
>          - *"Reference genome"*: `JBKZXA01.fasta.gz` (reference assembly)
>        - *"Type of organism"*: `Fungus: use of GeneMark-ES for gene finding, ...`
>
> 
{: .hands_on}

> <question-title></question-title>
>
> Compare the different metrics obtained for the assembly you generated and the reference genome.
>
> > <solution-title></solution-title>
> >
> > We compare the metrics of the three assemblies:
> > - The reference genome (JBKZXA01.fasta.gz): 8 contigs, Contig N50 = 3.5Mb, Contig length max = 6.2 Mb, size = 35.4Mb, 49.51% GC
> > - The Hifiasm assembly: 105 contigs, N50 = 4.9 Mb, length max = 7.1Mb, assembly size = 42.1 Mb, 47.91% GC
> > - The Flye assembly: 13 contigs, N50 = 4.9Mb, length max = 6.8Mb, size = 38.7 Mb, 49.31% GC
> >
> {: .solution}
>
{: .question}

## Genome assembly assessment with **BUSCO**

***BUSCO (Benchmarking Universal Single-Copy Orthologs)*** allows a measure for quantitative assessment of genome assembly based on evolutionarily informed expectations of gene content. Details for this tool are here: [Busco website](https://busco.ezlab.org/)

> <hands-on-title>BUSCO on assembly</hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.8.0+galaxy0) %} with the following parameters:
>    - *"Tool version"*: `Galaxy Version 5.8.0+galaxy0`
>    - {% icon param-file %} *"Sequences to analyse"*: Multiple datasets
>    - {% icon param-collection %} *"Sequences to analyse"*: `JBKZXA01.fasta.gz` (reference assembly), `fasta file` (output of **GFA to FASTA** {% icon tool %}) and/or `consensus` (output of **Flye** {% icon tool %})
>    - *"Lineage data source"*: `Use Cached Lineage Data`
>    - *"Auto-detect or select lineage"*: `Select lineage`
>        - *"Lineage"*: `Ascomycota`
>        - *"Which outputs should be generated"*: `short summary text; summary image`
{: .hands_on}

> <question-title></question-title>
>
> Compare the number of BUSCO genes identified in the generated assembly and the reference genome. What do you observe ?
>
> > <solution-title></solution-title>
> >
> > Short summary generated by BUSCO indicates that reference genome (JBKZXA01.fasta.gz) contains:
> > 1. 1678 Complete BUSCOs (of which 1675 are single-copy and 3 are duplicated),
> > 2. 7 fragmented BUSCOs,
> > 3. 21 missing BUSCOs.
> >
> > Short summary generated by BUSCO indicates that Hifiasm assembly contains:
> > 1. 1685 complete BUSCOs (1679 single-copy and 6 duplicated),
> > 2. 6 fragmented BUSCOs
> > 3. 15 missing BUSCOs.
> >
> > Short summary generated by BUSCO indicates that Flye assembly contains:
> > 1. 1685 complete BUSCOs (1679 single-copy and 6 duplicated),
> > 2. 6 fragmented BUSCOs
> > 3. 15 missing BUSCOs.
> >
> > BUSCO analysis confirms that these two assemblies are of similar quality, with similar number of complete, fragmented and missing BUSCOs genes.
> >
> {: .solution}
>
{: .question}



# Conclusion


This pipeline shows how to generate and evaluate a genome assembly from long reads PacBio data. Once you are satisfied with your genome sequence, you might want to annotate it: have a look at the [RepeatMasker]({% link topics/genome-annotation/tutorials/repeatmasker/tutorial.md %}) and [Funannoate]({% link topics/genome-annotation/tutorials/funannotate/tutorial.md %}) tutorials to learn how to do it!


topic: assembly
tutorial: mrsa-illumina
title: Genome Assembly of a bacterial genome (MRSA) sequenced using Illumina MiSeq
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: ["illumina", "assembly", "microgalaxy"]
content_hash: 820f016a959bb0d8966f353d15f95dad
last_modified: 2026-02-26T21:12:14.481353
zenodo_link: https://zenodo.org/record/10669812
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/mrsa-illumina/tutorial.html

content:
Sequencing (determining of DNA/RNA nucleotide sequence) is used all over
the world for all kinds of analysis. The product of these sequencers are
reads, which are sequences of detected nucleotides. Depending on the
technique these have specific lengths (30-500bp) or using Oxford
Nanopore Technologies sequencing have much longer variable lengths.

{% snippet faqs/galaxy/sequencing_illumina_miseq.md %}

In this training we will build an assembly of a bacterial genome, from data produced in
"Complete Genome Sequences of Eight Methicillin-Resistant
*Staphylococcus aureus* Strains Isolated from Patients in
Japan" {% cite Hikichi_2019 %}:

> Methicillin-resistant *Staphylococcus aureus* (MRSA) is a major pathogen
> causing nosocomial infections, and the clinical manifestations of MRSA
> range from asymptomatic colonization of the nasal mucosa to soft tissue
> infection to fulminant invasive disease. Here, we report the complete
> genome sequences of eight MRSA strains isolated from patients in Japan.
{: .quote cite="{% cite_url Hikichi_2019 %}"}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Galaxy and data preparation

Any analysis should get its own Galaxy history. So let's start by creating a new one and get the data into it.

> <hands-on-title>History creation</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

Now, we need to import the data: 2 FASTQ files containing the reads from the sequencer.

> <hands-on-title>Import datasets</hands-on-title>
> 1. {% tool [Import](upload1) %} the files from [Zenodo]({{ page.zenodo_link }}) or from Galaxy shared data libraries:
>
>    ```
>    {{ page.zenodo_link }}/files/DRR187559_1.fastqsanger.bz2
>    {{ page.zenodo_link }}/files/DRR187559_2.fastqsanger.bz2
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
> 2. Rename the datasets to remove `.fastqsanger.bz2` and keep only the sequence run ID (`DRR187559_1` and `DRR187559_2`)
>
>    {% snippet faqs/galaxy/datasets_rename.md name="DRR187559_1" %}
>
> 3. Create a paired collection named `Paired Reads`
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
> 4. Tag the collection `#unfiltered`
>
>    {% snippet faqs/galaxy/collections_add_tag.md %}
>
> 5. **View** {% icon galaxy-eye %} the renamed files in the collection
>
{: .hands_on}

The datasets are both FASTQ files.

> <question-title></question-title>
>
> 1. What are the 4 main features of each read in a FASTQ file.
> 2. What does the `_1` and `_2` mean in your filenames?
>
> > <solution-title></solution-title>
> > 1. The following:
> >
> >    -   A `@` followed by a name and sometimes information of the read
> >    -   A nucleotide sequence
> >    -   A `+` (optional followed by the name)
> >    -   The quality score per base of nucleotide sequence (Each symbol
> >        represents a quality score, which will be explained later)
> >
> > 2. Forward and reverse reads, by convention, are labelled `_1` and `_2`, but they might also be `_f`/`_r` or `_r1`/`_r2`.
> {: .solution}
{: .question}

# Quality Control

During sequencing, errors are introduced, such as incorrect nucleotides being called. These are due to the technical limitations of each sequencing platform. Sequencing errors might bias the analysis and can lead to a misinterpretation of the data. Adapters may also be present if the reads are longer than the fragments sequenced and trimming these may improve the number of reads mapped. **Sequence quality control is therefore an essential first step in any analysis.**

Assessing the quality by hand would be too much work. That's why tools like
[NanoPlot](https://github.com/wdecoster/NanoPlot) or
[Falco](https://falco.readthedocs.io/en/latest/) are made, as they  generate a summary and plots of the data statistics. NanoPlot is
mainly used for long-read data, like ONT and PACBIO and Falco for short read,
like Illumina and Sanger. Falco is an efficiency-optimized rewrite of [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/). You can read more in our dedicated [Quality Control
Tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).

Before doing any assembly, the first questions we should ask about the input
reads include:

- What is the coverage of my genome?
- How good are my reads?
- Do I need to ask/perform for a new sequencing run?
- Is it suitable for the analysis I need to do?

> <hands-on-title>Quality Control</hands-on-title>
>
> 1. {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Raw read data from your current history"*: `Paired Reads`
>
> 2. Inspect the webpage outputs
>
{: .hands_on}

Falco combines quality statistics from all separate reads and combines them in plots. An important plot is the Per base sequence quality. 

DRR187559_1 | DRR187559_2
----------- | -----------
![Falco plot showing reads that are green](../../images/mrsa/falco-1.png) | ![Same as previous plot, but the beginning of the reads are slightly better quality](../../images/mrsa/falco-2.png)

Here you have the reads sequence length on the x-axes against the quality score (Phred-score) on the y-axis. The colour of the beams indicates three quality levels: 
- Green = good quality, 
- Yellow = mediocre quality, and 
- Red = bad quality.

For each position, a boxplot is drawn with:

- the median value, represented by the central intense coloured line
- the inter-quartile range (25-75%), represented by the geen, yellow or red box
- the 10% and 90% values in the upper and lower whiskers

For Illumina data it is normal that the first few bases are of some lower quality and how longer the reads get the worse the quality becomes. This is often due to signal decay or phasing during the sequencing run.


Before doing any assembly, the first questions we should ask about the input
reads include:

- What is the coverage of my genome?
- How good are my reads?
- Do I need to ask/perform for a new sequencing run?
- Is it suitable for the analysis I need to do?

Depending on the analysis it could be possible that a certain quality or length
is needed. In this case we are going to trim the data using **fastp** ({% cite Chen2018 %}):

- Trim the start and end of the reads if those fall below a quality score of 20

  Different trimming tools have different algorithms for deciding when to cut but trimmomatic will cut based on the quality score of one base alone. Trimmomatic starts from each end, and as long as the base is below 20, it will be cut until it reaches one greater than 20. A sliding window trimming will be performed where if the average quality of 4 bases drops below 20, the read will be truncated there. 
  
- Filter for reads to keep only reads with at least 30 bases: Anything shorter will complicate the assembly

> <hands-on-title>Quality improvement</hands-on-title>
>
> 1. {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/1.0.1+galaxy3) %} with the following parameters:
>    - *"Single-end or paired reads"*: `Paired Collection`
>        - *"Select paired collection(s)"*: `Paired Reads`
>    - In *"Filter Options"*:
>        - In *"Length filtering Options"*:
>            - *Length required*: `30`
>    - In *"Read Modification Options"*:
>        - In *"Per read cuitting by quality options"*:
>            - *Cut by quality in front (5')*: `Yes`
>            - *Cut by quality in tail (3')*: `Yes`
>            - *Cutting window size*: `4`
>            - *Cutting mean quality*: `20`
>    - In *"Output Options"*:
>        - *"Output JSON report"*: `Yes`
>
> 2. Edit the tags of the **fastp** FASTQ outputs to 
>    1. Remove the `#unfiltered` tag
>    2. Add a new tag `#filtered`
{: .hands_on}

**fastp** generates also a report, similar to Falco, useful to compare the impact of the trimming and filtering.

> <question-title></question-title>
>
> Looking at **fastp** HTML report
>
> 1. How did the average read length change before and after filtering?
> 3. Did trimming improve the mean quality scores?
> 4. Did trimming affect the GC content?
> 5. Is this data ok to assemble? Do we need to re-sequence it?
>
> > <solution-title></solution-title>
> >
> > 1. Read lengths went down more significantly:
> >    - Before filtering: 190bp, 221bp
> >    - After filtering: 189bp, 219bp
> > 3. It increased the percentage of Q20 and Q30 bases (bases with quality score above 20 and 30 respectively)
> > 4. No, it did not. If it had, that would be unexpected.
> > 5. This data looks OK. The number of short reads in R1 is not optimal but assembly should partially work but not the entire, closed genome.
> >
> {: .solution}
{: .question}


# Assembly

Now that the quality of the reads is determined and the data is filtered and/or trimmed, we can try to assemble the reads together to build longer sequences.

There are many tools that create assembly for short-read data, e.g. SPAdes ({% cite prjibelski2020using %}), Abyss ({% cite simpson2009abyss %}). In this tutorial, we use **Shovill**. Shovill is a SPAdes-based genome assembler, improved to work faster and only for smaller (bacterial) genomes.

## Assembly

{% snippet faqs/galaxy/analysis_results_may_vary.md %}

> <hands-on-title>Assembly using Shovill</hands-on-title>
>
> 1. {% tool [Shovill](toolshed.g2.bx.psu.edu/repos/iuc/shovill/shovill/1.1.0+galaxy1) %} with the following parameters:
>    - *"Input reads type, collection or single library"*: `Paired Collection`
>        - *"Paired collection"*: **fastp** `Paired-end output`
>    - In *"Advanced options"*:
>        - *"Estimated genome size"*: `2914567`
><!--        - *"Minimum contig length"*: `500`
>-->
{: .hands_on}

**Shovill* generates 3 outputs:
- A log file
- A FASTA file with contigs, i.e. contiguous length of genomic sequences in which bases are known to a high degree of certainty.

  > <question-title></question-title>
  > Inspect the `Contigs` file
  >
  > 1. How big is the first contig?
  > 2. What is the coverage of your biggest (first) contig?
  >
  > > <solution-title></solution-title>
  > > Both of these can be found in the header line of the Contigs produced by Shovill
  > >
  > > *The results can differ from this example, because Shovill can differ a bit per assembly*
  > >
  > > 1. 589,438 bp, almost 6 times lower than the estimated genome size
  > > 2. 17.7
  > {: .solution}
  {: .question}

- A file with the contig graph or assembly graph

  The assembly graph is more information-rich because it not only contains the sequences of all assembled fragments (including the ones shorter than the threshold length defined for inclusion of the fragments into the FASTA output), but also indicates the relative average coverage of the fragments by sequenced reads and how some of the fragments could potentially be bridged after resolving ambiguities manually.

## Assembly inspection

Th assembly graph format takes some getting used to before you can make sense out of the information it provides. This issue can be alleviated through the use of **Bandage** ({% cite wick2015bandage %}), a package for exploring assembly graphs through summary reports and visualizations of their contents.

> <hands-on-title>Assembly inspection</hands-on-title>
> 1. {% tool [Bandage Info](toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_info/2022.09+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Graphical Fragment Assembly"*: **Shovill** `Contig Graph`
>    - *"Produce jpg, png or svg file?"*: `.jpg`
>
{: .hands_on}

Let's look at the assembly statistics

> <question-title></question-title>
>
> 1. What is the number of dead end?
> 2. What is the number of nodes?
>
> > <solution-title></solution-title>
> >
> > 1. Only 2
> > 2. 131: a bit disapointing as it should be encoded in one contig
> {: .solution}
{: .question}

Let's now visualize the graph.

> <hands-on-title>Assembly inspection</hands-on-title>
> 1. {% tool [Bandage Image](toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_image/2022.09+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"Graphical Fragment Assembly"*: **Shovill** `Contig Graph`
>    
{: .hands_on}

![Bandage output showing a mess of a genome graph](./images/bandage.jpg)


> <question-title></question-title>
> First read [this page in the Bandage wiki](https://github.com/rrwick/Bandage/wiki/Simple-example) to help understand what the graph means.
>
> What do you think of this assembly? Is it useful? Is it good enough?
>
> > <solution-title></solution-title>
> >
> > This is a very messy assembly, with a lot of potential paths through the sequence. We cannot feel confident in the output FASTA file (as it is much smaller than the expected 2.9Mbp). In real life we might consider doing a hybrid assembly with Nanopore or other long read data to help resolve these issues.
> >
> {: .solution}
{: .question}


## Assembly Evaluation

To evaluate the assembly, we use also [Quast](http://quast.bioinf.spbau.ru/) ({% cite Gurevich2013 %}) (QUality ASsessment Tool), a tool providing quality metrics for assemblies. This tool can be used to compare multiple assemblies, can take an optional reference file as input to provide complementary metrics, etc

> <hands-on-title>Quality Control of assembly using Quast</hands-on-title>
>
> 1. {% tool [Quast](toolshed.g2.bx.psu.edu/repos/iuc/quast/quast/5.2.0+galaxy1) %} with the following parameters:
>    - *Assembly mode?*: `Co-assembly`
>       - *"Use customized names for the input files?"*: `No, use dataset names`
>          - {% icon param-file %} *"Contigs/scaffolds file"*: **Shovill** `Contigs` output
>
{: .hands_on}

QUAST outputs assembly metrics as an HTML file with metrics and graphs. 

This fasta file contains 

> <question-title></question-title>
>
> 1. How many contigs is there?
> 2. What is the total length of all contigs?
> 1. What is you GC content?
>
> > <solution-title></solution-title>
> >
> > 1. 30 contigs, meaning the chromosome is separated over multiple contigs. These contigs can also contain (parts of) plasmids.
> > 2. 2,911,349 (*Total length (>= 0 bp)*). Not far from the estimated genome size: 2,914,567 
> > 1. The GC content for our assembly was 32.77%. For comparison [MRSA Isolate HC1335 Strain](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5174738/) GC% is 32.89%.
> >
> {: .solution}
{: .question}

The total length and the GC content of the assembly are coherent with expectations. 

# Conclusion

In this tutorial, we prepared short reads, assembled them, and inspect the produced assembly for its quality. The assembly, even if uncomplete, is reasonable good to be used in downstream analysis, like [AMR gene detection]({% link topics/genome-annotation/tutorials/amr-gene-detection/tutorial.md %})


topic: assembly
tutorial: debruijn-graph-assembly
title: De Bruijn Graph Assembly
difficulty: introductory
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 25035b6a9f664cfb83dbaedf3cd6a39d
last_modified: 2026-02-26T21:12:14.411978
zenodo_link: https://doi.org/10.5281/zenodo.582600
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/debruijn-graph-assembly/tutorial.html

content:
# Optimised de Bruijn Graph assemblies using the Velvet Optimiser and SPAdes

In this activity, we will perform *de novo* assemblies of a short read set using the Velvet Optimiser and the SPAdes assemblers. We are using the Velvet Optimiser for illustrative purposes. For real assembly work, a more suitable assembler should be chosen - such as SPAdes.

The Velvet Optimiser is a script written by @gtn:slugger70 to optimise the k-mer size and coverage cutoff parameters for Velvet. More information can be found [in its repository](https://github.com/slugger70/VelvetOptimiser).

SPAdes is a de novo genome assembler written by Pavel Pevzner's group in St. Petersburg. More details on it can be found [on Spades' website](http://cab.spbu.ru/software/spades/)>



> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. [Get the data](#get-the-data)
> 2. [Assemble with the Velvet Optimiser](#assembly-with-the-velvet-optimiser)
> 3. [Assemble with SPAdes](#assemble-with-spades)
{: .agenda}

# Get the data

We will be using the same data that we used in the introductory tutorial, so if you have already completed that and have the data, skip this section.

> <hands-on-title>Getting the data</hands-on-title>
>
> 1. Create and name a new history for this tutorial.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the sequence read raw data (\*.fastq) from [Zenodo](https://zenodo.org/record/582600)
>
>    ```
>    https://zenodo.org/record/582600/files/mutant_R1.fastq
>    https://zenodo.org/record/582600/files/mutant_R2.fastq
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the files {% icon galaxy-pencil %}
>    - The name of the files are the full URL, let's make the names a little clearer
>    - Change the names to just the last part, `Mutant_R1.fastq`, `Mutant_R2.fastq`, respectively
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
>    > <question-title></question-title>
>    >
>    > 1. What are four key features of a FASTQ file?
>    > 2. What is the main difference between a FASTQ and a FASTA file?
>    {: .question}
>
> 4. Create a paired collection named `Paired Reads`
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
>    - We will need to use both the individual datasets (`Mutant_R1.fastq` and `Mutant_R2.fastq`) and the paired end collection (`Paired Reads`), so toggle off the `Hide original elements` option when creating the collection.
>    - Alternatively, you can un-hide the datasets by selecting the {% icon galaxy-show-hidden %} icon on the hidden dataset in your history.
>
>    {% snippet faqs/galaxy/datasets_unhidden.md %}
>
{: .hands_on}

# Assembly with the Velvet Optimiser

We will perform an assembly with the Velvet Optimiser, which automatically runs and optimises the output of the Velvet assembler ({% cite Velvet2008 %}). It will automatically choose a suitable value for the k-mer size (**k**). It will then go on to optimise the coverage cutoff (**cov_cutoff**) which corrects for read errors. It will use the "*n50*" metric for optimising the k-mer size and the "*total number of bases in contigs*" for optimising the coverage cutoff.

> <hands-on-title>Assemble with the Velvet Optimiser</hands-on-title>
>
>  1. {% tool [Velvet Optimiser](toolshed.g2.bx.psu.edu/repos/simon-gladman/velvetoptimiser/velvetoptimiser/2.2.6) %}: Optimise your assembly with the following parameters:
>    - *"Start k-mer size"*: `45`
>    - *"End k-mer size"*: `73`
>    - *"Input Files"*:
>        - `1: Input Files`
>            - *"Input file type"*: `Fastq`
>            - *"Single or paired end reads"*: `Paired`
>            - {% icon param-file %} *"Select first set of reads"*: `mutant_R1.fastq`
>            - {% icon param-file %} *"Select second set of reads"*: `mutant_R2.fastq`
>
{: .hands_on}

Your history will now contain a number of new files:

* Velvet optimiser contigs
  * A fasta file of the final assembled contigs
* Velvet optimiser contig stats
  * A table of the lengths (in k-mer length) and coverages (k-mer coverages) for the final contigs.

Have a look at each file.


> <hands-on-title>Get contig statistics for Velvet Optimiser contigs</hands-on-title>
>
> 1. {% tool [Fasta Statistics](toolshed.g2.bx.psu.edu/repos/iuc/fasta_stats/fasta-stats/1.0.1) %}: Produce a summary of the velvet optimiser contigs:
>    - {% icon param-file %} *"fasta or multifasta file"*: Select your velvet optimiser contigs file
>
> 2. View the output
>
>    > <question-title></question-title>
>    >
>    > Compare the output we got here with the output of the simple assemblies obtained in the introductory tutorial.
>    > 1. What are the main differences between them?
>    > 2. Which has a higher "n50"? What does this mean?
>    {: .question}
>
{: .hands_on}

Tables of results from **(a)** Simple assembly and **(b)** optimised assembly.

**(a)** ![The results of the contigs from Simple assembly.](../../images/image12.png)

**(b)** ![The results of the contigs from Optimised assembly. In contrast to simple assembly produced much higher n_50, while num_seq is lower.](../../images/optstats.png)

> <details-title>Further reading on assembly with Velvet</details-title>
> - Heuristic Resolution of Repeats and Scaffolding in the Velvet Short-Read de Novo Assembler ({% cite Zerbino2009 %})
>
{: .details}

## Visualisation of the Assembly

Now that we've assembled the genomes, let's visualise this assembly using [Bandage](https://rrwick.github.io/Bandage/) ({% cite Wick2015 %}). This tool will let us better understand how the assembly graph really looks, and can give us a feeling for if the genome was well assembled or not.

Currently VelvetOptimiser does not include the LastGraph output, so we will manually run `velveth` and `velvetg` with the optimised parameters.

> <hands-on-title>Manually running velvetg/h</hands-on-title>
>
> 1. Locate the output called "VelvetOptimiser: Contigs" in your history
>
> 2. Click the {% icon dataset-info %} information icon
>
> 3. Check the tool `stderr` in the information page for the optimised k-mer value
{: .hands_on}

> <question-title></question-title>
> What was the optimal k-mer value? (referred to as *"hash"* in the stderr log)
> > <solution-title></solution-title>
> > 55
> {: .solution}
{: .question}

With this information in hand, let's run velvet:

> <hands-on-title>Manually running velvetg/h</hands-on-title>
>
> 1. {% tool [velveth](toolshed.g2.bx.psu.edu/repos/devteam/velvet/velveth/1.2.10.3) %}: Prepare a dataset for the Velvet `velvetg` Assembler
>    - *"Hash length"*: `55`
>    - *"Input Files"*:
>      - `+ Insert Input Files`
>      - `1: Input Files`
>        - *"Choose the input type"*: `separate paired reads`
>        - *"read type"*: `shortPaired reads`
>        - *"Dataset"*: `mutant_R1.fastq` (forward reads)
>        - *"Dataset"*: `mutant_R2.fastq` (reverse reads)
>
> 2. {% tool [velvetg](toolshed.g2.bx.psu.edu/repos/devteam/velvet/velvetg/1.2.10.2) %}: Velvet sequence assembler for very short reads
>    - *"Velvet dataset"*: output from **velveth** {% icon tool %}
>    - *"Coverage cutoff"*: `Specify Cutoff Value`
>      - *"Remove nodes with coverage below"*: `1.44`
>    - *"Additional outputs"*: `Generate velvet LastGraph file`
>    - *"Using Paired Reads"*: `Yes`
>
{: .hands_on}

The LastGraph contains a detailed representation of the De Bruijn graph, which can give us an idea how velvet has assembled the genome and potentially resolved any conflicts.

> <hands-on-title>Bandage</hands-on-title>
>
> 1. {% tool [Bandage Image](toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_image/2022.09+galaxy4) %}: visualize de novo assembly graphs
>    - *"Graphical Fragment Assembly"*: The "LastGraph" output of **velvetg** {% icon tool %}
>    - *"Produce jpg, png or svg file?"*: `.png`
>
> 2. Execute
> 3. View the output file
{: .hands_on}

And now you should be able to see the graph that velvet produced:

![velvet graph](../../images/bandage-velvet.svg)

## Interpreting Bandage Graphs

k-mer size has a [significant effect](https://github.com/rrwick/Bandage/wiki/Effect-of-kmer-size) on the assembly. You can play around with various k-mers to see this effect in practice.

k-mer | graph
----- | -----
21    | [![21](../../images/bandage-velvet-21.svg)](../../images/bandage-velvet-21.svg)
33    | [![33](../../images/bandage-velvet-33.svg)](../../images/bandage-velvet-33.svg)
53    | [![53](../../images/bandage-velvet-53.svg)](../../images/bandage-velvet-53.svg)
77    | [![77](../../images/bandage-velvet-77.svg)](../../images/bandage-velvet-77.svg)

The next thing to be aware of is that there can be multiple valid interpretations of a graph, all equally valid in absence of other data. The following is taken verbatim [from Bandage's wiki](https://github.com/rrwick/Bandage/wiki/Simple-example):

> For a simple case, imagine a bacterial genome that contains a single repeated element in two separate places in the chromosome:
>
> ![Simple example 1]({% link topics/assembly/images/debruijn_assembly/bandage1.png %})
>
> A researcher (who does not yet know the structure of the genome) sequences it, and the resulting 100 bp reads are assembled with a de novo assembler:
>
> ![Simple example 2]({% link topics/assembly/images/debruijn_assembly/bandage2.png %})
>
> Because the repeated element is longer than the sequencing reads, the assembler was not able to reproduce the original genome as a single contig. Rather, three contigs are produced: one for the repeated sequence (even though it occurs twice) and one for each sequence between the repeated elements.
>
> Given only the contigs, the relationship between these sequences is not clear. However, the assembly graph contains additional information which is made apparent in Bandage:
>
> ![Simple example 3]({% link topics/assembly/images/debruijn_assembly/bandage3.png %})
>
> There are two principal underlying sequences compatible with this graph: two separate circular sequences that share a region in common, or a single larger circular sequence with an element that occurs twice:
>
> ![Simple example 4]({% link topics/assembly/images/debruijn_assembly/bandage4.png %})
>
> Additional knowledge, such as information on the approximate size of the bacterial chromosome, can help the researcher to rule out the first alternative. In this way, Bandage has assisted in turning a fragmented assembly of three contigs into a completed genome of one sequence.
{: .quote cite="https://github.com/rrwick/Bandage/wiki/Simple-example"}

# Assemble with SPAdes

We will now perform an assembly with the much more modern SPAdes assembler ({% cite Bankevich2012 %}). It goes through a similar process to Velvet in the fact that it uses and simplifies de Bruijn graphs but it uses multiple values for k-mer size and combines the resultant graphs. This combination produces very good assemblies. When using SPAdes it is typical to choose at least 3 k-mer sizes. One low, one medium and one high. We will use 33, 55 and 91.

> <hands-on-title>Assemble with SPAdes</hands-on-title>
>
> 1. {% tool [SPAdes](toolshed.g2.bx.psu.edu/repos/nml/spades/spades/4.2.0+galaxy0) %}: Assemble the reads:
>    - *"Operation mode"*: `Only assembler (--only_assembler)`
>    - *"Single-end or paired-end short-reads"*: `Paired-end: list of dataset pairs`
>      - *"FASTA/FASTQ file(s): collection"*: `Paired Reads`
>    - *"Set coverage cutoff option"*: `auto`
>    - *"Select k-mer detection option"*: `User specific`
>      - *"K-mer size values"*: `33,55,91` [note: no spaces!]
>    - *"Select optional output file(s)"*: `Assembly graph`, `Assembly graph with scaffold`, `Contigs`, `Scaffolds`, `Log`
>
{: .hands_on}

You will now have 5 new files in your history:

* two Fasta files, one for contigs and one for scaffolds
* two statistics files, one for contigs and one for scaffolds
* the SPAdes log file.

Examine each file, especially the stats files.

![Contig stats file with NODE_5 being the shortest contig with the highest coverage and NODE_1 being the opposite.](../../images/contig_stats.png)

> <question-title></question-title>
>
> 1. Why would one of the contigs have much higher coverage than the others?
> 2. What could this represent?
>
{: .question}


> <hands-on-title>Visualize assembly with Bandage</hands-on-title>
>
> 1. {% tool [Bandage Image](toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_image/2022.09+galaxy4) %} with the following parameters:
>    - *"Graphical Fragment Assembly"*: `assembly graph with scaffolds` output from **SPAdes** {% icon tool %}
>    - *"Produce jpg, png or svg file?"*: `.png`
>
> 2. Examine the output image {% icon galaxy-eye %}
>
{: .hands_on}

The visualized assembly should look something like this:

![bandage spades](../../images/bandage_spades.svg)


> <question-title></question-title>
>
> Which assembly looks better to you? Why?
>
{: .question}



> <hands-on-title>Get contig statistics for SPAdes contigs</hands-on-title>
>
> 1. {% tool [Fasta Statistics](toolshed.g2.bx.psu.edu/repos/iuc/fasta_stats/fasta-stats/1.0.1) %}: Produce a summary of the SPAdes contigs:
>    - {% icon param-file %} *"fasta or multifasta file"*: Select your velvet optimiser contigs file
>
> 2. Look at the output file.
>
>    > <question-title></question-title>
>    >
>    > Compare the output we got here with the output of the simple assemblies obtained in the introductory tutorial.
>    > 1. What are the main differences between them?
>    > 2. Did SPAdes produce a better assembly than the Velvet Optimiser?
>    {: .question}
>
{: .hands_on}


topic: assembly
tutorial: chloroplast-assembly
title: Chloroplast genome assembly
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: ["plants", "nanopore", "jbrowse1", "biodiversity"]
content_hash: 52db5a66dadcb9a04cc298c5673df94b
last_modified: 2026-02-26T21:12:14.410516
zenodo_link: https://doi.org/10.5281/zenodo.3567224
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/chloroplast-assembly/tutorial.html

content:
## What is genome assembly?

Genome assembly is the process of joining together DNA sequencing fragments into longer pieces, ideally up to chromosome lengths.The DNA fragments are produced by DNA sequencing machines, and are called "reads". These are in lengths of about 150 nucleotides (base pairs), to up to a million+ nucleotides, depending on the sequencing technology used. Currently, most reads are from Illumina (short), PacBio (long) or Oxford Nanopore (long and extra-long).

It is difficult to assemble plant genomes as they are often large (for example, 3,000,000,000 base pairs), have many repeat regions (such as transposons), and may be polyploid. This tutorial shows genome assembly for a smaller data set - the plant chloroplast genome - a single circular chromosome which is typically about 160,000 base pairs. It is thought that the the chloroplast evolved from a cyanobacteria that was living in plant cells.

In this tutorial, we will use a subset of a real data set from sweet potato, from the paper {% cite Zhou2018 %}. To find out more about each of the tools used here, see the tool panel page for a summary and links to more information.

><agenda-title></agenda-title>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Upload data

Let's start with uploading the data.

> <hands-on-title>Import the data</hands-on-title>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import from [Zenodo](https://zenodo.org/record/3567224) or a data library (ask your instructor):
>   - FASTQ file with illumina reads: `sweet-potato-chloroplast-illumina-reduced.fastq`
>   - FASTQ file with nanopore reads: `sweet-potato-chloroplast-nanopore-reduced.fastq`
>   - Note: make sure to import the files with "reduced" in the names, not the ones with "tiny" in the names.
>    ```
>    https://zenodo.org/record/3567224/files/sweet-potato-chloroplast-illumina-reduced.fastq
>    https://zenodo.org/record/3567224/files/sweet-potato-chloroplast-nanopore-reduced.fastq
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
{: .hands_on}

# Check read quality

We will look at the quality of the nanopore reads.

> <hands-on-title>Check read quality</hands-on-title>
>
> 1. {% tool [Nanoplot](toolshed.g2.bx.psu.edu/repos/iuc/nanoplot/nanoplot/1.28.2+galaxy1) %}:
>    - *"Select multifile mode"*: `batch`
>    - *"Type of file to work on"*: `fastq`
>    - *"files"*: select the `nanopore FASTQ file`
> 2. **View output**:
>    * There are five output files.
>    *  Look at the `HTML report` to learn about the read quality.
{: .hands_on}

> <question-title></question-title>
> What summary statistics would be useful to look at?
>
> > <solution-title></solution-title>
> > This will depend on the aim of your analysis, but usually:
> > * **Sequencing depth** (the number of reads covering each base position; also called "coverage"). Higher depth is usually better, but at very high depths it may be better to subsample the reads, as errors can swamp the assembly graph.
> > * **Sequencing quality** (the quality score indicates probability of base call being correct). You may trim or filter reads on quality. Phred quality scores are logarithmic: phred quality 10 = 90% chance of base call being correct; phred quality 20 = 99% chance of base call being correct. More detail [on Wikipedia](https://en.wikipedia.org/wiki/Phred_quality_score).
> > * **Read lengths** (read lengths histogram, and reads lengths vs. quality plots). Your analysis or assembly may need reads of a certain length.
> {: .solution}
{: .question}

*Optional further steps:*
* Find out the quality of your reads using other tools such as fastp or FastQC.
* To visualize base quality using emoji you can also use FASTQE.
* Run FASTQE for the illumina reads. In the output, look at the mean values (the middle row)
* Repeat FASTQE for the nanopore reads. In the tool settings, increase the maximum read length to 30000.
* To learn more, see the [Quality Control tutorial]({{site.baseurl}}/topics/sequence-analysis/tutorials/quality-control/tutorial.html)

# Assemble reads

We will assemble the long nanopore reads.

> <hands-on-title>Assemble reads</hands-on-title>
> 1. {% tool [Flye](toolshed.g2.bx.psu.edu/repos/bgruening/flye/flye/2.6+galaxy0) %}:
>    - *"Input reads"*: `sweet-potato-chloroplast-nanopore-reduced.fastq`
>    - *"Estimated genome size"*: `160000`
>    - *Leave other settings as default*
>
> 2. Re-name the `consensus` output file to `flye-assembly.fasta`
>
>    {% snippet faqs/galaxy/datasets_rename.md name="flye-assembly.fasta" %}
>
> 3. **View output**:
>    * There are five output files.
>    * *Note: this tool is heuristic; your results may differ slightly from the results here, and if repeated.*
>    * View the `log` file and scroll to the end to see how many contigs (fragments) were assembled and the length of the assembly.
>    * View the `assembly_info` file to see contig names and lengths.
{: .hands_on}


> <hands-on-title>View the assembly</hands-on-title>
> 1. {% tool [Bandage Info](toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_info/0.8.1+galaxy1) %}
>    - *"Graphical Fragment Assembly"*: the Flye output file `Graphical Fragment Assembly` (not the "assembly_graph" file)
>    - *Leave other settings as default*
> 2. {% tool [Bandage Image](toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_image/0.8.1+galaxy2) %}
>    - *"Graphical Fragment Assembly"*: the Flye output file `Graphical Fragment Assembly` (not the "assembly_graph" file)
>    - *"Node length labels"*: `Yes`
>    - *Leave other settings as default*
{: .hands_on}

Your assembly graph may look like this:

![assembly graph](../../images/sweet-potato-assembly-graph.png "Assembly graph of the nanopore reads, for the sweet potato chloroplast genome. In your image, some of the labels may be truncated; this is a known bug under investigation.")

*Note: a newer version of the Flye assembly tool now resolves this assembly into a single circle.*

> <question-title></question-title>
> What is your interpretation of this assembly graph?
> > <solution-title></solution-title>
> > One interpretation is that this represents the typical circular chloroplast structure: There is a long single-copy region (the node of around 78,000 bp), connected to the inverted repeat (a node of around 28,000 bp), connected to the short single-copy region (of around 11,000 bp). In the graph, each end loop is a single-copy region (either long or short) and the centre bar is the collapsed inverted repeat which should have about twice the sequencing depth.
> {: .solution}
{: .question}

> <comment-title>Further Learning</comment-title>
> * Repeat the Flye assembly with different parameters, and/or a filtered read set.
> * You can also try repeating the Flye assembly with an earlier version of the tool, to see the difference it makes. In the tool panel for Flye, click on the 'Versions' button at the top to change.
> * Try an alternative assembly tool, such as Canu or Unicycler.
{: .comment}

# Polish assembly

Short illumina reads are more accurate the nanopore reads. We will use them to correct errors in the nanopore assembly.

First, we will map the short reads to the assembly and create an alignment file.

> <hands-on-title>Map reads</hands-on-title>
> 1. {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.1) %}:
>    - *"Will you select a reference genome from your history"*: `Use a genome from history`
>    - *"Use the following dataset as the reference sequence"*: `flye-assembly.fasta`
>    - *"Algorithm for constructing the BWT index"*: `Auto. Let BWA decide`
>    - *"Single or Paired-end reads"*: `Single`
>    - *"Select fastq dataset"*: `sweet-potato-illumina-reduced.fastq`
>    - *"Set read groups information?"*: `Do not set`
>    - *"Select analysis mode"*: `Simple Illumina mode`
>2. **Re-name output file**:
>   - Re-name this file `illumina.bam`
{: .hands_on}

Next, we will compare the short reads to the assembly, and create a polished (corrected) assembly file.

> <hands-on-title>Polish</hands-on-title>
> 1. {% tool [pilon](toolshed.g2.bx.psu.edu/repos/iuc/pilon/pilon/1.20.1) %}:
>    - *"Source for reference genome used for BAM alignments"*: `Use a genome from history`
>    - *"Select a reference genome"*: `flye-assembly.fasta`
>    - *"Type automatically determined by pilon"*: `Yes`
>    - *"Input BAM file"*: `illumina.bam`
>    - *"Variant calling mode"*: `No`
>    - *"Create changes file"*: `Yes`
>
> 2. **View output**:
>   * What is in the `changes` file?
>   * Rename the fasta output to `polished-assembly.fasta`
>
>    {% snippet faqs/galaxy/datasets_rename.md name="polished-assembly.fasta" %}
>
> 3. {% tool [Fasta Statistics](toolshed.g2.bx.psu.edu/repos/iuc/fasta_stats/fasta-stats/1.0.1) %}
>    - Find and run the tool called "Fasta statistics" on both the original flye assembly and the polished version.
{: .hands_on}

> <question-title></question-title>
> How does the polished assembly compare to the unpolished assembly?
> > <solution-title></solution-title>
> > This will depend on the settings, but as an example: your polished assembly might be about 10-15 Kbp longer. Nanopore reads can have homopolymer deletions - a run of AAAA may be interpreted as AAA - so the more accurate illumina reads may correct these parts of the long-read assembly. In the changes file, there may be a lot of cases showing a supposed deletion (represented by a dot) being corrected to a base.
> {: .solution}
{: .question}

*Optional further steps:*
* Run a second round (or more) of Pilon polishing. Keep track of file naming; you will need to generate a new bam file first before each round of Pilon.
* Run an alternative polishing tool, such as Racon. This uses the long reads themselves to correct the long-read (Flye) assembly. It would be better to run this tool on the Flye assembly before running Pilon, rather than after Pilon.

# Annotate the assembly

We can now annotate our assembled genome with information about genomic features.

* A chloroplast genome annotation tool is not yet available in Galaxy; for an approximation, here we can use the tool for bacterial genome annotation, Prokka.

> <hands-on-title>Annotate with Prokka</hands-on-title>
>
> 1. {% tool [Prokka](toolshed.g2.bx.psu.edu/repos/crs4/prokka/prokka/1.14.5+galaxy0) %} with the following parameters (leave everything else unchanged)
>    - {% icon param-file %} *"contigs to annotate"*: `polished-assembly.fasta`
> 2. **View output**:
>    * The GFF and GBK files contain all of the information about the features annotated (in different formats.)
>    * The .txt file contains a summary of the number of features annotated.
>    * The .faa file contains the protein sequences of the genes annotated.
>    * The .ffn file contains the nucleotide sequences of the genes annotated.
{: .hands_on}


Alternatively, you might want to use a web-based tool designed for chloroplast genomes.

* One option is the GeSeq tool, described here. Skip this step if you have already used Prokka above.

> <hands-on-title>Annotate with GeSeq</hands-on-title>
> * Download `polished-assembly.fasta` to your computer (click on the file in your history; then click on the disk icon).
> * In a new browser tab, go to [Chlorobox](https://chlorobox.mpimp-golm.mpg.de/geseq.html) where we will use the GeSeq tool ({% cite Tillich_2017 %}) to annotate our sequence.
> * Upload the `fasta` file there. Information about how to use the tool is available on the page.
> * Once the annotation is completed, download the required files.
> * In Galaxy, import the annotation `GFF3` file.
{: .hands_on}

Now make a JBrowse file to view the annotations (the GFF3 file - produced from either Prokka or GeSeq) under the assembly (the polished-assembly.fasta file).

> <hands-on-title>View annotations</hands-on-title>
> 1. {% tool [JBrowse genome browser](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.4+galaxy3) %}:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - *"Select a reference genome"*: `polished-assembly.fasta`
>    - *"Output JBrowse"*: `Minimal for viewing (Documentation removed)`
>    - *"Genetic Code"*: `11. The Bacterial, Archaeal and Plant Plastid Code`
>    - *"JBrowse-in-Galaxy Action"*: `New JBrowse instance`
>    - *"Insert Track Group"*
>        - *"Insert Annotation Track"*
>            - *"Track Type"*: `GFF/GFF3/BED Features`
>            - *"GFF/GFF3/BED Track Data"*: the `GFF3` file
>            - *Leave the other track features as default*
> 2. **Re-name output file**:
>    - JBrowse may take a few minutes to run. There is one output file: re-name it `view-annotations`
> 3. **View output**:
>    * Click on the eye icon to view the annotations file.
>    * Select the right contig to view, in the drop down box.
>    * Zoom out (with the minus button) until annotations are visible.
{: .hands_on}

Here is an embedded snippet showing JBrowse and the annotations:

{% snippet topics/visualisation/faqs/visualizations_jbrowse.html datadir="data2" %}

# View reads

We will look at the original sequencing reads mapped to the genome assembly. In this tutorial, we will import very cut-down read sets so that they are easier to view.

> <hands-on-title>Import cut-down read sets</hands-on-title>
> 1. {% tool [Import](upload1) %} from [Zenodo](https://zenodo.org/record/3567224) or a data library (ask your instructor):
>   - FASTQ file with illumina reads: `sweet-potato-chloroplast-illumina-tiny.fastq`
>   - FASTQ file with nanopore reads: `sweet-potato-chloroplast-nanopore-tiny.fastq`
>   - Note: these are the "tiny" files, not the "reduced" files we imported earlier.
>    ```
>    https://zenodo.org/record/3567224/files/sweet-potato-chloroplast-illumina-tiny.fastq
>    https://zenodo.org/record/3567224/files/sweet-potato-chloroplast-nanopore-tiny.fastq
>    ```
{: .hands_on}

> <hands-on-title>Map the reads to the assembly</hands-on-title>
> * Map the Illumina reads (the new "tiny" dataset) to the `polished-assembly.fasta`, the same way we did before, using bwa mem.
> * This creates one output file: re-name it `illumina-tiny.bam`
> * Map the Nanopore reads (the new "tiny" dataset) to the `polished-assembly.fasta`. The settings will be the same, except `Select analysis mode` should be `Nanopore`
> * This creates one output file: re-name it `nanopore-tiny.bam`
{: .hands_on}

> <hands-on-title>Visualise mapped reads</hands-on-title>
> 1. {% tool [JBrowse genome browser](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.4+galaxy3) %}:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - *"Select a reference genome"*: `polished-assembly.fasta`
>    - *"Output JBrowse"*: `Minimal for viewing (Documentation removed)`
>    - *"Genetic Code"*: `11. The Bacterial, Archaeal and Plant Plastid Code`
>    - *"JBrowse-in-Galaxy Action"*: `New JBrowse instance`
>    - *"Insert Track Group"*
>        - *"Insert Annotation Track"*
>            - *"Track Type"*: `BAM pileups`
>            - *"BAM track data"*: `nanopore-tiny.bam`
>            - *"Autogenerate SNP track"*: `No`
>            - *Leave the other track features as default*
>        - *"Insert Annotation Track"*.
>            - *"Track Type"*: `BAM pileups`
>            - *"BAM track data"*: `illumina-tiny.bam`
>            - *"Autogenerate SNP track"*: `No`
>            - *Leave the other track features as default*
> 2. **Re-name output file**:
>    - JBrowse may take a few minutes to run. There is one output file: re-name it `assembly-and-reads`
> 3. **View output**:
>    * Click on the eye icon to view. (For more room, collapse Galaxy side menus with corner < > signs).
>    *  Make sure the bam files are ticked in the left hand panel.
>    * Choose a contig in the drop down menu. Zoom in and out with + and - buttons.
{: .hands_on}

Here is an embedded snippet showing JBrowse and the mapped reads:

{% snippet topics/visualisation/faqs/visualizations_jbrowse.html datadir="data" %}


> <question-title></question-title>
>
> 1. What are the differences between the nanopore and the illumina reads?
> 2. What are some reasons that the read coverage may vary across the reference genome?
>
> > <solution-title></solution-title>
> > 1. Nanopore reads are longer and have a higher error rate.
> > 2. There may be lots of reasons for varying read coverage. Some possibilities: In areas of high read coverage: this region may be a collapsed repeat. In areas of low or no coverage: this region may be difficult to sequence; or, this region may be a misassembly.
> {: .solution}
{: .question}

* To learn more about JBrowse and its features, see the [Genomic Data Visualisation with JBrowse tutorial]({{site.baseurl}}/topics/visualisation/tutorials/jbrowse/tutorial.html)


# Repeat with new data

*Optional extension exercise*

We can assemble another chloroplast genome using sequence data from a different plant species: the snow gum, *Eucalyptus pauciflora*. This data is from {% cite Wang_etal_2018 %}. It is a subset of the original FASTQ read files (Illumina - SRR7153063, Nanopore - SRR7153095).

> <hands-on-title>Assembly and annotation</hands-on-title>
> * Get data: at this [Zenodo link](https://doi.org/10.5281/zenodo.3600662), then upload to Galaxy.
> * Check reads: Run Nanoplot on the nanopore reads.
> * Assemble: Use Flye to assemble the nanopore reads, then get Fasta statistics *Note: this may take several hours.*
> * Polish assembly: Use Pilon to polish the assembly with short Illumina reads. *Note: Don't forget to map these Illumina reads to the assembly first using bwa-mem, then use the resulting `bam` file as input to Pilon.*
> * Annotate: Use the GeSeq tool at [Chlorobox](https://chlorobox.mpimp-golm.mpg.de/geseq.html) or the Prokka tool within Galaxy.
> * View annotations:Use JBrowse to view the assembled, annotated genome.
{: .hands_on}

# Conclusion


topic: assembly
tutorial: metagenomics-assembly
title: Assembly of metagenomic sequencing data
difficulty: introductory
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: ["assembly", "metagenomics", "microgalaxy"]
content_hash: 7aded2a0508a7f4b7e293f0144d5d647
last_modified: 2026-02-26T21:12:17.300219
zenodo_link: https://zenodo.org/record/7818827
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/metagenomics-assembly/tutorial.html

content:
Metagenomics involves the extraction, sequencing and analysis of combined genomic DNA from **entire microbiome** samples. It includes then DNA from **many different organisms**, with different taxonomic background.

Reconstructing the genomes of microorganisms in the sampled communities is critical step in analyzing metagenomic data. To do that, we can use **assembly** and assemblers, *i.e.* computational programs that stich together the small fragments of sequenced DNA produced by sequencing instruments.

Assembling seems intuitively similar to putting together a jigsaw puzzle. Essentially, it looks for reads "that work together" or more precisely, reads that overlap. Tasks like this are **not straightforward**, but rather complex because of the complexity of the genomics (specially the repeats), the missing pieces and the errors introduced during sequencing.

> <comment-title></comment-title>
>
> Do you want to learn more about the principles behind single genome assembly? Follow our [tutorials]({% link topics/assembly/index.md %}).
{: .comment}

**Metagenomic assembly** is further complicated by
- the large volume of data produced
- the quality of the sequence
- the unequal representation of members of the microbial community
- the presence of closely related microorganisms with similar genomes
- the presence of several strains of the same microorganism
- an insufficient amount of data for minor community members

> <details-title>Strategies for assembly</details-title>
> For assembly, there are 3 main strategies:
> 1. Greedy extension
> 2. Overlap Layout Consensus
> 3. De Bruijn graphs. The following figure illustrates these strategies in brief.
>
> ![Image shows greedy extention, overlap layout consensus, and de Brujin graphs assembly algorithms.](./images/assembly-algorithms.png "Assembly algorithms. Image from {% cite carpentries %}."){:width="70%"}
>
> The nice paper {% cite miller2010 %} on assemblers based on these algorithms will help you to better understand how they work.
{: .details}

For metagenomic assembly, several tools exist: metaSPAdes ({% cite nurk2017 %}), MEGAHIT ({% cite li2015 %}), etc. The different assemblers have different computational characteristics and their performance varies according to the microbiome as shown in by both rounds of  Critical Assessment of Metagenome Interpretation initiative ({% cite sczyrba2017 %}, {% cite meyer2022 %}, {% cite meyer2021 %}). The preference of one assembler over another depends on the purpose at hand.

In this tutorial, we will learn how to run metagenomic assembly tool and evaluate the quality of the generated assemblies. To do that, we will use data from the study: [Temporal shotgun metagenomic dissection of the coffee fermentation ecosystem](https://www.ebi.ac.uk/metagenomics/studies/MGYS00005630#overview). For an in-depth analysis of the structure and functions of the coffee microbiome, a temporal shotgun metagenomic study (six time points) was performed. The six samples have been sequenced with Illumina MiSeq utilizing whole genome sequencing.

Based on the 6 original dataset of the coffee fermentation system, we generated mock datasets for this tutorial.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare analysis history and data

To run assembly, we first need to get the data into Galaxy. Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Prepare the Galaxy history</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

We need to get the data into our history.

In case of a not very large dataset it's more convenient to upload data directly from your computer to Galaxy.

> <hands-on-title>Upload data into Galaxy</hands-on-title>
>
> 1. Import the sequence read raw data (\*.fastqsanger.gz) from [Zenodo]({{ page.zenodo_link }}) or a data library:
>
>    ```text
>    {{ page.zenodo_link }}/files/ERR2231567_1.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231567_2.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231568_1.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231568_2.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231569_1.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231569_2.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231570_1.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231570_2.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231571_1.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231571_2.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231572_1.fastqsanger.gz
>    {{ page.zenodo_link }}/files/ERR2231572_2.fastqsanger.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    > <comment-title></comment-title>
>    > In case of large dataset, we can use FTP server or the [Galaxy Rule-based Uploader]({% link topics/galaxy-interface/tutorials/upload-rules/tutorial.md %}).
>    {: .comment}
>
> 2. Create a paired collection named `Raw reads`, rename your pairs with the sample name
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
{: .hands_on}

# Assembly

As explained before, there are many challenges to metagenomics assembly, including:

1. Differences in coverage between samples, resulting from differences in abundance;
2. The fact that different species often share conserved regions ({%cite kececioglu2001%}), and
3. The presence of multiple strains of a single species ({%cite miller2010%}).

To reduce the differences in coverage between samples, we can use a **co-assembly** approach, where reads from all samples are aligned together:

![Image show one pile of sample1 reads and another pile of sample2 reads, then, green arrow leads to assembled reads from both piles](./images/co-assembly.png "Co-assembly"){:width="60%"}

Pros of co-assembly | Cons of co-assembly
--- | ---
More data | Higher computational overhead
Better/longer assemblies | Risk of shattering your assembly
Access to lower abundant organisms | Risk of increased contamination

Co-assembly is then not always beneficial:
- Changes in strain can cause the assembly graph to collapse
- Binned contigs are likely to be misclassified: MAGs must be treated as a population genome.

![Image shows the process of assembled reads from 2 samples followed by binning and there is detailed information about broken contigs, dominant coverage, chimeric contigs found during assembly process](./images/co-assembly-process.png "Co-assembly process"){:width="80%"}

In these cases, co-assembly is reasonable if:
- Same samples
- Same sampling event
- Longitudinal sampling of the same site
- Related samples

{% snippet faqs/galaxy/fastq_groupmerge.md %}

Examples where co-assembly would be reasonable:
- Repeated sampling of the **same patient** along a particular amount of time.
- Multiple samples taken from the **same site** and **similar environmental conditions**, eg. a patch of soil during the same sampling season.

Examples where co-assembly would NOT be recommended:
- Samples from different patients.
If samples differ as described, **individual assembly** is preferred. In the case of individual assembly, **contigs should be binned** per sample and an extra step of **de-replication** should be used as binning:

If samples differ as described, **individual assembly** is preferred. In the case of individual assembly, **contigs should be binned** per sample and an extra step of **de-replication** should be used after binning:


> <comment-title></comment-title>
> For more information on dereplication, check out the [metagenomic binning tutorial]({% link topics/microbiome/tutorials/metagenomics-binning/tutorial.md %}).
{: .comment}

In this tutorial, to show all steps, we will run an **individual assembly**.

> <comment-title>Why not both?</comment-title>
> It is also possible to run both individual assembly and co-assembly, and this approach can recover MAGs effectively. In this case: individual assembly can recover MAGs with a low amount of contamination, while co-assembly also allows for the recovery of low-abundance MAGs, with the downside of potentially more contamination. Although this approach can be effective, it also requires high computational resources and should be considered carefully.
>
> {% snippet faqs/galaxy/fastq_groupmerge.md %}
>
{: .comment}

As mentioned in the introduction, several tools are available for metagenomic assembly. But 2 are the most used ones:

- **MetaSPAdes** ({%cite nurk2017%}): an short-read assembler designed specifically for large and complex metagenomics datasets.

  MetaSPAdes is part of the SPAdes toolkit, which has several assembly pipelines. Since SPAdes handles non-uniform coverage, it is useful for assembling simple communities, but metaSPAdes also handles other problems, allowing it to assemble complex communities' metagenomes.

  As input for metaSPAdes it can accept short reads. However, there is an option to use additionally long reads besides short reads to produce hybrid input.

- **MEGAHIT** ({% cite li2015 %}): a single node assembler for large and complex metagenomics NGS reads, such as soil

  It makes use of the Succinct de Bruijn Graph (SdBG) approach to achieve low memory assembly.

Both tools are available in Galaxy. But currently, only **MEGAHIT** can be used in individual mode for several samples, as a option in the wrapper.
**MetaSPAdes** performs co-assembly by default if multiple samples are supplied. However, individually assembly can be enforced:

{% snippet faqs/galaxy/collections_force_mapping_over.md %}

# Assembly with MEGAHIT

> <hands-on-title>Individual assembly of short-reads with MEGAHIT</hands-on-title>
> 1.  {% tool [MEGAHIT](toolshed.g2.bx.psu.edu/repos/iuc/megahit/megahit/1.2.9+galaxy2) %} with parameters:
>     - *"Select your input option"*: `Paired-end collection`
>       - *"Run in batch mode?"*: `Run individually`
>
>          > <comment-title></comment-title>
>          > To run as co-assembly, select `Merge all fastq pair-end`, instead of `Run individually`
>          {: .comment}
>
>       - {% icon param-collection %} *"Select a paired collection"*: `Raw reads`
>     - In *Basic assembly options*
>       - *"K-mer specification method"*: `Specify min, max, and step values`
>         - *"Minimum kmer size"*: 21
>         - *"Maximum kmer size"*: 91
>         - *"Increment of kmer size of each iteration"*: 12
>
{: .hands_on}

**MEGAHIT** produced a collection of output assemblies - one per sample - that can be used for the subsequent step of **metagenomic binning**. The output contains **contigs**, contiguous lengths of genomic sequences in which bases are known to a high degree of certainty.

> <comment-title>Scaffolds</comment-title>
> Contrary to **MetaSPAdes**, **MEGAHIT** does not output **scaffolds**. **Scaffolds** are segments of genome sequence reconstructed fron contigs and gaps. The gaps occur when reads from the two sequenced ends of at least one fragment overlap with other reads from two different contigs (as long as the arrangement is otherwise consistent with the contigs being adjacent). It is possible to estimate the number of bases between contigs based on fragment lengths.
{: .comment}

> <comment-title></comment-title>
>
> Since the assembly process would take ~1h we are just going to import the results of the assembly previously run.
>
> > <hands-on-title>Import generated assembly files</hands-on-title>
> >
> > 1. Import the six contig files from [Zenodo]({{ page.zenodo_link }}) or the Shared Data library:
> >
> >    ```text
> >    {{ page.zenodo_link }}/files/contigs_ERR2231567.fasta
> >    {{ page.zenodo_link }}/files/contigs_ERR2231568.fasta
> >    {{ page.zenodo_link }}/files/contigs_ERR2231569.fasta
> >    {{ page.zenodo_link }}/files/contigs_ERR2231570.fasta
> >    {{ page.zenodo_link }}/files/contigs_ERR2231571.fasta
> >    {{ page.zenodo_link }}/files/contigs_ERR2231572.fasta
> >    ```
> >
> >
> > 2. Create a collection named `MEGAHIT Contigs`, rename your pairs with the sample name
> >
> {: .hands_on}
{: .comment}

> <question-title>Contig metrics</question-title>
>
> 1. How many contigs have been assembled for sample ERR2231568 ?
> 2. And how many have been assembled for ERR2231572?
> 3. What is the minimum length of the contigs?
>
> > <solution-title></solution-title>
> >
> > 1. There are 228,719 sequences in the file so 228,719 contigs
> > 2. 122,526 contigs
> > 3. Sequences seems bigger than 200 bp (`len` attribute of the sequence information in Fasta files). It is the default value set up in MEGAHIT.
> >
> {: .solution}
>
{: .question}

> <details-title>Co-assembly with MetaSPAdes</details-title>
> MetaSPAdes supports co-assembly by passing a list of paired-end read files.
>
> <hands-on-title>Assembly with MetaSPAdes</hands-on-title>
> 1. {% tool [MetaSPAdes](toolshed.g2.bx.psu.edu/repos/nml/metaspades/metaspades/4.2.0+galaxy0) %} with the following parameters:
>     - *"Pair-end reads input format"*: `Paired-end: list of dataset pairs`
>       - {% icon param-collection %} *"FASTQ file(s): collection"*: `Raw reads`
>
>       > <comment-title>Tip</comment-title>
>       > To run **individual assembly** instead, follow the previous FAQ to create a **nested list of samples**.
>       {: .comment}
>
>     - *"Select k-mer detection option"*: `User specific`
>       - *"K-mer size values"*: `21,33,55,77`
{: .hands_on}
{: .details}

# Quality control of assembly

Once assembly is done, it is important to check its quality.

Assemblies can be evaluated with **metaQUAST** ({%cite mikheenko2016%}), the metagenomics mode of **QUAST** ({%cite gurevich2013%}).

> <hands-on-title>Evaluation of assembly quality with metaQUAST</hands-on-title>
>
> 1. {% tool [QUAST](toolshed.g2.bx.psu.edu/repos/iuc/quast/quast/5.2.0+galaxy1) %} with parameters:
>     - *"Assembly mode?"*: `Individual assembly (1 contig file per samples)`
>       - *"Use customized names for the input files?"*: `No, use dataset names`
>         - {% icon param-collection %} *"Contigs/scaffolds file"*: output **MEGAHIT**
>       - *"Reads options"*: `Illumina paired-end reads in paired collection`
>
>       > <comment-title>Tip</comment-title>
>       > To speed up the processing, you may choose `Disabled` here. In this case, raw reads will **not** be mapped back to the assembly to compute metrics such as read coverage.
>       {: .comment}
>
>       - {% icon param-collection %} *"FASTQ/FASTA files"*: `Raw reads`
>     - *"Type of assembly"*: `Metagenome`
>     - *"Output files"*: `HTML report`, `PDF report`, `Tabular reports`, `Log file`, `Key metric summary (metagenome mode)`, `Krona charts (metagenome mode without reference genomes)`
>
> 2. Inspect the HTML reports:
> ![Screenshot of QUAST HTML report](./images/quast_html_report.png)
{: .hands_on}

> <hands-on-title>Import MetaQUAST results</hands-on-title>
>
> 1. Since the QUAST process would take times we are just going to import the results from [Zenodo]({{ page.zenodo_link }}):
>
>    ```text
>    {{ page.zenodo_link }}/files/quast_ERR2231567.html
>    {{ page.zenodo_link }}/files/quast_ERR2231568.html
>    {{ page.zenodo_link }}/files/quast_ERR2231569.html
>    {{ page.zenodo_link }}/files/quast_ERR2231570.html
>    {{ page.zenodo_link }}/files/quast_ERR2231571.html
>    {{ page.zenodo_link }}/files/quast_ERR2231572.html
>    ```
>
> {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}


QUAST main output are HTML reports which aggregate different metrics.

## Assembly statistics

On the top of each report is a table with in rows statistics for contigs larger than 500 bp for the different sample assemblies (columns). Let's now look at the table and go from top to bottom:

1. **Genome statistics**

    - **Genome fraction (%)**: percentage of aligned bases in the reference genome

      A base in the reference genome is counted as aligned if at least one contig has at least one alignment to this base.

      We did not provide any reference genome, but QUAST tries to identify the genome content of the metagenome by aligning contigs to [SILVA](https://www.arb-silva.de/) 16S rRNA database. For each assembly, 50 reference genomes with top scores are chosen. The full reference genomes of the identified organisms are downloaded from NCBI to map the assemblies on them and compute the genome fractions.

      > <comment-title>Metagenome reference</comment-title>
      > The alignment to automatically downloaded genomes for metagenomes is rather ambiguous and time-consuming. Most large-scale pipelines skip this step and set the **Maximum number of reference genomes (per each assembly) to download after searching in the SILVA database*** option to `0`.
      {: .comment}

      For each identified genomes, the genome fraction is given when clicking on **Genome fraction (%)**

      > <question-title></question-title>
      >
      > 1. What is the genome fraction for ERR2231568? And for ERR2231572?
      > 2. Which reference genome has the highest genome fraction for ERR2231568? And for ERR2231572?
      >
      > > <solution-title></solution-title>
      > >
      > > 1. The genome fraction is 29.429% for ERR2231568 and 60.463% for ERR2231572
      > > 2. The highest genome fraction was found for *Leuconostoc pseudomesenteroides* for ERR2231568 (84.2%) and for *Lactobacillus vaccinostercus* for ERR2231572 (92.2%). The genomes of *Leuconostoc pseudomesenteroides* and *Lactobacillus vaccinostercus* could be then almost completely recovered from the assemblies of ERR2231568 and ERR2231572 respectively.
      > >
      > {: .solution}
      >
      {: .question}

    - **Duplication ratio**: total number of aligned bases / genome fraction * reference length

      If an assembly contains many contigs that cover the same regions of the reference, the duplication ratio may be much larger than 1.

      > <question-title></question-title>
      >
      > 1. What is the duplication ratio for ERR2231568? And for ERR2231572?
      > 2. Which reference genome has the highest duplication ratio for ERR2231568? And for ERR2231572?
      >
      > > <solution-title></solution-title>
      > >
      > > 1. The duplication ratio is 1.062 for ERR2231568 and 1.112 for ERR2231572 (column ERR2231572 in ERR2231572 report)
      > > 2. The highest duplication ratio was found for *Gluconobacter oxydans* for ERR2231568 (1.156) and for *Lactobacillus pseudomesenteroides* for ERR2231572 (1.204).
      > >
      > {: .solution}
      >
      {: .question}

2. **Read mapping**: results of the mapping of the raw reads on the different assemblies (only if the *"Reads options"* is not disabled)

    > <question-title></question-title>
    >
    > 1. What is the % of read mapped for ERR2231568 assembly to ERR2231568 raw reads? And for ERR2231572 assembly to ERR2231572 raw reads?
    > 2. What is the percentage of reads used to build the assemblies for ERR2231568? and ERR2231572?
    >
    > > <solution-title></solution-title>
    > >
    > > 1. 79.47% of ERR2231568 raw reads were mapped to ERR2231568 assembly and 86.98% of ERR2231572 raw reads to ERR2231572 assembly.
    > > 2. 79.47% of reads were used to the assemblies for ERR2231568 and  86.98% for ERR2231572.
    > {: .solution}
    >
    {: .question}

    > <details-title>Alternative ways to compute coverage</details-title>
    >
    > There are 2 alternative ways to compute coverage:
    > 1. Use [CoverM](https://github.com/wwood/CoverM), which is available in Galaxy
    >
    >    > <hands-on-title>Calculate coverage using CoverM</hands-on-title>
    >    >
    >    > 1. {% tool [CoverM contig](toolshed.g2.bx.psu.edu/repos/iuc/coverm_contig/coverm_contig/0.7.0+galaxy0) %} with parameters:
    >    >    - *"Have the reads already been mapped to contigs?"*: `No`
    >    >       - *"Assembly mode?"*: `Individual assembly (1 contig file per sample)`
    >    >          - *"Read type"*: `Paired collection`
    >    >             - {% icon param-collection %} *"Collection of paired-end FASTA/Q files(s) for mapping"*: `Raw reads`
    >    >    - {% icon param-collection %} *"Contigs"*: output of MEGAGIT
    >    >
    >    > 2. Open the {% icon dataset-info %} "Dataset details" for ERR2231568
    >    > 3. Expand the `Tool Standard Error`
    >    {: .hands_on}
    >
    >    > <question-title></question-title>
    >    >
    >    > 1. What is the overall alignment rate for ERR2231568
    >    > 2. What is the percentage of reads used in assemblies for ERR2231568?
    >    > 3. Are the results similar to the ones in QUAST report?
    >    >
    >    > > <solution-title></solution-title>
    >    > >
    >    > > 1. The overall alignment rate for ERR2231567 is 75.48%
    >    > > 2. 75.48% of the reads were used in assemblies for ERR2231567 and 73.67% for ERR2231571.
    >    > > 3. Slighlty lower results that the ones found by QUAST (79.47% for ERR2231568)
    >    > >
    >    > {: .solution}
    >    >
    >    {: .question}
    >
    > 2. Map the original reads onto contigs and extract the percentage of mapped reads:
    >
    >    > <hands-on-title>Computation of the % reads used in assemblies</hands-on-title>
    >    >
    >    > 1. {% tool [Bowtie2](toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.3+galaxy1) %} with the following parameters:
    >    >    - *"Is this single or paired library"*: `Paired-end Dataset Collection`
    >    >      - {% icon param-collection %} *"FASTQ Paired Dataset"*: `Raw reads`
    >    >    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from the history and build index`
    >    >      - {% icon param-collection %} *"Select reference genome"*: MEGAHIT output
    >    >    - *"Save the Bowtie2 mapping statistics to the history"*: `Yes`
    >    > 3. Inspect the mapping statistics output
    >    {: .hands_on}
    >
    >    > <question-title></question-title>
    >    >
    >    > 1. What is the overall alignment rate for ERR2231568? and ERR2231572?
    >    > 2. What is the percentage of reads used in assemblies for ERR2231568? and ERR2231571?
    >    > 3. Are the results similar to the ones in QUAST report?
    >    >
    >    > > <solution-title></solution-title>
    >    > >
    >    > > 1. The overall alignment rate for ERR2231568 is 65.67% and 78.42% for ERR2231572.
    >    > > 2. 65.97% of the reads were used in assemblies for ERR2231568 and 78.42% for ERR2231572.
    >    > > 3. Lower results that the ones found by QUAST (79.47% for ERR2231568, 86.98%  for ERR2231572)
    >    > >
    >    > {: .solution}
    >    >
    >    {: .question}
    {: .details}

3. **Misassemblies**: joining sequences that should not be adjacent.

    QUAST identifies missassemblies by mapping the contigs to the reference genomes of the identified organisms. 3 types of misassemblies can be identified:

    ![Image shows on the top a contig with a blue and a gren parts with white arrows (pointing on the right) on them and below a reference with 2 chromosomes. The 3 types of misassemblies are after schematized. Relocation: the blue and gren parts of the contig are on chr 1 but separated. Inversion: the blue and gren parts of the contig are on chr 1 but separated and with the arrows facing each other. Translocation: the blue part is on chr 1 and gren part on chr 2.](./images/quast_misassemblies.png "Source: <a href="https://quast.sourceforge.net/docs/manual.html#sec3.1.2">QUAST manual</a>"){:width="60%"}

    1. **Relocation** occur based on signal from two mappings of the same contig against the same chromosome, with 2 cases:
        1. the 2 mappings are separated by an unmapped region of at least 1 kbp
        2. they map on the same chromosome with a shared mapping area of at least 1 kbp

        ![Image shows on the top a contig with a blue and a gren parts with white arrows (pointing on the right) on them and below a reference with 2 chromosomes. The 3 types of misassemblies are after schematized. Relocation: the blue and gren parts of the contig are on chr 1 but separated. Inversion: the blue and gren parts of the contig are on chr 1 but separated and with the arrows facing each other. Translocation: the blue part is on chr 1 and gren part on chr 2.](./images/relocation.svg "Source: <a href="https://blog.pierre.marijon.fr/misassemblies-in-noisy-assemblies/">Yet Another Bioinformatic blog by Pierre Marijon</a>"){:width="80%"}

        > <question-title></question-title>
        >
        > 1. How many relocations has been found for ERR2231568? And for ERR2231572?
        > 2. For which reference genomes are there the most relocation found for ERR2231568? And for ERR2231572?
        >
        > > <solution-title></solution-title>
        > >
        > > 1. 83 for ERR2231568 and 159 for ERR2231572
        > > 2. *Leuconostoc pseudomesenteroides* and *Tatumella morbirosei* for ERR2231568 and *Lactobacillus hordei* for ERR2231572
        > {: .solution}
        >
        {: .question}

    2. **Translocation** occur when a contig has mapped on more than one reference chromosomes

        > <question-title></question-title>
        >
        > 1. How many translocations has been found for ERR2231568? And for ERR2231572?
        > 2. For which reference genomes are there the most translocations found for ERR2231568? And for ERR2231572?
        > 3. What are the interspecies translocations?
        > 4. How many interspecies translocations has been found for ERR2231568? And for ERR2231572?
        >
        > > <solution-title></solution-title>
        > >
        > > 1. 30 for ERR2231568 and 90 for ERR2231572.
        > > 2. *Leuconostoc pseudomesenteroides* for ERR2231568 and *Lactobacillus vaccinostercus* for ERR2231572.
        > > 3. Interspecies translocations are translocations where a contig has mapped on different reference genomes.
        > > 4. 114 for ERR2231568 and 85 for ERR2231572.
        > {: .solution}
        >
        {: .question}

    3. **Inversion** occurs when a contig has two consecutive mappings on the same chromosome but in different strands

        > <question-title></question-title>
        >
        > 1. How many inversion has been found for ERR2231568? And for ERR2231572?
        > 2. For which reference genomes are there the most inversions found for ERR2231568? And for ERR2231572?
        >
        > > <solution-title></solution-title>
        > >
        > > 1. 4 for ERR2231568 and 9 for ERR2231572.
        > > 2. *Tatumella morbirosei* for ERR2231568 and *Lactobacillus plantarum* for ERR2231572.
        > {: .solution}
        >
        {: .question}

4. **Mismatches** or mismatched bases in the contig-reference alignment

    > <question-title></question-title>
    >
    > 1. How many mismatches have been identified for ERR2231568? And for ERR2231572?
    > 2. For which reference genomes are there the most mismatches for ERR2231568? And for ERR2231572?
    >
    > > <solution-title></solution-title>
    > >
    > > 1. 469,292 for ERR2231568 and 279,694 for ERR2231572.
    > > 2. *Pantoea rwandensis* for ERR2231568 and *Leuconostoc pseudomesenteroides* for ERR2231572.
    > {: .solution}
    >
    {: .question}

5. **Statistics without reference**

    - **# contigs**: total number of contigs

        > <question-title></question-title>
        >
        > 1. How many contigs are for ERR2231568? And for ERR2231572?
        > 2. How many sequences are in the output of MEGAHIT for ERR2231568? And for ERR2231572?
        > 3. Why are these numbers different from the number of sequences in the output of MEGAHIT?
        > 4. Which statistics in the metaQUAST report corresponds to number of sequences in the output of MEGAHIT?
        > 5. Which reference genomes have the most contigs ($$\geq$$ 500 bp) in ERR2231568? And in ERR2231572?
        >
        > > <solution-title></solution-title>
        > >
        > > 1. 66,434 contigs for ERR2231568 and 36,112 for ERR2231572.
        > > 2. In the outputs of MEGAHIT, there are 228,719 contigs for ERR2231568 and 122,526 contigs.
        > > 3. The numbers are lower in the metaQUAST results because metaQUAST reports there only the contigs longer than 500bp.
        > > 4. The **# contigs (>= 0 bp)**
        > > 5. Except the non aligned contigs, *Tatumella morbirosei* for ERR2231568 and *Leuconostoc brevis* for ERR2231572.
        > {: .solution}
        >
        {: .question}

    - **Largest contig**: length of the longest contig in the assembly

        > <question-title></question-title>
        >
        > 1. What is the length of the longest contig in ERR2231568? And in ERR2231572?
        > 2. Is the longest contig assigned to a reference genome in ERR2231568? And in ERR2231572?
        >
        > > <solution-title></solution-title>
        > >
        > > 1. 63,871 bp in ERR2231568 and 65,608 for ERR2231572.
        > > 2. It is assigned to *Leuconostoc pseudomesenteroides KCTC 3652* in ERR2231568 and not assigned (`not_aligned`) in ERR2231572.
        > {: .solution}
        >
        {: .question}

    - **N50**: length for which the collection of all contigs of that length or longer covers at least half an assembly

        N50 statistic defines assembly quality in terms of contiguity. If all contigs in an assembly are ordered by length, the N50 is the minimum length of contigs that contains 50% of the assembled bases. For example, an N50 of 10,000 bp means that 50% of the assembled bases are contained in contigs of at least 10,000 bp.

        Another example. Let's consider 9 contigs with the lengths 2, 3, 4, 5, 6, 7, 8, 9, and 10:
        - The sum of the length is 54
        - Half of the sum is 27
        - 10 + 9 + 8 = 27 (half the length of the sequence)
        - N50 = 8, i.e. the size of the contig which, along with the larger contigs, contain half of sequence of a particular genome

        > <question-title></question-title>
        >
        > 1. What is N50 for ERR2231568? And for ERR2231572?
        > 2. What is N90?
        >
        > > <solution-title></solution-title>
        > >
        > > 1. 921 for ERR2231568 and 1,233 for ERR2231572.
        > > 2. N90 is similar to the N50 metric but with 90% of of the sum of the lengths of all contigs
        > {: .solution}
        >
        {: .question}

        When comparing N50 values from different assemblies, the assembly sizes must be the same size in order for N50 to be meaningful.

        Also the N50 alone is not a useful measure to assess the quality of an assembly. For example, the assemblies with the following contig lengths:
        - 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 25, 25, 150, 1500
        - 50, 500, 530, 650

        Both assemblies have the same N50 although one is more contiguous than the other.

    - **L50**: number of contigs equal to or longer than N50

        In other words, L50, for example, is the minimal number of contigs that cover half the assembly.

        If we take the previous example in N50, L50 = 3.

        > <question-title></question-title>
        >
        > 1. What is the L50 for ERR2231568? And for ERR2231572?
        >
        > > <solution-title></solution-title>
        > >
        > > 1. 17,280 for ERR2231568 and 7,496 for ERR2231572.
        > {: .solution}
        >
        {: .question}

## Icarus contig browser

Icarus generates contig size viewer and one or more contig alignment viewers (if reference genome/genomes are provided) that are accessible from the HTML report, by clicking on `View on Icarus contig browser`.

### Contig size viewer

This viewer draws contigs ordered from longest to shortest. Let's inspect this viewer for ERR2231568.

> <question-title></question-title>
>
> Open the Contig size viewer for ERR2231568 and define start as `0` and end as `500000`
>
> ![Image shows on the Icarus Contig Size Viewer for ERR2231568, with a zoom between 0 500000. Below the menu, the contigs are drawn from the longest on the left to the shortest on the right. Each contig is filled with a different color: green for correct, red for misassembled, etc. Below the contigs is dispayed a bar to navigate through the contigs](./images/ERR2231568-contig-size-viewer.png)
>
> 1. What is the color of the first contig? Why?
> 2. What is the red contig?
>
> > <solution-title></solution-title>
> >
> > 1. The first contig is white because >50% of the contigs is unaligned. By clicking on the contig, we see that only a small block is aligned: 223.41 – 223.65 kbp to Leuconostoc_pseudomesenteroides_KCTC_3652_NZ_BMBP01000002.1.
> > 2. The red contig is a missamblied contig: it contains 2 blocks, with a translocation between them.
> {: .solution}
>
{: .question}

Click `Main menu` on the top left to go back to the main Icarus page.

### Contig alignment viewer

If a reference genome is provided, there should be a table on the main Icarus page that looks like:

Genome | # fragments | Length, bp | Mean genome fraction, % | # misassembled blocks
--- | --- | --- | --- | ---
Gluconobacter oxydans H24 | 2 | 3 816 232 | 10.989 | 38
Kosakonia cowanii | 5 | 4 806 998 | 28.224 | 60
Lactococcus lactis subsp. lactis CV56 | 6 | 2 518 737 | 18.940 	23

When clicking on the genome name, the contigs are displayed according to their mapping to the reference genome. The viewer can additionally visualize genes, operons, and read coverage distribution along the genome, if any of those were fed to QUAST.

> <question-title></question-title>
>
> Open the Contig alignment viewer for ERR2231568 and *Leuconostoc pseudomesenteroides* KCTC 3652, the most covered by contigs
>
> ![Image shows on the Icarus alignment size viewer of ERR2231568 contigs for Leuconostoc pseudomesenteroides KCTC 3652. On the top are displayed the contigs aligend on the reference genome, with a zoom. Each contig is filled with a different color: green for correct, red for misassembled, etc. Below the contigs is dispayed a bar to navigate through the contigs. Below is a graph representing the GC percentage and coverage along the reference genome](./images/ERR2231568-contig-alignment-viewer-leuconostoc.png)
>
> 1. How are the organized the contigs on the top?
> 2. What the different colors for the contigs?
> 3. Why is there a big red block on the right?
> 4. What is the graph on the bottom?
>
> > <solution-title></solution-title>
> >
> > 1. The contigs are displayed based on their mapping on the reference genome of *Leuconostoc pseudomesenteroides* KCTC 3652
> > 2. The different colors represent the different status for the contig: correct contigs, correct contigs but with >50% of the contig unaligned, misassembled blocks. unchecked misassembled blocks, ambiguously mapped contigs, alternative blocks of misassembled contigs, etc.
> > 3. The big red block on the right is contig k91_183138 with a misassembly on the left side, and overlap with 2 other contigs
> > 4. The graph on the bottom represents the GC percentage and coverage by contigs along the reference genome
> {: .solution}
>
{: .question}

> <details-title>Visualization of the *de novo* assembly graph</details-title>
>
> Current metagenome assemblers like MEGAHIT and MetaSPAdes use **graphs**, most typically a de Bruijn graph to stich reads together. In an ideal case, the graph would contain one distinct path for each genome of each micro-organisms, but complexities such as repeated sequences usually prevent this.
>
> Assembly graphs contain then **branching structures**: one node may lead into multiple others. **Contigs** correspond to the longest sequences in the graph that can be determined unambiguously. They are the final results of most assembler. But the assembly graph contains more information. It can be useful for finding sections of the graph, such as rRNA, or to try to find parts of a genome.
>
> **Bandage** ({% cite wick2015bandage %}) is a tool creating interactive visualisations of assembly graphs.
>
> > <hands-on-title>Visualization the assembly graph</hands-on-title>
> >
> > 1. {% tool [megahit contig2fastg](toolshed.g2.bx.psu.edu/repos/iuc/megahit_contig2fastg/megahit_contig2fastg/1.1.3+galaxy10) %} with parameters:
> >    - {% icon param-collection %} *"Contig file"*: Output of **MEGAHIT**
> >    - *"K-mer length"*: `91`
> >
> >      > <comment-title></comment-title>
> >      > To get the value, you need to
> >      > 1. Go into the **MEGAHIT** output collection
> >      > 2. Expand one of the contig file by clicking on it in the history
> >      > 3. Check in the dataset peek the name of the contig
> >      > 4. Extract the value after the first `k` in the contig names
> >      {: .comment}
> >
> > 2. {% tool [Bandage Image](toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_image/0.8.1+galaxy4) %} with parameters:
> >    - {% icon param-collection %} *"Graphical Fragment Assembly"*: Output of **megahit contig2fastg**
> >
> > 3. Inspect the generated image for ERR2231571
> {: .hands_on}
>
> ![Image shows the assembly graphs, with longer stretch on the top and many small contigs on the bottom](./images/ERR2231571_graph.jpg "Assembly graph for ERR2231571 sample")
>
> The graph is quite disconnected. On the top, we can see the longer stretches, that includes multiples contigs (each contig having a different color). On the bottom are the shortest stretches or single contigs.
>
> But it is really hard to read or extract any information from the graph. Let's inspect the information about the assembly graph
>
> > <hands-on-title>Visualization the *de novo* assembly graph</hands-on-title>
> >
> > 1. {% tool [Bandage Info](toolshed.g2.bx.psu.edu/repos/iuc/bandage/bandage_info/0.8.1+galaxy2) %} with parameters:
> >    - {% icon param-collection %} *"Graphical Fragment Assembly"*: Output of **megahit contig2fastg**
> >
> > 2. {% tool [Column join](toolshed.g2.bx.psu.edu/repos/iuc/collection_column_join/collection_column_join/0.0.3) %} with parameters:
> >    - {% icon param-collection %} *"Tabular files"*: Output of **Bandage Info**
> >
> > 3. Inspect the generated output
> {: .hands_on}
>
> > <question-title></question-title>
> >
> > 1. How many nodes are in the graph for ERR2231568? And for ERR2231572? What does they correspond to?
> > 2. How many edges are in the graph for ERR2231568? And for ERR2231572? What is the impact of these numbers in relation to the number of nodes on the graph?
> > 3. How many connected components are there for ERR2231568? And for ERR2231572? What does they correspond to?
> > 4. What is the percentage of dead ends are there for ERR2231568? And for ERR2231572?
> > 5. What are the smallest and larges edge overlaps?
> > 6. What is the largest component? For which sample?
> > 7. What is the shortest node? What does they correspond to?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. There are 228,719 nodes for ERR2231568 and 122,526 for ERR2231572. They correspond to the number of contigs
> > > 2. There are 16,580 edges for ERR2231568 and 13,993 for ERR2231572. There are less edges than nodes in the graph. It means that many nodes/contigs are disconnected
> > > 3. There are 212,598 connected components, i.e. number of regions of the graph which are disconnected from each other, for ERR2231568 and 109,044 for ERR2231572
> > > 4. There are 94.0702% dead ends, i.e. the end of a node not connected to any other nodes, for ERR2231568 and 90.7032% for ERR2231572. It confirms the previous observation
> > > 5. The smallest and larges edge overlaps are 91bp, i.e. the k-mer length
> > > 6. The largest component is 340,003 bp for ERR2231567
> > > 7. The shortest node is 200 bp, i.e. the minimal size for a contig
> > >
> > {: .solution}
> >
> {: .question}
{: .details}

# Conclusion

Metagenomic data can be assembled to, ideally, obtain the genomes of the species that are represented within the input data. But metagenomic assembly is **complex** and there are
- **different approaches** like de Bruijn graphs methods
- **different strategies**, such as co-assembly, when we assembly all samples together, and individual assembly, when we assembly samples one by one
- **different tools** like MetaSPAdes and MEGAHIT

Once the choices made, metagenomic assembly can start:
1. Input data are assembled to obtain contigs and sometimes scaffolds.
2. Assembly quality is evaluated with various metrics.
3. The assembly graph can be visualized.

Once all these steps done, we can move to the next phase to build Metagenomics Assembled Genomes (MAGs): [metagenomic binning]({% link topics/microbiome/tutorials/metagenomics-binning/tutorial.md %}).


topic: assembly
tutorial: unicycler-assembly
title: Unicycler Assembly
difficulty: introductory
time_estimation: 4h
tools_json: []
requirements_json: []
tags_json: ["prokaryote", "microgalaxy", "biodiversity"]
content_hash: 1b4a9aeab44fce4f0c0dc42e70fea01d
last_modified: 2026-02-26T21:12:14.494519
zenodo_link: https://doi.org/10.5281/zenodo.940733
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/unicycler-assembly/tutorial.html

content:
# The goal: *E. coli* C-1 assembly

In this tutorial we assemble and annotate the genome of *E. coli* strain [C-1](http://cgsc2.biology.yale.edu/Strain.php?ID=8232). This strain is routinely used in experimental evolution studies involving bacteriophages. For instance, now classic works by Holly Wichman and Jim Bull ({% cite Bull1997 %}, {% cite Bull1998 %}, {% cite Wichman1999 %}) have been performed using this strain and bacteriophage phiX174.

To sequence the genome we have obtained the strain from the [Yale E. coli Stock Center](http://cgsc2.biology.yale.edu/). The stock center sent us a filter paper disk infused with cells. The disk was placed in the center of an LB-agar plate. A single colony was picked and resuspended in a liquid LB medium, grown overnight, and genomic DNA was isolated. The DNA was then sequenced using two methods. To obtain high coverage, high accuracy data we used Illumina miSEQ to generated 250-bp paired end reads. To generate high length reads we used the Oxford Nanopore MinION machine.

Our goal is to reconstruct and annotate the full genome of *E. coli* C-1. As you will see in this tutorial a combination of many short, high accuracy reads with long, error-prone reads helps us produce an almost perfect assembly.

> ### Outline step-by-step
>
> In this tutorial, we will deal with:
>
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Background on data and tools


## The data

In this tutorial we will assemble a genome using two types of input data: (1) Illumina 250 bp paired-end reads and (2) Oxford Nanopore reads.

### Illumina data

We generated 9,345,897 250 bp read pairs (library preparation performed on genomic DNA fragmented to mean size of 600 bp). However, to make sure that you can complete this tutorial in a finite amount of time we have downsampled (reduced in size) to 1,000,000 paired end reads - just enough to produce an accurate assembly.

### Oxford Nanopore Data

There are 12,738 [2d-reads](http://www.nature.com/nmeth/journal/v12/n4/fig_tab/nmeth.3290_SF13.html). Maximum read length is 27,518 bp. The distribution of reads lengths looks like this:

![Nanopore read length distribution](../../images/ont_length.png "Distribution of nanopore read lengths.")

You can see that there many reads under the second peak with median of approximately 7.5 kb.

> <warning-title>Oxford Nanopore Data Format</warning-title>
> Oxford Nanopore machines output data in [fast5](http://bioinformatics.cvr.ac.uk/blog/exploring-the-fast5-format/) format that contains additional information besides sequence data. In this tutorial we assume that this data is *already* converted into [fastq](https://en.wikipedia.org/wiki/FASTQ_format). An additional tutorial dedicated to handling fast5 datasets will be developed shortly.
{: .warning}


## The tools

In this analysis we will perform two tasks: (1) assembly and (2) annotation. Below we will briefly outline the main ideas behind these two procedures and will describe the tools we will be using.

### Assembly

> <comment-title>Knowing your assembly</comment-title>
>
> Here we assume that you know a thing or two about assembly process. If you don't: look at the slides accompanying this tutorial as well as other tutorials is this section.
{: .comment}

![Logo unicycler](https://github.com/rrwick/Unicycler/raw/master/misc/logo.png)

For assembly we will be using [Unicycler](https://github.com/rrwick/Unicycler) (also see publication {% cite Wick2017 %}). Unicycler is designed specifically for *hybrid assembly* (that is, using both short- and long-read sequencing data) of small (e.g., bacterial, viral, organellar) genomes. In our hands it has produced complete high quality assemblies. Unicycler employs a multi-step process that utilizes a number of software tools:

![Unicycler process](../../images/unicycler.png "Simplified view of the Unicycler assembly process (From {% cite Wick2017 %}) In short, Unicycler uses SPAdes (see below) to produce an assembly graph, which is then bridged (simplified) using long reads to produce the longest possible set of contigs. These are then polished by aligning the original short reads against contigs and feeding these alignments to Pilon - an assembly improvement tool. {% cite Wick2017 %}")



As you can see Unicycler relies heavily on SPAdes ({% cite Bankevich2012 %}) and [Pilon](https://github.com/broadinstitute/pilon/wiki). We will briefly describe these two tools.

#### Spades

##### Multisized deBruijn graph

Assemblers usually construct graphs for *k*-mers of a fixed size. We have noted that when *k* is small it is difficult to resolve the repeats. If *k* is too large a corresponding graph may be fragmented (especially if read coverage is low). SPAdes uses several values for *k* (that are either manually set or inferred automatically) to create a *multisized* graph that minimized tangledness and fragmentation by combining various *k*-mers ({% cite Bankevich2012 %})):

![Multigraph approach implemented in SPAdes](../../images/multiGraph.jpg "Multisized de Bruijn graph. A circular Genome CATCAGATAGGA is covered by a set of Reads consisting of nine 4-mers, {ACAT, CATC, ATCA, TCAG, CAGA, AGAT, GATA, TAGG, GGAC}. Three out of 12 possible 4-mers from Genome are missing from Reads (namely {ATAG,AGGA,GACA}), but all 3-mers from the Genome are present in the Reads. (A) The outside circle shows a separate black edge for each 3-mer from Reads. Dotted red lines indicate vertices that will be glued. The inner circle shows the result of applying some of the glues. (B) The graph DB(Reads, 3) resulting from all the glues is tangled. The three h-paths of length 2 in this graph (shown in blue) correspond to h-reads ATAG, AGGA, and GACA. Thus Reads<sub>3,4</sub> contains all 4-mers from Genome. (C) The outside circle shows a separate edge for each of the nine 4-mer reads. The next inner circle shows the graph DB(Reads, 4), and the innermost circle represents the Genome. The graph DB(Reads, 4) is fragmented into 3 connected components. (D) The multisized de Bruijn graph DB (Reads, 3, 4). Figure and text from {% cite Bankevich2012 %}.")

##### Read pair utilization

While the use of paired reads and mate pairs is key to genome assembly, and not new, SPAdes utilizes so called paired DeBruin graphs to take the advantage of the paired end data. One of the key issues with paired DeBruin graphs is that the resulting genome assemblies do not tolerate variability in insert sizes: The initial formulation of paired DeBruijn graphs assumed constant distance between pairs of reads. In practice this distance is always variable. SPAdes performs *k*-bimer (these are *k*-mers derived from *paired* reads) adjustment to identify exact or nearly-exact distances for each *k*-bimer pair.

##### Error correction

Sequencing data contains a substantial number of sequencing errors that manifest themselves as deviations (bulges and non-connected components) within the assembly graph. One way to improve the graph before assembly it is to minimize the number of sequencing errors by performing error correction. SPAdes uses BayesHammer ({% cite Nikolenko2013 %}) to correct the reads. Here is a brief summary of what it does:

1. SPAdes (or rather BayesHammer) counts *k*-mers in reads and computes *k*-mer statistics that take into account base quality values.
2. A [Hamming graph](https://en.wikipedia.org/wiki/Hamming_graph) is constructed in which *k*-mers are nodes. In this graph edges connect nodes (*k*-mers) if they differ from each other by a number of nucleotides up to a certain threshold (the [Hamming distance](https://en.wikipedia.org/wiki/Hamming_distance)). The graph is central to the error correction algorithm.
3. Then Bayesian subclustering is done on the graph from the previous step. For each *k*-mer we now know the center of its subcluster.
4. **Solid** *k*-mers are derived from cluster centers and are assumed to be *error free*.
5. Solid *k*-mers are mapped back to the reads.
6. Reads are corrected using solid *k*-mers:

![Read correction with BayesHammer](../../images/readCorrection.jpg "Read correction. Black <em>k</em>-mers are solid. Grey <em>k</em>-mers are non-solid. Red <em>k</em>-mers are the centers of the corresponding clusters (two grey <em>k</em>-mers striked through on the right are non-solid singletons). As a result, one nucleotide is changed based on majority rule. (From {% cite Nikolenko2013 %})")

In the case of the full dataset, SPAdes error correction changed 14,013,757 bases in 3,382,337 reads - a substantial fraction of the full ~18 million read dataset.

#### Pilon

Pilon improves draft assemblies by using the information from the original reads aligned to the draft assembly. The following image from a publication by {% cite Walker2014 %} highlights the steps of this process:

![Pilon workflow](../../images/pilon.png "The left column depicts the conceptual steps of the Pilon process, and the center and right columns describe what Pilon does at each step while in assembly improvement and variant detection modes, respectively. During the first step (top row), Pilon scans the read alignments for evidence where the sequencing data disagree with the input genome and makes corrections to small errors and detects small variants. During the second step (second row), Pilon looks for coverage and alignment discrepancies to identify potential mis-assemblies and larger variants. Finally (bottom row), Pilon uses reads and mate pairs which are anchored to the flanks of discrepant regions and gaps in the input genome to reassemble the area, attempting to fill in the true sequence including large insertions. The resulting output is an improved assembly and/or a VCF file of variants. (From {% cite Walker2014 %})")

### Annotation

For annotation we are using [Prokka](https://vicbioinformatics.com/software.prokka.shtml) (also see {% cite Seemann2014 %}). It scans the assembly generated with Unicycler with a set of feature prediction tools and compiles a list of genome annotation. It predicts the following features (Table from {% cite Seemann2014 %}):

| Feature | Tool used by Prokka |
|---------|----------------------|
| Protein-coding sequences (CDS) | [Prodigal](https://github.com/hyattpd/Prodigal) |
| Ribosomal RNA genes | [RNAmmer](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1888812) |
| Transfer RNA genes | [Aragorn](https://www.ncbi.nlm.nih.gov/pubmed/14704338) |
| Signal leader peptides | [SignalP](https://www.ncbi.nlm.nih.gov/pubmed/21959131) |
| Non-coding RNA genes | [Infernal](http://eddylab.org/infernal/) |

Prokka predicts protein-coding regions using a two step process. It first identifies coordinates of putative genes using [Prodigal](https://github.com/hyattpd/Prodigal) and then compares the gene sequence against databases of known sequences at protein level using [Blast+](https://www.ncbi.nlm.nih.gov/books/NBK279690/) and [HMMer](http://hmmer.org/).

# Let's try it


## Load data and assess quality

In this example we will use a downsampled version of *E. coli* C-1 Illumina and ONT sequencing data. These include 3 files: forward and reverse reads for Illumina, and Long read file produced by ONT. All data are in [fastq](https://en.wikipedia.org/wiki/FASTQ_format) format.

> <hands-on-title>Obtaining our data</hands-on-title>
>
> 1. Make sure you have an empty analysis history. Give it a name.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the following file from [Zenodo](https://zenodo.org/record/940733)
>
>    ```
>    https://zenodo.org/record/940733/files/illumina_f.fq
>    https://zenodo.org/record/940733/files/illumina_r.fq
>    https://zenodo.org/record/940733/files/minion_2d.fq
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

If all goes well you will see datasets uploading and changing states from gray to green as shown below. The figure below also shows how datasets can be tagged.

![Datasets in History](../../images/starting_data.png  "Sequencing data loaded into Galaxy history. The full progression from gray (scheduling) to green (all OK) state is shown. To make it easier to identify datasets as we progress through the analysis we use so-called <em>Hashtags</em>. To tag a dataset: click on dataset to expand it (as shown in panel four); click the tag icon (<i class='fa fa-tags' aria-hidden='true'></i>) and a text field will appear. Add a tag (in this case <b>F</b>) pre-pended with hash (#). Hit enter. Do this for all three datasets and it will appear as in panel five.")

### Assess Read Quality

To assess quality we will use two tools: FastQC ({% cite FastQC %}) to generate quality statistics and multiQC ({% cite Ewels2016 %}) to summarize these statistics.

> <hands-on-title>Quality Control</hands-on-title>
>
> 1. **FastQC** {% icon tool %}:
>
>    - {% icon param-files %} *"Short read data from your current history"*: Select all three FastQ datasets simultaneously
>
> 2. **MultiQC** {% icon tool %}: to generate a summary of the FastQC reports with
>   - *"Which tool was used generate logs?"*: `FastQC`
>   - *"Type of FastQC output"*: `Raw data`
>   - {% icon param-files %} *"FastQC Output"*: `RawData` outputs of FastQC
>
{: .hands_on}

A quick look at quality score distribution will show a confusing picture:

![QC reported zoomed out](../../images/multiqc1.png "Because Illumina reads (green) are <b>much</b> shorter that ONT reads (red) the plot looks strange. ONT reads generally have low quality scores and so they are not really meaningful in the context of this technology. However, in the case of Illumina data they mean a lot...")

So let's zoom in into Illumina data:

![QC reported zoomed in](../../images/multiqc2.png "Zooming in shows quality distribution for Illumina reads. This is excellent data with mean base qualities above 30 across all reads.")

## Assembly with Unicycler

Now it is time to perform assembly.

> <hands-on-title>Unicycler Assembly</hands-on-title>
>
> 1. **Unicycler** {% icon tool %} with the following parameters :
>   - "Paired or Single end data?" to `Paired`
>   - "First Set of reads" to the forward reads file `f`
>   - "Second Set of reads" to the reverse reads file `r`
>   - "Long reads" to the MinION file
>   - Use default parameters
>
{: .hands_on}


> ### <i class="fa fa-cutlery" aria-hidden="true"></i> <i class="fa fa-coffee" aria-hidden="true"></i> Assembly takes time!
>
> There is no such thing as Assembly in real time. It takes time so it is a good time to have lunch or at least coffee. This Unicycler run will take anywhere between 90 minutes and two hours.
{: .warning}

## Assess Assembly quality with Quast

Quast ({% cite Gurevich2013 %}) is a tool providing quality metrics for assemblies, and can also be used to compare multiple assemblies. The tool can also take an optional reference file as input, and will provide complementary metrics.

> <hands-on-title>Assembly Quality</hands-on-title>
>
> 1. **Quast** {% icon tool %}: with the following parameters
>   - *"Contigs/scaffolds output file"*: Select the fasta file resulting from the Unicycler assembly.
>
{: .hands_on}

The Quast tool outputs assembly metrics as an html file with metrics and graphs. The image below looks exceptionally boring. This is a **good** thing:

![Quast Interface](../../images/quast_output.png  "Quast Output: Quast provides different statistics such as the number of contigs or scaffolds, the N50 and N75, and the total length of the assembly. You can also access 3 plots, the cumulative length of the contigs, the Nx, or the GC content.")

One can see that there are two (!) contigs. The largest contig is 4,576,290 bp (for comparison *E. coli* K12 MG1655 strain genome length is [4,656,144 bp](https://www.ncbi.nlm.nih.gov/nuccore/NZ_APIN00000000.1)) and the smallest is 4,581,676 (total length) - 4,576,290 (length of the largest) = 5,386 bp. When we analyzed this dataset for the first time we were initially puzzled by this second contig. But we quickly realized that this is simply the genome of bacteriophage [phiX174](https://www.ncbi.nlm.nih.gov/nuccore/NC_001422.1) which is routinely used as a spike-in in Illumina sequencing. Thus we have two genomes: the one of *E.coli* C-1 and phiX174! We can now use Prokka to annotate our two genomes.

## Annotation with Prokka


> <hands-on-title>Annotation</hands-on-title>
>
> 1. **Prokka** {% icon tool %}:
>   - *"Contigs to annotate"*: Select the assembly ouput of `Unicycler`
>   - *"Genus name"*: `Escherichia`
>   - *"Species name"*: `coli`
>   - *"Strain name"*: `C-1`
>   - *"Use genus-specific BLAST database"*: `yes`
>
{: .hands_on}

Prokka outputs 10 datasets (including two log files). These are in various formats:

* **txt** : Provides Statistics on the annotation : number of CDS predicted, number of rRNA etc.
* **tbl** : Provides a tabulated list of annotated features.
* **fsa** : Nucleotide fasta file of the input contig sequence.
* **sqn** : [ASN.1](https://www.ncbi.nlm.nih.gov/Structure/asn1.html) format file for submission to GenBank.
* **ffn** : Nucleotide FASTA file of all the prediction transcripts.
* **faa** : Protein FASTA file of the translated CDS sequences.
* **fna** : Nucleotide fasta file of the input contig sequence.
* **gbk** : [GenBank](https://www.ncbi.nlm.nih.gov/Sitemap/samplerecord.html) file.
* **gff** : [gff3](http://gmod.org/wiki/GFF3) file.

## Visualize the results in IGV

Let's look at the entire assembly and its annotation in the genome browser. We can do this using Integrated Genome Browser ([IGV](http://software.broadinstitute.org/software/igv/)).

Visualization requires a local installation of IGV. If you have IGV installed - just start it. If you don't - read on.

### Starting IGV

Go to IGV [download page](http://software.broadinstitute.org/software/igv/download) and select one of the options. The one I would try first would be **Java Web Start**. Simply click the **Launch** button for 10 GB distribution.


> <hands-on-title>Visualization in IGV</hands-on-title>
>
>
> 1. Start IGV. It will look something like this:
>
>      ![IGV just started](../../images/igv1.png "IGV started by using Java Web Start. Note that it is currently showing Human Genome (hg38) build. This is obviously <b>not</b> what we want.")
>
> 2. Locate the output of Unicycler and expand it :
>
>      ![Unicycler result Visualization](../../images/unicycler_igv.png  "Expanding Unicycler assembly reveals IGV link.")
>
> 3. Click on the *local* link highlighted with orange outline. The browser will change:
>
>      ![IGV with Unicycler assembly](../../images/igv2.png "Now you can see the major contig shown in the browser window.")
>
> 4. Let's add Prokka annotations to the browser image. For this simply expand Prokka's GFF3 dataset and click on the *local* link:
>
>      ![Expanded GFF3 dataset representing Prokka annotations](../../images/prokka_item.png "Expanded GFF dataset generated with Prokka. Click on the <em>local</em> link (highlighted with orange outline) to display this dataset within IGV.")
>
{: .hands_on}

You will now see the annotations within the browser window:


![Prokka result Visualization](../../images/igv3.png  "Viewing annotated assembly withing IGV.")


topic: assembly
tutorial: general-introduction
title: An Introduction to Genome Assembly
difficulty: introductory
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 00ecc8da31dabd327e2164f1d40198fd
last_modified: 2026-02-26T21:12:14.463442
zenodo_link: https://doi.org/10.5281/zenodo.582600
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/general-introduction/tutorial.html

content:
# Genome assembly with Velvet: Background


Velvet is one of a number of *de novo* assemblers that use short read sets as input (*e.g.* Illumina Reads). The assembly method is based on the manipulation of de Bruijn graphs, via the removal of errors and the simplification of repeated regions.

> <comment-title></comment-title>
>
> For information about Velvet, you can check its (nice) [Wikipedia page](https://en.wikipedia.org/wiki/Velvet_assembler).
{: .comment}

For this tutorial, we have a set of reads from an imaginary *Staphylococcus aureus* bacterium with a miniature genome (197,394 bp). Our mutant strain read set was sequenced with the whole genome shotgun method, using an Illumina DNA sequencing instrument. From these reads, we would like to rebuild our imaginary *Staphylococcus aureus* bacterium via a *de novo* assembly of a short read set using the Velvet assembler.

We also have a sequence for a reference genome that we will use later in the tutorial.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get the data

> <hands-on-title>Getting the data</hands-on-title>
>
> 1. Create and name a new history for this tutorial.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import from [Zenodo](https://doi.org/10.5281/zenodo.582600) or from the data library the files:
>
>    ```
>    https://zenodo.org/record/582600/files/mutant_R1.fastq
>    https://zenodo.org/record/582600/files/mutant_R2.fastq
>    https://zenodo.org/record/582600/files/wildtype.fna
>    ```
>
>    > <tip-title>Importing data via links</tip-title>
>    >
>    > * Copy the link location (Right-click on the filename then "Copy Link Address")
>    > * Open the Galaxy Upload Manager
>    > * Select **Paste/Fetch Data**
>    > * Paste the link into the text field
>    > * For the read files, change the data-type to **fastqsanger**
>    > * Press **Start**
>    {: .tip}
>
> 3. Change the name of the files to `mutant_R1`, `mutant_R2` and `wildtype.fna`.
>
>    As a default, Galaxy uses the link as the name of the new dataset. It also does not link the dataset to a database or a reference genome.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Inspect the content of a file.
>
>    > <tip-title>Inspecting the content of a dataset</tip-title>
>    >
>    > * Click on the {% icon galaxy-eye %} (eye) icon next to the relevant history entry
>    > * View the content of the file in the central panel
>    {: .tip}
>
>    > <question-title></question-title>
>    >
>    > 1. What are four key features of a FASTQ file?
>    > 2. What is the main difference between a FASTQ and a FASTA file?
>    >
>    > > <solution-title></solution-title>
>    > > 1. Each sequence in a FASTQ file is represented by 4 lines: 1st line is the id, 2nd line is the sequence, 3rd line is not used, and 4th line is the quality of sequencing per nucleotide
>    > > 2. In a FASTQ file, not only are the sequences present, but information about the quality of sequencing is also included.
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

The reads have been sequenced from an imaginary *Staphylococcus aureus* bacterium using an Illumina DNA sequencing instrument. We obtained the 2 files we imported (`mutant_R1` and `mutant_R2`)

> <question-title></question-title>
>
> Why do we have 2 files here if we only sequenced the bacteria once?
>
> > <solution-title></solution-title>
> > 1. The bacteria has been sequenced using paired-end sequencing. The first file corresponds to forward reads and the second file to reverse reads.
> {: .solution }
{: .question}

# Evaluate the input reads

Before doing any assembly, the first questions you should ask about your input reads include:

- What is the coverage of my genome?
- How good is my read set?
- Do I need to ask for a new sequencing run?
- Is it suitable for the analysis I need to do?

We will evaluate the input reads using the FastQC tool. This tool runs a standard series of tests on your read set and returns a relatively easy-to-interpret report. We will use it to evaluate the quality of our FASTQ files and combine the results with MultiQC.

> <hands-on-title>FastQC on a fastq file</hands-on-title>
>
> 1. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0) %} with the following parameters:
>    - {% icon param-files %} *"Raw read data from your current history"*:  `mutant_R1.fastq` and `mutant_R2.fastq`
>
>    {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>  
> 2. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy1) %} with the following parameters:
>    - *"Results: Which tool was used to generate logs?"*: `FastQC`
>    - *Click "Insert FastQC output"*
>    - *"Type of FastQC output?"*: `multiple datasets, select the raw data files from FastQC`
{: .hands_on}

MultiQC generates a webpage combining reports for FastQC on both datasets. It includes these graphs and tables:

- General statistics

    We need to know about the data for our analysis. In particular, we need to know the read lengths as it is important in setting the maximum k-mer size for an assembly.

    > <comment-title>Getting the length of sequences</comment-title>
    >
    > * Find the MultiQC output that is a webpage and click to view
    > * The first table shows General Statistics for the input read files
    > * At the top of this table, click on **Configure Columns**
    > * Make sure the box next to **Length** is ticked
    > * Close the window
    > * This table should now show a column for read lengths
    {: .comment}

    > <question-title></question-title>
    >
    > 1. How long are the sequences?
    > 2. What is the average coverage of the genome, given our imaginary *Staphylococcus aureus* bacterium has a genome of 197,394 bp?
    >
    > > <solution-title></solution-title>
    > > 1. The sequences are 150 bp long
    > > 2. We have 2 x 12,480 sequences of 150 bp, so the average genome coverage is: 2 * 12480 * 150 / 197394, or approximately 19 X coverage.
    > {: .solution }
    {: .question}

- Sequence Quality Histograms

    Dips in quality near the beginning, middle or end of the reads may determine the trimming/cleanup methods and parameters to be used, or may indicate technical problems with the sequencing process/machine run.

    ![Sequence Quality Histograms with the mean quality value across each base position in the read](../../images/fastqc_per_base_sequence_quality_plot.png "The mean quality value across each base position in the read")

    > <question-title></question-title>
    >
    > 1. What does the y-axis represent?
    > 2. Why is the quality score decreasing across the length of the reads?
    >
    > > <solution-title></solution-title>
    > > 1. The y-axis represents the quality score for each base (an estimate of the error during sequencing).
    > > 2. The quality score is decreasing accross the length of the reads because the sequencing become less and less reliable at the end of the reads.
    > {: .solution }
    {: .question}

- Per Sequence GC Content

    High GC organisms tend not to assemble well and may have an uneven read coverage distribution.

- Per Base N Content

    The presence of large numbers of Ns in reads may point to a poor quality sequencing run. You will need to trim these reads to remove Ns.

- k-mer content

    The presence of highly recurring k-mers may point to contamination of reads with barcodes or adapter sequences.


> <comment-title></comment-title>
>
> For a fuller discussion of FastQC outputs and warnings, see the [FastQC website link](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/), including the section on each of the output [reports](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/3%20Analysis%20Modules/), and examples of ["good"](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/good_sequence_short_fastqc.html) and ["bad"](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/bad_sequence_fastqc.html) Illumina data.
{: .comment}

We won't be doing anything to these data to clean it up as there isn't much need. Therefore we will get on with the assembly!


# Assemble reads with Velvet

Now, we want to assemble our reads to find the sequence of our imaginary *Staphylococcus aureus* bacterium. We will perform a *de novo* assembly of the reads into long contiguous sequences using the Velvet short read assembler.

The first step of the assembler is to build a de Bruijn graph. For that, it will break our reads into k-mers, *i.e.* fragments of length *k*. Velvet requires the user to input a value of *k* (k-mer size) for the assembly process. Small k-mers will give greater connectivity, but large k-mers will give better specificity.

> <hands-on-title>Assemble the reads</hands-on-title>
>
> 1. {% tool [FASTQ interlacer](toolshed.g2.bx.psu.edu/repos/devteam/fastq_paired_end_interlacer/fastq_paired_end_interlacer/1.2.0.1+galaxy0) %} with the following parameters:
>    - *"Type of paired-end datasets"*: `2 separate datasets`
>    - *"Left-hand mates"*: `mutant_R1.fastq`
>    - *"Right-hand mates"*: `mutant_R2.fastq`
>
>    Currently our paired-end reads are in 2 files (one with the forward reads and one with the reverse reads), but Velvet requires only one file, where each read is next to its mate read. In other words, if the reads are indexed from 0, then reads 0 and 1 are paired, 2 and 3, 4 and 5, etc. Before doing the assembly *per se*, we need to prepare the files by combining them.
>
> 2. {% tool [velveth](toolshed.g2.bx.psu.edu/repos/devteam/velvet/velveth/1.2.10.3) %} with the following parameters:
>    - *"Hash Length"*: `29`
>    - *"Input Files"*
>       - Click on {% icon param-repeat %} *"Input Files"*
>       - In "1: Input Files"
>          - *"Choose the input type"*: `interleaved paired end`
>          - *"read type"*: `shortPaired reads`
>          - {% icon param-files %} *"Dataset"*:  pairs output of **FASTQ interlacer** 
>
>    The tool takes our reads and break them into k-mers.
>
> 3. {% tool [velvetg](toolshed.g2.bx.psu.edu/repos/devteam/velvet/velvetg/1.2.10.2) %} with the following parameters:
>    - {% icon param-files %} *"Velvet Dataset"*:  outputs of **velveth**
>    - *"Using Paired Reads"*: `Yes`
>
>    This last tool actually does the assembly.
{: .hands_on}

Five files are generated. We will look at the contigs file and the stats file:

- The **Contigs** file

    This file contains the sequences of the contigs. In the header of each contig, a bit of information is added:
    - the k-mer length (called "length"): For the value of k chosen in the assembly, a measure of how many k-mers overlap (by 1 bp each overlap) to give this length
    - the k-mer coverage (called "coverage"): For the value of k chosen in the assembly, a measure of how many k-mers overlap each base position (in the assembly).
    - Note that your results may look different to the example in the image below. 

    ![Contigs output](../../images/image10.png)

- The **Stats** file

    This is a tabular file giving for each contig the k-mer lengths, k-mer coverages and other measures.  Note that your results may look different to the example in the image below. 

    ![Contigs stats output](../../images/image11.png)

# Collect some statistics on the contigs

> <question-title></question-title>
>
> 1. How many contigs have been built?
> 2. What is the mean, min and max length of the contigs?
>
> > <solution-title></solution-title>
> > 1. 306
> > 2. To compute this information, we can use the Datamash tool on the 2nd columns (length). Be careful with the first line, the header. As a result, we obtain approximately: 516 as mean, 1 as min and 8836 as max. It would mean that the smallest contig has a length of 1 bp, even smaller than k. The length on the 2nd column corresponds to length of the contig in k-mers. This means that the smallest contig has a length of 1k = 29. So to obtain the real length, we need to add k-1 to the length. We then obtain a mean contig length of 544 bp, a min contig of 29 bp and a max contig of 8864 bp.
> {: .solution }
{: .question}

This table is limitted, but we will now collect more basic statistics on our assembly.

> <hands-on-title>Collect fasta statistics on our contigs</hands-on-title>
>
> 1. {% tool [Quast](toolshed.g2.bx.psu.edu/repos/iuc/quast/quast/5.2.0+galaxy1) %} with the following parameters:
>    - *"Assembly mode"*: `Individual assembly (1 contig file per sample)`
>    - *"Use customized names?"*: `No`
>    - *"Contigs/scaffolds file"*: contigs output of **velvetg**
>    - *"Type of assembly"*: `Genome`
>    - *"Use a reference genome?"*: `Yes`
>    - *"Reference genome"*: `wildtype.fna`
>    - *"Type of organism"*: `Prokaryotes`
>    - *"Lower Threshold"*: `500`
>    - *"Advanced options: Comma-separated list of contig length thresholds"*: `0,1000`
{: .hands_on}


> <question-title></question-title>
>
> 1. What is represented in the Icarus viewer?
>
> > <solution-title></solution-title>
> > 1. Icarus is a novel genome visualizer for accurate assessment and analysis of genomic draft assemblies. It draws contigs ordered from longest to shortest, highlights N50, N75 (NG50, NG75) and long contigs larger than a user-specified threshold
> {: .solution }
{: .question}

The HTML report reports many statistics computed by QUAST to assess the quality of the assembly:

- Statistics about the quality of the assembly when compared to the reference (fraction of the genome, duplication ratio, etc)
- Misassembly statistics, including the number of misassemblies

    A misassembly is a position in the contigs (breakpoints) that satisfy one of the following criteria:
    - the left flanking sequence aligns over 1 kbp away from the right flanking sequence on the reference;
    - flanking sequences overlap on more than 1 kbp
    - flanking sequences align to different strands or different chromosomes

- Unaligned regions in the assembly
- Mismatches compared to the reference genomes
- Statistics about the assembly *per se*, such as the number of contigs and the length of the largest contig

> <question-title></question-title>
>
> 1. How many contigs have been constructed?
> 2. Which proportion of the reference genome do they represent?
> 3. How many misassemblies have been found?
> 4. Has the assembly introduced mismatches and indels?
> 5. What are N50 and L50?
> 6. Is there a bias in GC percentage induced by the assembly?
>
> > <solution-title></solution-title>
> > 1. 190 contigs have been constructed, but only 47 have a length > 500 bp.
> > 2. The contigs represents 87.965% of the reference genome.
> > 3. 1 misassembly has been found: it corresponds to a relocation, *i.e.* a misassembly event (breakpoint) where the left flanking sequence aligns over 1 kbp away from the right flanking sequence on the reference genome.
> > 4. 8.06 mismatches per 100 kbp and 4.03 indels per 100 kbp are found.
> > 5. N50 is the length for which the collection of all contigs of that length or longer covers at least half an assembly. In other words, if contigs were ordered from small to large, half of all the nucleotides will be in contigs this size or larger. And L50 is the number of contigs equal to or longer than N50: L50 is the minimal number of contigs that cover half the assembly.
> > 6. The GC % in the assembly is 33.64%, really similar to the one of the reference genome (33.43%).
> {: .solution }
{: .question}

# Discussion

> <hands-on-title>(Optional) Rerun for values <i>k</i> ranging from 31 to 101</hands-on-title>
>
> 1. {% tool [velveth](toolshed.g2.bx.psu.edu/repos/devteam/velvet/velveth/1.2.10.3) %} with the same parameters as before except
>    - *"Hash Length"*: a value between 31 and 101
> 2. {% tool [velvetg](toolshed.g2.bx.psu.edu/repos/devteam/velvet/velvetg/1.2.10.2) %} with the same parameters as before
> 3. {% tool [Quast](toolshed.g2.bx.psu.edu/repos/iuc/quast/quast/5.2.0+galaxy1) %} with the same parameters as before
{: .hands_on}

We have completed an assembly on this data set for a number of k values ranging from 29 to 101. A few of the assembly metrics appear below.

![contigs](../../images/number_of_contigs.png "Number of contigs in the assembly for various k-mer sizes")

![largest_contig](../../images/largest_contig.png "Largest contig in each of the assemblies by k-mer size")

![total_bp](../../images/total_bp.png "Total number of base pairs in all the contigs for each assembly by k-mer size")

![n50](../../images/n50.png "N50 metric for each of the assemblies by k-mer size")

> <question-title></question-title>
>
> 1. Are there any distinct features in the charts?
> 2. Does it look like one assembly might be better than some of the others?
>
{: .question}

The reasons for these patterns will be discussed in detail in the De Bruijn graph assembly slides and tutorial.


topic: assembly
tutorial: ecoli_comparison
title: Making sense of a newly assembled genome
difficulty: intermediate
time_estimation: 4h
tools_json: []
requirements_json: []
tags_json: ["gmod", "prokaryote", "microgalaxy", "jbrowse1"]
content_hash: d97667fd8dda5da56e7e0232205885ad
last_modified: 2026-02-26T21:12:14.460880
zenodo_link: https://zenodo.org/record/3382053
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/ecoli_comparison/tutorial.html

content:
In this tutorial we begin with a new genome assembly just produced in the [Unicycler tutorial]({{site.baseurl}}/topics/assembly/tutorials/unicycler-assembly/tutorial.html). This is an assembly of *E. coli* C, which we will be comparing to assemblies of all other complete genes of this species.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Finding closely related genomes

[*E. coli*](https://en.wikipedia.org/wiki/Escherichia_coli) is one of the most studied organisms. There are thousands of complete genomes (in fact, the total number of *E. coli* assemblies in Genbank is over 10,500). Here we will shows how to uploaded all (!) complete *E. coli* genomes at once.

> <comment-title>Slow steps ahead</comment-title>
> The first part of this tutorial can take a significant amount of time to find the most related genomes. If you want, you can upload this (outdated) copy of the NCBI E. Coli Genomes table to your history:
>
> 1. Import the following URL:
>
>    ```
>    https://zenodo.org/record/3382053/files/genomes_proks.txt
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 2. And skip ahead to [comparing the most related genomes](#comparing-genome-architectures).
{: .comment}


## Getting complete *E. coli* genomes into Galaxy

Our initial objective is to compare our assembly against all complete *E. coli* genomes to identify the most related ones and to find any interesting genome alterations. In order to do this we need to align our assembly against all other genomes. And in order to do that we need to first obtain all these other genomes.

[NCBI](https://www.ncbi.nlm.nih.gov/) is the resource that would store [all complete *E. coli* genomes](https://www.ncbi.nlm.nih.gov/genome/genomes/167). This list contains over 500 genomes and so uploading them by hand will likely result in carpal tunnel syndrome, which we want to prevent. Galaxy has several features that are specifically designed for uploading and managing large sets of similar types of data. The following two **Hands-on** sections show how they can be used to import all completed *E. coli* genomes into Galaxy.

> <hands-on-title>Preparing a list of all complete <i>E. coli</i> genomes</hands-on-title>
>
> 1. Import the genome list from Zenodo:
>
>    ```
>    https://zenodo.org/record/3382053/files/genomes_proks.txt
>    ```
>
{: .hands_on}


> <details-title>Getting the data directly from NCBI</details-title>
>
> For this tutorial we made this dataset available from Zenodo, but it is of course also possible to obtain the data directly from NCBI.
> Note that the format of the files on NCBI may change, which means some of the parameter settings of tools in this tutorial will need
> to be altered (e.g. column numbers) when using data directly from NCBI.
>
> Below we describe how you could obtain this data from NCBI.
>
> 1. Open [the NCBI list of of *E. coli* genomes](https://www.ncbi.nlm.nih.gov/genome/genomes/167) in a new window
>
> 2. Click on "Filters" at the top right:
>
>    ![Filter menu button](../../images/ecoli-list-filter.png)
>
> 3. Select only the "Complete" genomes with the filter at the top
>
>    ![Filter settings, only "complete" is checked](../../images/ecoli-list.png)
>
> 3. At the top right, click "Download"
>
> 4. Upload this table to Galaxy
>
> 5. As this file is a CSV file, we need to convert it to TSV before Galaxy can use it.
>
>    {% snippet faqs/galaxy/datasets_convert_datatype.md conversion="Convert CSV to Tabular" %}
>
> 6. Rename this file to `genomes.tsv`
>
>    {% snippet faqs/galaxy/datasets_convert_datatype.md conversion="Convert CSV to Tabular" %}
>
{: .details}


> <hands-on-title>Preparing a list of all complete <i>E. coli</i> genomes</hands-on-title>
>
> 1. {% tool [Cut](Cut1) %}   columns from a table:
>
>    - *"Cut columns"*: `c8,c20`
>    - *"From"*: `genome_proks.txt`
>
> > <question-title></question-title>
> >
> > How does your data look?
> >
> > > <solution-title></solution-title>
> > >
> > > It should look like this:
> > > 1               | 2
> > > --              | --
> > > GCA_000005845.2 | ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/005/845/GCA_000005845.2_ASM584v2
> > > GCA_000008865.2 | ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/008/865/GCA_000008865.2_ASM886v2
> > > GCA_003697165.2 | ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/003/697/165/GCA_003697165.2_ASM369716v2
> > > GCA_003018455.1 | ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/003/018/455/GCA_003018455.1_ASM301845v1
> > > GCA_001650295.1 | ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/650/295/GCA_001650295.1_ASM165029v1
> > > GCA_003018035.1 | ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/003/018/035/GCA_003018035.1_ASM301803v1
> > > GCA_003112225.1 | ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/003/112/225/GCA_003112225.1_ASM311222v1
> > > GCA_001695515.1 | ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/695/515/GCA_001695515.1_ASM169551v1
> > > GCA_001721125.1 | ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/721/125/GCA_001721125.1_ASM172112v1
> > > GCA_000091005.1 | ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/091/005/GCA_000091005.1_ASM9100v1
> > > GCA_005037725.2 | ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/005/037/725/GCA_005037725.2_ASM503772v2
> > > GCA_005037815.2 | ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/005/037/815/GCA_005037815.2_ASM503781v2
> > > GCA_004358405.1 | ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/004/358/405/GCA_004358405.1_ASM435840v1
> > > GCA_003018575.1 | ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/003/018/575/GCA_003018575.1_ASM301857v1
> > >
> > {: .solution}
> {: .question}
>
>
{: .hands_on}

Now that the list is formatted as a table in a spreadsheet, it is time to upload it into Galaxy. There is a problem though: the URLs (web addresses) in the list do not actually point to sequence files that we would need to perform alignments. Instead they point to directories. For example, this URL: [GCA_000008865.1_ASM886v1](ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/008/865/GCA_000008865.1_ASM886v1) points to a directory (rather than a file) containing many files, most of which we do not need.

![GenBank assembly files for an E. coli strain](../../images/genbank_dir.png "A list of files for an <i>E. coli</i> assembly. For further analyses we only need the dataset ending with <code>_genomic.fna.gz</code>.")

So to download sequence files we need to edit URLs by adding filenames to them. For example, in the case of the URL shown above we need to add `/GCA_000008865.1_ASM886v1` and `_genomic.fna.gz` to the end to get this:

```
ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/008/865/GCA_000008865.1_ASM886v1/GCA_000008865.1_ASM886v1_genomic.fna.gz
```

This can be done as a two step process where we first copy the end part of the existing URL (`/GCA_000008865.1_ASM886v1`) and then add a fixed string `_genomic.fna.gz` to the end of it. Doing this by hand is crazy and trying to do it in a spreadsheet is complicated. Fortunately, Galaxy's new rule-based uploader can help, as shown in the next **Hands-on** section:

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Again **Upload** {% icon tool %} data
>
> 2. Switch to the `Rule-based` tab on the right
>
>    > <tip-title>Using the Rule-based Uploader</tip-title>
>    > There is a detailed tutorial on using the [Rule based Uploader]({% link topics/galaxy-interface/tutorials/upload-rules/tutorial.md %}) if you want to learn about the more advanced features available.
>    {: .tip}
>
>    - *"Upload data as"*: `Collection(s)`
>    - *"Load tabular data from"*: `History Dataset`
>    - *"Select dataset to load"*: output of the cut tool
>
>    > <tip-title>dataset not there?</tip-title>
>    > If the dataset doesn't appear in the select list, refresh your page.
>    {: .tip}
>
> 3. From **Column**, select `Using a Regular Expression`
>    - *"From Column"*: `B`
>    - Select `Create columns matching expression groups`
>    - *"Regular Expression"*: `.*(\/GCA.*$)`
>    - *"Number of Groups"*: `1`
>    - Click `Apply`
> 4. From **Column**, select `Concatenate Columns`
>    - *"From Column"*: `B`
>    - *"From Column"*: `C`
>    - Click `Apply`
> 6. From **Column**, select `Fixed Value`
>    - *"Value"*: `_genomic.fna.gz`
>    - Click `Apply`
> 7. From **Column**, select `Concatenate Columns`
>    - *"From Column"*: `D`
>    - *"From Column"*: `E`
>    - Click `Apply`
> 9. From **Rules** menu, select `Add / Modify Column Definitions`
>    - `Add Definition`, `List Identifier(s)`, Select Column `A`
>    - `Add Definition`, `URL`, Column `F`
>    - Click `Apply`
> 10. Set the **Type** in the bottom left to `fasta.gz`
> 11. Give the upload a name like `Complete genomes`
> 12. **Upload**
>
>
{: .hands_on}

Now we have all complete *E. coli* genomes in Galaxy's history. It is time to do a few things to our assembly.

## Preparing assembly


Before starting any analyses we need to upload the assembly produced in [Unicycler tutorial]({{site.baseurl}}/topics/assembly/tutorials/unicycler-assembly/tutorial.html) from Zenodo:

> <hands-on-title>Uploading <i>E. coli</i> assembly into Galaxy</hands-on-title>
>
> 1. {% tool [Upload](upload1) %}  :
>   - Click **Paste/Fetch data** button (Bottom of the interface box)
>   - **Paste** `https://zenodo.org/record/1306128/files/Ecoli_C_assembly.fna` into the box.
>   - *"Type"*: `fasta`
>   - Click **Start**
{: .hands_on}

> <tip-title>Finding tools mentioned in this tutorial</tip-title>
>Galaxy instances contain hundreds of tools. As a result, it can be hard to find tools mentioned in tutorials such as this one. To help with this challenge, Galaxy has a search box at the top of the left panel. Use this box to find the tools mentioned here.
>![Tool search](../../images/tool_search.png "Use search box to find tools!")
{: .tip}

The assembly we just uploaded has two issues that need to be addressed before proceeding with our analysis:

1. It contains two sequences: the one of *E. coli* C genome (the one we really need) and another representing phage phiX174 (a by product of Illumina sequencing where it is used as a spike-in DNA).
2. Sequences have unwieldy names like `>1 length=4576293 depth=1.00x circular=true`. We need to rename it to something more meaningful.

Let's fix these two problems.

Because phiX173 is around 5,000bp, we can remove those sequences by setting a minimum length of 10,000:

> <hands-on-title>Fixing assembly</hands-on-title>
>
> 1. {% tool [Filter sequences by length](toolshed.g2.bx.psu.edu/repos/devteam/fasta_filter_by_length/fasta_filter_by_length/1.2) %} with the following parameters:
>   - *"Fasta file"*: the dataset you've just uploaded. (`https://zenodo.org/record/1306128/files/Ecoli_C_assembly.fna`).
>   - *"Minimal length"*: `10000`
>
>
> 2. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2) %} in entire line:
>    - *"File to process"*: the output of the Filter sequences by length {% icon tool %}
>    - *"1: Replacement"*
>      - *"Find Pattern"*: `^>1.*`
>      - *"Replace with"*: `>Ecoli_C`
>
>    {% snippet faqs/galaxy/datasets_rename.md name="E. coli C" %}
>
>    > <tip-title>Regular Expressions</tip-title>
>    >
>    > The program we just entered is a so-called [Regular Expression](https://en.wikipedia.org/wiki/Regular_expression)
>    >
>    > The expression `^>1.*` contains several pieces that you need to understand. Let's write it top-to-bottom and explain:
>    >
>    > - `^` - says start looking at *the beginning* of each line
>    > - `>` - is the first character we want to match. Remember that name of the sequence in FASTA files starts with `>`
>    > - `1` - is the number present is our old name (`>1 length=4576293 depth=1.00x circular=true` to `>Ecoli_C`)
>    > - `.` - dot has a special meaning. It signifies *any* character
>    > - `*` - is a *quantifier*. From [Wikipedia](https://en.wikipedia.org/wiki/Regular_expression): "The asterisk indicates zero or more occurrences of the preceding element. For example, ab*c matches `ac`, `abc`, `abbc`, `abbbc`, and so on."
>    >
>    >So in short we are replacing `>1 length=4576293 depth=1.00x circular=true` with `>Ecoli_C`. The *Regular expression* `^>1.*` is used here to represent `>1 length=4576293 depth=1.00x circular=true`.<br>
>    >Detailed description of regular expressions is outside of the scope of this tutorial, but there are other great resources. Start with [Software Carpentry Regular Expressions tutorial](http://v4.software-carpentry.org/regexp/index.html).
>    {: .tip}
>
> > <question-title></question-title>
> >
> > 1. What is the meaning of `^` character is SED expression?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. It tells SED to start matching from the beginning of the string.
> > >
> > {: .solution}
> {: .question}
>
{: .hands_on}

## Generating alignments


Now everything is loaded and ready to go. We will now align our assembly against each of the *E. coli* genomes we have uploaded into the collection. To do this we will use [LASTZ](https://lastz.github.io/lastz/)&mdash;an aligner designed for long sequences.

> <hands-on-title>Running LASTZ</hands-on-title>
> 1. {% tool [LASTZ](toolshed.g2.bx.psu.edu/repos/devteam/lastz/lastz_wrapper_2/1.3.2) %} with the following parameters:
>   - *"Select TARGET sequence(s) to align against"*: `from your history`
>   - {% icon param-collection %} *"Select a reference dataset"*: the "Complete genomes" collection we uploaded earlier
>   - {% icon param-file %} *"Select QUERY sequence(s)"*: our *E. coli* assembly which was prepared in the previous step.
>   - **Chaining**
>        - *"Perform chaining of HSPs with no penalties"*: `Yes`
>   - **Output**
>        - *"Specify the output format"*: `blastn`
>
{: .hands_on}

Note that because we started LASTZ on *a collection* of *E. coli* genomes, it will output alignment information as *a collection* as well. A collection is simply a way to represent large sets of similar data in a compact way within Galaxy's interface.

> <warning-title>It will take a while!</warning-title>
> Please understand that alignment is not an instantaneous process: allow several hours for these jobs to clear.
{: .warning}

## Finding closely related assemblies


### Understanding LASTZ output

LASTZ produced data in so-called `blastn` format (because we explicitly told LASTZ to output in this format, see previous step), which looks like this:

```
      1          2     3   4  5 6       7       8    9   10      11    12
-------------------------------------------------------------------------
Ecoli_C BA000007.2 66.81 232 51 6 3668174 3668397 5936 6149 3.2e-40 162.7
Ecoli_C BA000007.2 57.77 206 38 8  643802  643962 5945 6146 1.6e-18  90.6
Ecoli_C BA000007.2 67.03 185 32 6 4849373 4849528 5965 6149 2.9e-28 122.9
Ecoli_C BA000007.2 63.06 157 33 3 1874604 1874735 5991 6147 5.8e-26 115.3
```

where columns are:

1. `qseqid` - query (e.g., gene) sequence id
2. `sseqid` - subject (e.g., reference genome) sequence id
3. `pident` - percentage of identical matches
4. `length` - alignment length
5. `mismatch` - number of mismatches
6. `gapopen` - number of gap openings
7. `qstart` - start of alignment in query
8. `qend` - end of alignment in query
9. `sstart` - start of alignment in subject
10. `send` - end of alignment in subject
11. `evalue` - [expect value](https://blast.ncbi.nlm.nih.gov/Blast.cgi?CMD=Web&PAGE_TYPE=BlastDocs&DOC_TYPE=FAQ#expect)
12. `bitscore` - [bit score](https://www.ncbi.nlm.nih.gov/BLAST/tutorial/Altschul-1.html)

The alignment information produced by LASTZ is a collection. In this collection each element contains alignment data between each of the *E. coli* genomes and our assembly:

![LASTZ collection](../../images/lastz_collection.png "LASTZ produced a collection where each element corresponds to an alignment between an <i>E. coli</i> genome and our assembly. Here one of the elements is expanded (to expand an element simply click on it).").

## Collapsing collection

Collections are a wonderful way to organize large sets of data and parallelize data processing like we did here with LASTZ. However, at this point we need to combine all data into one dataset. Follow the steps below to accomplish this:

> <hands-on-title>Combining collection into a single dataset</hands-on-title>
> 1. {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/4.2) %} with the following parameters:
>   - *"Collection of files to collapse"*: the output of **LASTZ** (collecion input)
{: .hands_on}

This will produce one gigantic table (over 12 million lines) containing combined LASTZ output for all genomes.

## Getting taste of the alignment data

To make further analyses we need to get an idea about alignment data generated with LASTZ. To do this let's select a random subsample of the large dataset we've generated above. This is necessary because processing the entire dataset will take time and will not give us a better insight anyway. So first we will select 10,000 lines from the alignment data:

> <hands-on-title>Selecting random subset of data</hands-on-title>
> 1. {% tool [Select random lines from a file](random_lines1) %}   with the following parameters:
>   -  *"Randomly select"*: `10000`
>   -  *"from"*: the output from `Collapse Collection`
{: .hands_on}

Now we can visualize this dataset to discover generalities:

> <hands-on-title>Graphing alignment data</hands-on-title>
> 1. Expand random subset of alignment data generated on the previous step by clicking on it.
> 2. You will see "chart" button {% icon galaxy-barchart %}. Click on it.
> 3. In the central panel you will see a list of visualizations. Select **Scatter plot (NVD3)**
> 4. Click **Select data** {% icon galaxy-chart-select-data %}
> 5. Set **Values for x-axis** to `Column: 3` (alignment identity)
> 6. Set **Values for y-axis** to `Column: 4` (alignment length)
> 7. You can also click on configuration button {% icon galaxy-gear %} and specify axis labels etc.
{: .hands_on}

The relationship between the alignment identity and alignment length looks like this (remember that this is only a subsample of the data):

![Identity versus length](../../images/id_vs_len.png "Alignment identity (%) versus length (bp). This graph is truncated at the top")

You can see that most alignments are short and have relatively low identity. Thus we can filter the original dataset by identity and length. Judging from this graph we can select alignment longer than 10,000 bp with identity above 90%.

> <hands-on-title>Filtering data</hands-on-title>
> 1. {% tool [Filter](Filter1) %} data on any column using simple expressions:
>   - *"Filter"*: the full dataset, from the output of the **Collapse Collection** {% icon tool %}.
>   - *"With following condition"*: `c3 >= 90 and c4 >= 10000` (here `c` stands for *column*).
>
>  NOTE: You need to select the full dataset; not the down-sampled one, but [the one generated by the collection collapsing operation](#hands-on-combining-collection-into-a-single-dataset).
>
{: .hands_on}

## Aggregating data

Remember, our objective is to find the genomes that are most similar to ours. Given the alignment data in the table we just created we can define similarity as follows:

> Genomes that have the smallest number of alignment blocks but the highest overall alignment length are most similar to our assembly. This essentially means that they have longest uninterrupted region of high similarity to our assembly.
{: .quote}

However, to extract this information from our data we need to aggregate it. In other words, for each *E. coli* genome we need to calculate the total number of alignment blocks, their combined length, and average identity. The following section explains how to do this:

> <hands-on-title>Aggregating the data</hands-on-title>
> 1. {% tool [Datamash (operations on tabular data)](toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.1.0) %}   with the following parameters:
>     - *"Input tabular dataset"*: output of the previous `Filter` step.
>     - *"Group by fields"*: `2`. (column 1 contains name of the *E. coli* genome we mapped against)
>     - *"Sort input"*: `Yes`
>     - *"Operation to perform on each group"*:
>         - *"Type"*: `Count`
>         - *"On column"*: `Column: 2`
>     - {% icon param-repeat %} *"Insert operation to perform on each group"*
>       - *"Operation to perform on each group"*:
>           - *"Type"*: `Mean`
>           - *"On column"*: `Column: 3`.
>     - {% icon param-repeat %} *"Insert operation to perform on each group"*
>       - *"Operation to perform on each group"*:
>           - *"Type"*: `Sum`
>           - *"On column"*: `Column: 4`
{: .hands_on}

## Finding closest relatives

The dataset generated above lists each *E. coli* genome accession only once and will have aggregate information for the number of alignment blocks, mean identity, and total length. Let's graph these data:

> <hands-on-title>Graphing aggregated data</hands-on-title>
> 1. Expand the aggregated data generated on the previous step by clicking on it.
> 2. You will see "chart" button {% icon galaxy-barchart %}. Click on it.
> 3. In the central panel you will see a list of visualizations. Select **Scatter plot (NVD3)**
> 4. Click **Select data** {% icon galaxy-chart-select-data %}
> 5. Set **Data point labels** to `Column: 1` (Accession number of each *E. coli* genome)
> 5. Set **Values for x-axis** to `Column: 2` (# of alignment blocks)
> 6. Set **Values for y-axis** to `Column: 4` (Total alignment length)
> 7. You can also click on configuration button {% icon galaxy-gear %} and specify axis labels etc.
{: .hands_on}

The relationship between the number of alignment blocks and total alignment length looks like this:

![Identity versus length](../../images/best_genomes_chart.png "Number of alignment blocks versus total alignment length (bp).")

A group of three dots in the upper left corner of this scatter plot represents genomes that are most similar to our assembly: they have a SMALL number of alignment blocks but HIGH total alignment length. Mousing over these three dots (if you set **Data point labels** correctly in the previous step) will reveal their accession numbers: `LT906474.1`, `CP024090.1`, and `CP020543.1`.

> <warning-title>Things change</warning-title>
> It is possible that when you repeat these steps the set of sequences in NCBI will have changed and you will obtain different accession numbers. Keep this in mind.
{: .warning}

Let's find table entries corresponding to these:

> <hands-on-title>Extracting into about best hits</hands-on-title>
> 1. {% tool [Select lines that match an expression](Grep1) %} with the following parameters:
>   - *"Select lines from"*: to the output from `Datamash`
>   - *"the pattern"*: `LT906474|CP024090|CP020543`. (Here `|` means `or`).
{: .hands_on}

This will generate a short table like this:


CP020543.1 | 11 | 99.926363636364 | 4486976
CP024090.1 | 12 | 99.911666666667 | 4540487
LT906474.1 | 8  | 99.94           | 4575200

From this it appears that `LT906474.1` is closest to our assembly because it has eight alignment blocks, the longest total alignment length (4,575,223) and highest mean identity (99.94%).

# Comparing genome architectures

Now that we know the three genomes most closely related to ours, let's take a closer look at them. First we will re-download sequence and annotation data.

## Getting sequences and annotations

> <hands-on-title>Uploading sequences and annotations</hands-on-title>
> Using the three accession listed above we will fetch necessary data from NCBI. We will use the spreadsheet we uploaded at the start to accomplish this.
>
> 1. {% tool [Upload](upload1) %} the *E. coli* C genome if you have not done so already:
>   - Click **Paste/Fetch data** button (Bottom of the interface box)
>   - **Paste** `https://zenodo.org/record/1306128/files/Ecoli_C_assembly.fna` into the box.
>   - *"Type"*: `fasta`
>   - Click **Start**
>
> 1. {% tool [Select lines that match an expression](Grep1) %} with the following parameters:
>   - *"Select lines from"*: the `genomes.tsv` you uploaded earlier
>   - *"the pattern"*: `LT906474|CP024090|CP020543`
>
> 2. {% tool [Cut](Cut1) %} columns from a table:
>
>    - *"Cut columns"*: `c8,c20`
>    - *"From"*: the output of the **select lines** {% icon tool %}
>
>      It should look like:
>
>      ```
>      GCA_002079225.1	ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/002/079/225/GCA_002079225.1_ASM207922v1
>      GCA_002761835.1	ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/002/761/835/GCA_002761835.1_ASM276183v1
>      GCA_900186905.1	ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/900/186/905/GCA_900186905.1_49923_G01
>      ```
>
> 1. Again **Upload** {% icon tool %} data
>
>    2. Switch to the `Rule-based` tab on the right
>
>       - *"Upload data as"*: `Collection(s)`
>       - *"Load tabular data from"*: `History Dataset`
>       - *"Select dataset to load"*: output of the cut tool
>
>       > <tip-title>dataset not there?</tip-title>
>       > If the dataset doesn't appear in the select list, refresh your page.
>       {: .tip}
>
>       > <tip-title>Take a Shortcut</tip-title>
>       > This step is quite long and potentially error prone. If you want to skip those steps, you can copy and paste this bit of text:
>       >
>       > ```json
>       > {"rules":[{"type":"add_column_regex","target_column":1,"expression":".*(\\/GCA.*$)","group_count":1},{"type":"add_column_concatenate","target_column_0":1,"target_column_1":2},{"type":"remove_columns","target_columns":[1,2]},{"type":"add_column_value","value":"_feature_table.txt.gz"},{"type":"add_column_value","value":"_genomic.fna.gz"},{"type":"add_column_concatenate","target_column_0":1,"target_column_1":2},{"type":"add_column_concatenate","target_column_0":1,"target_column_1":3},{"type":"remove_columns","target_columns":[1,2,3]},{"type":"add_column_value","value":"Genes"},{"type":"add_column_value","value":"DNA"},{"type":"add_column_regex","target_column":1,"expression":".*\\/(.*)","group_count":1},{"type":"swap_columns","target_column_0":0,"target_column_1":5},{"type":"remove_columns","target_columns":[5]},{"type":"split_columns","target_columns_0":[1,3],"target_columns_1":[2,4]}],"mapping":[{"type":"list_identifiers","columns":[0],"editing":false},{"type":"url","columns":[1]},{"type":"collection_name","columns":[2]}]}
>       > ```
>       >
>       > You can click the {% icon tool %} next to the header **Rules** {% icon tool %}, and paste the contents there, before clicking **Apply**, checking "Add nametag for name" and then **Upload**.
>       {: .tip}
>
>    3. From **Column**, select `Using a Regular Expression`
>       - *"From Column"*: `B`
>       - Select `Create columns matching expression groups`
>       - *"Regular Expression"*: `.*(\/GCA.*$)`
>       - *"Number of Groups"*: `1`
>       - Click `Apply`
>    4. From **Column**, select `Concatenate Columns`
>       - *"From Column"*: `B`
>       - *"From Column"*: `C`
>       - Click `Apply`
>    5. From **Rules**, select `Remove Columns(s)`
>       - *"From Column"*: `B`, `C`
>       - Click `Apply`
>    6. From **Column**, select `Fixed Value`
>       - *"Value"*: `_feature_table.txt.gz`
>       - Click `Apply`
>    6. From **Column**, select `Fixed Value`
>       - *"Value"*: `_genomic.fna.gz`
>       - Click `Apply`
>    7. From **Column**, select `Concatenate Columns`
>       - *"From Column"*: `B`
>       - *"From Column"*: `C`
>       - Click `Apply`
>    7. From **Column**, select `Concatenate Columns`
>       - *"From Column"*: `B`
>       - *"From Column"*: `D`
>       - Click `Apply`
>    8. From **Rules**, select `Remove Columns(s)`
>       - *"From Column"*: `B`, `C`, `D`
>       - Click `Apply`
>    6. From **Column**, select `Fixed Value`
>       - *"Value"*: `Genes`
>       - Click `Apply`
>    6. From **Column**, select `Fixed Value`
>       - *"Value"*: `DNA`
>       - Click `Apply`
>    3. From **Column**, select `Using a Regular Expression`
>       - *"From Column"*: `B`
>       - Select `Create columns matching expression groups`
>       - *"Regular Expression"*: `.*\/(.*)`
>       - *"Number of Groups"*: `1`
>       - Click `Apply`
>    9. From **Rules** menu, select `Swap Column(s)`
>       - *"Swap Column"*: `A`
>       - *"With Column"*: `F`
>       - Click `Apply`
>    8. From **Rules**, select `Remove Columns(s)`
>       - *"From Column"*: `F`
>       - Click `Apply`
>    9. From **Rules** menu, select `Split Column(s)`
>       - *"Odd Row Column(s)"*: `B`, `D`
>       - *"Even Row Column(s)"*: `C`, `E`
>       - Click `Apply`
>    9. From **Rules** menu, select `Add / Modify Column Definitions`
>       - `Add Definition`, `List Identifier(s)`, Select Column `A`
>       - `Add Definition`, `URL`, Column `B`
>       - `Add Definition`, `Collection Name`, Column `C`
>       - Click `Apply`
>    12. Check **Add nametag for name**
>    12. **Upload**
>
> At the end of this you should have two collections: one containing genomic sequences and another containing annotations.
{: .hands_on}

## Visualizing rearrangements

Now we will perform alignments between our assembly and the three most closely related genomes to get a detailed look at any possible genome architecture changes. We will again use LASTZ:

> <hands-on-title>Aligning again</hands-on-title>
>
> 1. {% tool [LASTZ](toolshed.g2.bx.psu.edu/repos/devteam/lastz/lastz_wrapper_2/1.3.2) %} with the following parameters:
>   - *"Select TARGET sequence(s) to align against"*: `from your history`
>   - {% icon param-collection %} *"Select a reference dataset"*: `DNA`, the *E. coli* genomes we uploaded earlier
>   - {% icon param-file %} *"Select QUERY sequence(s)"*: `E. coli C` fasta file
>   - **Chaining**
>        - *"Perform chaining of HSPs with no penalties"*: `Yes`
>          > <tip-title>What does chaining do?</tip-title>
>          > For more information about chaining [look here](https://lastz.github.io/lastz/#ex_stages)
>          {: .tip}
>
>   - **Output**
>        - *"Specify the output format"*: `Customized general`
>        - *"Select which fields to include"*: select the following
>             - `score` alignment score
>             - `name1` name of the *target* sequence
>             - `strand1` strand for the *target* sequence
>             - `zstart1` 0-based start of alignment in *target*
>             - `end1` end of alignment in *target*
>             - `length1` length of alignment in *target*
>             - `name2` name of *query* sequence
>             - `strand2` strand for the *query* sequence
>             - `zstart2` 0-based start of alignment in *query*
>             - `end2` end of alignment in *query*
>             - `identity` alignment identity
>             - `number` alignment number
>        - *"Create a dotplot representation of alignments?"*: `Yes`
>
> 2. Rename the `LASTZ on collection... mapped reads` something more memorable like `LASTZ Alignments`
>
>    {% snippet faqs/galaxy/collections_rename.md name="LASTZ Alignments" %}
>
{: .hands_on}

Because we chose to produce Dot Plots as well, LASTZ will generate two collections: one containing alignment data and the other containing DotPlots in PNG format:

![Dot Plots](../../images/three_dot_plots.png "Dot Plot representations of alignments between three <i>E. coli</i> genomes and our assembly. Target (X-axis) is indicated above each dot plot. Query (Y-axis) is our assembly. Red circle indicates a region deleted in our assembly.")

A quick conclusion that can be drawn here is that there is a large inversion in CP020543 and deletion in our assembly.

> <details-title>Interpreting Dot Plots</details-title>
> If you are not sure how to interpret Dot Plots here is a great explanation by [Michael Schatz](http://schatz-lab.org/):
>
> ![Interpreting Dot Plots](../../images/dotplot.png "A quick reference to interpreting Dot Plots. Our case is identical to <i>Insertion into Reference</i> shown in the upper left.")
{: .details}

For a moment let's leave LASTZ result and create a browser that would allows us to display our results.

## Producing a Genome Browser for this experiment

The dot plots we've produced above are great, but they are static. It would be wonderful to load these data into a genome browser where one can zoom in and out as well as add tracks such as those containing genes. To create a browser we need a genome and a set of tracks. Tracks are features such as genes or SNPs with start and end positions corresponding to a coordinate system provided by the genome. Thus the first thing to do is to create a *genome* that would represent our experiment. We can create such a genome by simply combining the three genomes of closely related strains with our assembly in a single dataset&mdash;a hybrid genome.

### Collecting the genomes

The first step will be collapsing the collection containing the three genomes into a single file:

> <hands-on-title>Creating a single FASTA dataset with all genomes</hands-on-title>
>
> 1. {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/4.0) %}
>
>    - {% icon param-collection %} *"Collection of files to collapse"* the three genomes (collection) named `DNA`
>
> 2. Convert the datatype of this output to uncompress it
>
>    {% snippet faqs/galaxy/datasets_convert_datatype.md conversion="Convert compressed to uncompressed" %}
>
> 3. {% tool [Concatenate datasets](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/0.1.0) %} tail-to-head (cat):
>    - *"Datasets to concatenate"*: `Collapse collection ... uncompressed`, the output from the uncompression step.
>    - Click **Insert Dataset** button
>        - *"Select"*: the `E. coli C` file from the start of the history
>
> 4. Rename the output to `DNA (E. coli C + Relatives)`
>
>    {% snippet faqs/galaxy/datasets_rename.md name="DNA (E. coli C + Relatives)" %}
{: .hands_on}

The resulting dataset contains four sequences: three genomes plus our assembly.

### Preparing the alignments

Above we computed alignments using LASTZ. Because we ran LASTZ on a collection containing genomic sequences, LASTZ produced a collection as well (actually two collections: one containing alignments an the other with dot plots). To display alignments in the browser we need to do several things:

1. Fix unwanted `%` signs in LASTZ output
2. Create names for alignment blocks
3. Convert LASTZ output into [BED](https://genome.ucsc.edu/FAQ/FAQformat.html#format1) format
4. Create a single BED track containing alignments against all four genomes.

To begin, let's look at the LASTZ output:

1        | 2            | 3 | 4      | 5      | 6      | 7         | 8 | 9      | 10     | 11            | 12     | 13
-------- | ----------   | - | ------ | ------ | ------ | -------   | - | ------ | ------ | -------       | ------ | --
10141727 | `CP020543.1` | + | 48     | 106157 | 106109 | `Ecoli_C` | + | 0      | 106109 | 106107/106109 | 100.0% | 1
    5465 | `CP020543.1` | + | 121267 | 121367 | 100    | `Ecoli_C` | + | 109317 | 109418 | 76/100        | 76.0%  | 2
    4870 | `CP020543.1` | + | 159368 | 159512 | 144    | `Ecoli_C` | + | 128706 | 128828 | 95/115        | 82.6%  | 3

One immediate problem is `%` character in column 12 (alignment identity). We need to remove it as we will use this for the score column of the BED file, and that must be a normal number and not a percentage.

Column 13 of the fields chosen by us for [LASTZ run](#hands-on-aligning-again) is `number`. This is an incrementing number given by LASTZ to every alignment block so it can be uniquely identified. The problem is that by running LASTZ on a collection of three genomes it generated a number for each output independently starting with `1` each time. So these alignments identified are unique within each individual run but are redundant for multiple runs. We can fix that by pre-pending each alignment identified (column 12) with the name of the target sequence (column 2). This would create alignments that are truly unique. For example, in the case of the LASTZ output shown above alignment identifier `1` will become `CP020543.11`, `2` will become `CP020543.12` and so on.

> <comment-title>BED format</comment-title>
> Our goal is to convert this into a format that will be acceptable to the genome browser. One of such formats is [BED](https://genome.ucsc.edu/FAQ/FAQformat.html#format1). In one of its simplest forms (there is one even simpler - 3 column BED) it has six columns:
>
> 1. Chromosome ID
> 2. Start
> 3. End
> 4. Name of the feature
> 5. Score (must be between 0 and 1000)
> 6. Strand (`+`, `-`, or `.` for no strand data).
{: .comment}

> <hands-on-title>Convert LASTZ output to BED</hands-on-title>
>
> 1. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.1.0) %} in a specific column:
>    - {% icon param-collection %} *"File to process"*: output of LASTZ (`LASTZ Alignments`)
>    - *"in column"*: `Column 12`
>    - *"Find pattern"*: `%`
>    - *"Replace with"*: leave empty
>
> 2. {% tool [Merge Columns together](toolshed.g2.bx.psu.edu/repos/devteam/merge_cols/mergeCols1/1.0.1) %} with the following parameters:
>    - {% icon param-collection %} *"Select data"*: the output of the previous step, `Replace Text on collection ...`
>    - *"Merge column"*: `Column: 2` (this is the Target sequence name)
>    - *"with column"*: `Column: 13` (this is the alignment block created by LASTZ)
>
>    > <details-title>Output information</details-title>
>    > The tool added a new column (Column 14) containing a merge between the target name and alignment id. Now we can differentiate between alignment blocks that exist between, for example, `CP020543.1` and `LT906474.1` because they will have accessions embedded within alignment block IDs. For example, the first alignment between `CP020543.1` and our assembly `Ecoli_C` will have alignment block id `CP020543.11`, while the 225th alignment between `LT906474.1` and `Ecoli_C` will have ID `LT906474.1225`. Because of this we can collapse the entire collection of alignments into a single dataset:
>    {: .details}
>
> 3. {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/4.0) %} with the following parameters:
>    - {% icon param-collection %} *"Collection of files to collapse"*: the output of the previous step, `Merge Columns on collection...`
>
>    This will produce a single dataset combining all alignment info. We can tell which alignments are between which genomes because we have set identifiers such as `CP020543.13`.
>
> 4. We will reuse this file later so let's rename it `Unprocessed Alignments`
>
>    {% snippet faqs/galaxy/datasets_rename.md name="Unprocessed Alignments" %}
>
> 4. {% tool [Cut](Cut1) %} columns from a table:
>
>    - *"Cut columns"*: `c2,c4,c5,c14,c12,c8`
>    - {% icon param-file %} *"From"*: the output of the previous step (`Unprocessed alignments`)
>
>    > <details-title>Converting to BED</details-title>
>    > Let's look again at the data we generated in the last step:
>    >
>    > 1        | 2          | 4 | 4      | 5      | 6      | 7       | 8 | 9      | 10     | 11            | 12    | 13 | 14
>    > -------- | ---------- | - | ------ | ------ | ------ | ------- | - | ------ | ------ | --            | ----- | -- | -----------
>    > 10141727 | CP020543.1 | + | 48     | 106157 | 106109 | Ecoli_C | + | 0      | 106109 | 106107/106109 | 100.0 | 1  | CP020543.11
>    >     5465 | CP020543.1 | + | 121267 | 121367 | 100    | Ecoli_C | + | 109317 | 109418 | 76/100        | 76.0  | 2  | CP020543.12
>    >     4870 | CP020543.1 | + | 159368 | 159512 | 144    | Ecoli_C | + | 128706 | 128828 | 95/115        | 82.6  | 3  | CP020543.13
>    >
>    > Alignments are regions of high similarity between two sequences. Therefore each alignment block has two sets of coordinates associated with it: start/end in the first sequences (target) and start/end in the second sequence (query). But BED only has one set of coordinates. Thus we can create two BEDs: one using coordinates from the target and the other one from query. The first file will depict alignment data from the standpoint of target sequences `CP020543.1`, `CP024090.1`, `LT906474.1` and the second from the standpoint of query - our own assembly [we called](#hands-on-fixing-assembly) `Ecoli_C`.
>    > In the first BED, column 1 will contain names of targets (`CP020543.1`, `CP024090.1`, and `LT906474.1`).
>    > In the second BED, column 1 will contain name of our assembly: `Ecoli_C`.
>    >
>    > To create the first BED we will cut six columns from the dataset produced at the last step. Specifically, to produce the target BED we will cut columns 2, 4, 5, 14, 12, and 8. To produce the query BED columns 7, 9, 10, 14, 12, 8 will be cut.
>    {: .details}
>
>    > <warning-title>There are multiple <b>CUT</b> tools!</warning-title>
>    > The Hands-On box below uses **Cut** tool. Beware that some Galaxy instances contain multiple **Cut** tools. The one that is used below is called **Cut columns from a table** while the other one, which we will NOT use is called **Cut columns from a table (cut)**. It is a small difference, but the tools are different.
>    {: .warning}
>
>    This will produce a dataset looking like this:
>
>    1          | 2      | 3      | 4           | 5     | 6
>    ---------- | ------ | ------ | ----------- | ----- | -
>    CP020543.1 | 48     | 106157 | CP020543.11 | 100.0 | +
>    CP020543.1 | 121267 | 121367 | CP020543.12 | 76.0  | +
>    CP020543.1 | 159368 | 159512 | CP020543.13 | 82.6  | +
>
>    > <tip-title>Not exactly the same?</tip-title>
>    > Depending on the steps and other choices, the genomes may be in a different order here. This is unimportant, as all of the same alignments are contained in the file, just the ordering is different. As long as these columns look correct (start/end in column 2/3 are reasonable, a number between 0-100 in column 5, and a + or - in column 6) then it is OK.
>    {: .tip}
>
> 5. Rename this "Target Alignments"
>
>    {% snippet faqs/galaxy/datasets_rename.md name="Target Alignments" %}
>
> 6. {% tool [Cut columns from a table](Cut1) %} with the following parameters
>    - *"Cut columns"*: `c7,c9,c10,c14,c12,c8` (look at the data shown above and the definition of BED to see why we make these choices.)
>    - *"From"*: `Unprocessed alignments`, the output of **collection collapse**
>
> 5. Rename this "Query Alignments"
>
>    {% snippet faqs/galaxy/datasets_rename.md name="Query Alignments" %}
>
> 6. {% tool [Concatenate datasets tail-to-head](cat1) %}
>    - *"Concatenate Dataset"*: `Query Alignments`
>    - Click "Insert Dataset" button
>    - *"1: Dataset"*: `Target Alignments`
>
> 7. Change the datatype of the output to BED and rename the output "Target & Query Alignments"
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="bed" %}
>
>    {% snippet faqs/galaxy/datasets_rename.md name="Target & Query Alignments" %}
>
{: .hands_on}

This will produce a dataset looking like this:

1       | 2      | 3      | 4           | 5     | 6
------- | ------ | ------ | ----------- | ----- | -
Ecoli_C | 0      | 106109 | CP020543.11 | 100.0 | +
Ecoli_C | 109317 | 109418 | CP020543.12 | 76.0  | +
Ecoli_C | 128706 | 128828 | CP020543.13 | 82.6  | +


### Extracting Genes

Earlier we [downloaded](#hands-on-uploading-sequences-and-annotations) gene annotations for the three genomes most closely related to our assembly. The data was downloaded as a collection containing annotations for `CP020543.1`, `CP024090.1`, and `LT906474.1`. The annotation data contains multiple columns described by NCBI as follows (you can look at the actual data by finding the annotation collection from above (called `Genes`)):

> Tab-delimited text file reporting locations and attributes for a subset of
> annotated features. Included feature types are: gene, CDS, RNA (all types),
> operon, C/V/N/S_region, and V/D/J_segment.
>
> The file is tab delimited (including a #header) with the following columns:
>
> Column | Definition
> ---    | ---
> 1      | feature: INSDC feature type
> 2      | class: Gene features are subdivided into classes according to the gene biotype computed based on the set of child features for that gene. See the description of the gene_biotype attribute in the GFF3 documentation for more details: ftp://ftp.ncbi.nlm.nih.gov/genomes/README_GFF3.txt ncRNA features are subdivided according to the ncRNA_class. CDS features are subdivided into with_protein and without_protein, depending on whether the CDS feature has a protein accession assigned or not. CDS features marked as without_protein include CDS features for C regions and V/D/J segments of immunoglobulin and similar genes that undergo genomic rearrangement, and pseudogenes.
> 3      | assembly: assembly accession.version
> 4      | assembly_unit: name of the assembly unit, such as "Primary Assembly", "ALT_REF_LOCI_1", or "non-nuclear"
> 5      | seq_type: sequence type, computed from the "Sequence-Role" and "Assigned-Molecule-Location/Type" in the `*_assembly_report.txt` file. The value is computed as: if an assembled-molecule, then reports the location/type value. e.g. chromosome, mitochondrion, or plasmid if an unlocalized-scaffold, then report "unlocalized scaffold on <type>". e.g. unlocalized scaffold on chromosome else the role, e.g. alternate scaffold, fix patch, or novel patch
> 6      | chromosome
> 7      | genomic_accession
> 8      | start: feature start coordinate (base-1). start is always less than end
> 9      | end: feature end coordinate (base-1)
> 10     | strand
> 11     | product_accession: accession.version of the product referenced by this feature, if exists
> 12     | non-redundant_refseq: for bacteria and archaea assemblies, the non-redundant `WP_` protein accession corresponding to the CDS feature. May be the same as column 11, for RefSeq genomes annotated directly with `WP_` RefSeq proteins, or may be different, for genomes annotated with genome-specific protein accessions (e.g. `NP_` or `YP_` RefSeq proteins) that reference a `WP_` RefSeq accession.
> 13     | related_accession: for eukaryotic RefSeq annotations, the RefSeq protein accession corresponding to the transcript feature, or the RefSeq transcript accession corresponding to the protein feature.
> 14     | name: For genes, this is the gene description or full name. For RNA, CDS, and some other features, this is the product name.
> 15     | symbol: gene symbol
> 16     | GeneID: NCBI GeneID, for those RefSeq genomes included in NCBI's Gene resource
> 17     | locus_tag
> 18     | feature_interval_length: sum of the lengths of all intervals for the feature (i.e. the length without introns for a joined feature)
> 19     | product_length: length of the product corresponding to the accession.version in column 11. Protein product lengths are in amino acid units, and do not include the stop codon which is included in column 18. Additionally, product_length may differ from feature_interval_length if the product contains sequence differences vs. the genome, as found for some RefSeq transcript and protein products based on mRNA sequences and also for INSDC proteins that are submitted to correct genome discrepancies.
> 20     | attributes: semi-colon delimited list of a controlled set of qualifiers. The list currently includes: partial, pseudo, pseudogene, ribosomal_slippage, trans_splicing, anticodon=NNN (for tRNAs), old_locus_tag=XXX
>
{: .quote cite="https://ftp.ncbi.nlm.nih.gov/genomes/genbank/README.txt"}


Our objective is to convert these data into BED. In this analysis we want to initially concentrate on protein coding regions. To do this let's select all lines from the annotation datasets that contain the term `CDS`, then
we will produce a collection with three datasets just like the original `Genes` collection but containing only CDS data. Next we need to cut out only those columns that need to be included in the BED format. There is one problem with this. We are trying to convert these data into [6 column BED](#comment-bed-format). In this format the fifth column (score) must have a value between 0 and 1000. To satisfy this requirement we will create a dummy column that will always have a value of `0`.
Finally we can cut necessary columns from these datasets. These columns are 8 (start), 9 (end), 15 (gene symbol), 21 (dummy column we just created), and c10 (strand), and then we can add the genome name.

> <hands-on-title>Extract CDSs from annotation datasets</hands-on-title>
> 1. {% tool [Select lines that match an expression](Grep1) %} with the following parameters:
>    - {% icon param-collection %} *"Select lines from"*: the collection containing annotations, `Genes`
>    - *"the pattern"*: `^CDS`
>
>    This is because we want to retain all lines that begin (`^`) with `CDS`.
>
> 2. {% tool [Add column to an existing dataset](toolshed.g2.bx.psu.edu/repos/devteam/add_value/addValue/1.0.0) %}   with the following parameters:
>    - *"Add this value"*: `0`
>    - {% icon param-collection %} *"to Dataset"*: the collection produced by the previous step (`Select on collection...`)
>
>    This will be used for the "score" field of the BED file since we do not have a proper "score"
>
> 3. {% tool [Cut columns from a table](Cut1) %} with the following parameters:
>
>    We will produce two BED files, one using the product name (e.g. "chromosomal replication initiator protein DnaA") and one using the symbol (e.g. "thrA"). The product name is much more interesting to see in visualisations, but the symbol is more often used in other analyses and we will use that file later. We will start with the product name:
>
>    - *"Cut columns"*: `c8,c9,c14,c21,c10`
>    - {% icon param-collection %} *"From"* the collection produced 


