# tutorials

topic: metabolomics
tutorial: gc_ms_with_xcms
title: Mass spectrometry: GC-MS data processing (with XCMS, RAMClustR, RIAssigner, and matchms)
difficulty: intermediate
time_estimation: 2H
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: 47dad6ee3e7298bf9a395e7269120b32
last_modified: 2026-05-15T22:28:13.783462
zenodo_link: https://zenodo.org/record/7890956
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/gc_ms_with_xcms/tutorial.html

content:
The study of metabolites in biological samples is routinely defined as metabolomics. Metabolomics' studies based on untargeted mass spectrometry provide the capability to investigate metabolism on a global and relatively unbiased scale in comparison to traditional targeted studies focused on specific pathways of metabolism and a small number of metabolites. The untargeted approach enables the detection of thousands of metabolites in hypothesis-generating studies and links previously unknown metabolites with biologically important roles {% cite Patti2012 %}. There are two major issues in contemporary mass spectrometry-based metabolomics: the first is enormous loads of signal generated during the experiments, and the second is the fact that some metabolites in the studied samples may not be known to us. These obstacles make the task of processing and interpreting the metabolomics data a cumbersome and time-consuming process {% cite Nash2019 %}.

Many packages are available for the analysis of GC-MS or LC-MS data - for more details see the reviews by {% cite Stanstrup2019 %} and {% cite Misra2021 %}. In this tutorial, we focus on open-source solutions integrated within the Galaxy framework, namely **XCMS**, **RAMClustR**, **RIAssigner**, and **matchms**. In this tutorial, we will learn how to (1) extract features from the raw data using **XCMS** ({% cite Smith2006 %}), (2) deconvolute the detected features into spectra with **RAMClustR** ({% cite broeckling2014ramclust %}), (3) compute retention indices with **RIAssigner** ({% cite hecht2022riassigner %}), and (4) identify the present compounds leveraging spectra and retention indices using **matchms** ({% cite Huber2020 %}). For demonstration, we use three GC-[EI+] high-resolution mass spectrometry data files generated from quality control seminal plasma samples.


> <details-title> Seminal plasma samples </details-title>
> 
> The seminal plasma samples were analyzed according to the standard operating procedure [(SOP) for metabolite profiling of seminal plasma via GC Orbitrap](https://zenodo.org/record/5734331).
> The 3 samples used in this training are pooled quality control (QC) samples coming from about 200 samples. The pooled samples were analyzed in dilution series to test the system suitability and the quality of the assay.
>
{: .details}

To process the data, we use several tools. **XCMS** ({% cite Smith2006 %}) is a general package for untargeted metabolomics profiling. It can be used for any type of mass spectrometry acquisition (centroid and profile) or resolution (from low to high resolution), including FT-MS data coupled with a different kind of chromatography (liquid or gas). We use it to detect chromatographic peaks within our samples. Once we have detected them, they need to be deconvoluted into spectra representing chemical compounds. For that, we use **RAMClustR** ({% cite broeckling2014ramclust %}) tool. To normalise the retention time of deconvoluted spectra in our sample, we compute the retention index using **RIAssigner** ({% cite hecht2022riassigner %}) by comparing the data to a well-defined list of reference compound (commonly alkanes) analyzed on the same GC column. Finally, we identify detected spectra by aligning them with a database of known compounds. This can be achieved using **matchms** ({% cite Huber2020 %}), resulting in a table of identified compounds weighted by a matching score.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data preparation and prepocessing

Before we can start with the actual analysis pipeline, we first need to download and prepare our dataset. Many of the preprocessing steps can be run in parallel on individual samples. Therefore, we recommend using the Dataset collections in Galaxy. This can be achieved by using the dataset collection option from the beginning of your analysis when uploading your data into Galaxy.

## Import the data into Galaxy

> <hands-on-title> Upload data </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) into a collection:
>
>    ```
>    https://zenodo.org/record/7890956/files/8_qc_no_dil_milliq.raw
>    https://zenodo.org/record/7890956/files/21_qc_no_dil_milliq.raw
>    https://zenodo.org/record/7890956/files/29_qc_no_dil_milliq.raw
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md collection=true format="mzml" collection_name="input" renaming=false %}
>
> 3. Make sure your data is in a **collection**. You can always manually create the collection from separate files:
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
>    In the further steps, this dataset collection will be referred to as `input` (and we recommend naming this collection like that to avoid confusion).
>
> 4. Import the following extra files from [Zenodo]({{ page.zenodo_link }}):
>
>    ```
>    https://zenodo.org/record/7890956/files/reference_alkanes.csv
>    https://zenodo.org/record/7890956/files/reference_spectral_library.msp
>    https://zenodo.org/record/7890956/files/sample_metadata.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
> 
>    Please pay attention to the format of all uploaded files, and make sure they were correctly imported.
>
>    {% snippet faqs/galaxy/datatypes_understanding_datatypes.md %}
>
>    > <comment-title> The extra files </comment-title>
>    >
>    > The three additional files contain the **reference alkanes**, the **reference spectral library**, and the **sample metadata**. Those files are auxiliary inputs used in the data processing and contain either extra information about the samples or serve as reference data for indexing and identification.
>    > 
>    > The **list of alkanes** (`.tsv` or `.csv`) with retention times and carbon number or retention index is used to compute the retention index of the deconvoluted peaks. The alkanes should be measured in the same batch as the input sample collection.
>    > 
>    > The **reference spectral library** (`.msp`) is used for the identification of spectra. It contains the recorded and annotated mass spectra of chemical standards, ideally from a similar instrument. The unknown spectra which can be detected in the sample can then be confirmed via comparison with this library. The specific library is an in-house library of metabolite standards measured at RECETOX using a GC Orbitrap.
>    > 
>    > The **sample metadata** (`.csv` or `.tsv`) is a table containing information about our samples. In particular, the tabular file contains for each sample its associated sample name, class (QC, blank, sample, etc.), batch number, and injection order. It is possible to add more columns to include additional details about the samples.
>    {: .comment}
>
{: .hands_on}

As a result of this step, we should have in our history a green Dataset collection with three `.raw` files as well as three separate files with reference alkanes, reference spectral library, and sample metadata.

## Convert the raw data to mzML

Our input data are in `.raw` format, which is not suitable for the downstream tools in this tutorial. We use the tool {% tool [msconvert](toolshed.g2.bx.psu.edu/repos/galaxyp/msconvert/msconvert/3.0.20287.2) %} to convert our samples to the appropriate format (`.mzML` in this case).

> <hands-on-title> Convert the raw data to mzML </hands-on-title>
>
> 1. {% tool [msconvert](toolshed.g2.bx.psu.edu/repos/galaxyp/msconvert/msconvert/3.0.20287.2) %} with the following parameters:
>    - {% icon param-collection %} *"Input unrefined MS data"*: `input` (Input dataset collection)
>    - *"Do you agree to the vendor licenses?"*: `Yes`
>    - *"Output Type"*: `mzML`
>    - In *"Data Processing Filters"*:
>        - *"Apply peak picking?"*: `Yes` (This option will compute centroids in the m/z domain.)
>
>    > <comment-title> Centroids </comment-title>
>    >
>    > `msconvert` with selected parameters computes centroids in the m/z domain. MS instruments continuously sample and record signals. Therefore, a mass peak for a single ion in one spectrum consists of multiple intensities at discrete m/z values. Centroiding is the process of reducing these mass peaks to a single representative signal, the centroid. This results in much smaller file sizes without losing too much information. In the further steps, **XCMS** uses the _centWave_ chromatographic peak detection algorithm, which was designed for centroided data. That is the reason we perform the centroiding prior to chromatographic peak detection in this step.
>    {: .comment}
>
{: .hands_on}

## Create the XCMS object

The first part of data processing is using the **XCMS** tool to detect peaks in the MS signal. For that, we first need to take the `.mzML` files and create a format usable by the **XCMS** tool. {% tool [MSnbase readMSData](toolshed.g2.bx.psu.edu/repos/lecorguille/msnbase_readmsdata/msnbase_readmsdata/2.16.1+galaxy0) %} ({% cite gatto2012msnbase %}. {% cite gatto2020msnbase %}) takes as input our files and prepares `RData` files for the first **XCMS** step.

> <hands-on-title> Create the XCMS object </hands-on-title>
>
> 1. {% tool [MSnbase readMSData](toolshed.g2.bx.psu.edu/repos/lecorguille/msnbase_readmsdata/msnbase_readmsdata/2.16.1+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"File(s) from your history containing your chromatograms"*: `input.mzML` (output of **msconvert** {% icon tool %})
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
>    > <comment-title> Output - `input.raw.RData` </comment-title>
>    >
>    > Collection of `rdata.msnbase.raw` files. `Rdata` file that is necessary in the next step of the workflow. These serve for an internal R representation of **XCMS** objects needed in the further steps.
>    {: .comment}
{: .hands_on}

# Peak detection using XCMS

The first step in the workflow is to detect the peaks in our data using **XCMS**. This part, however, is covered by a [separate tutorial]({{ site.baseurl }}/topics/metabolomics/tutorials/lcms-preprocessing/tutorial.html). Although the tutorial is dedicated to LC-MS data, it can also be followed for our GC-MS data. Therefore, in this section, we do not explain this part of the workflow in detail but rather refer the reader to the dedicated tutorial. Please also pay attention to the parameter values for individual Galaxy tools, as these can differ from the referred tutorial and are adjusted to our dataset.

> <details-title> Skip this step </details-title>
> 
> Since this step is already covered in a [separate tutorial]({{ site.baseurl }}/topics/metabolomics/tutorials/lcms-preprocessing/tutorial.html), it is possible to skip it. Instead, you can continue with [Peak deconvolution]({{ site.baseurl }}/topics/metabolomics/tutorials/gc_ms_with_xcms/tutorial.html#peak-deconvolution) step using a preprocessed **XCMS** object file prepared for you.
>
> > <hands-on-title> Upload data </hands-on-title>
> >
> > 1. Create a new history for this tutorial
> >
> >    {% snippet faqs/galaxy/histories_create_new.md %}
> >
> > 2. Import the following files from [Zenodo]({{ page.zenodo_link }}):
> >
> >    ```
> >    https://zenodo.org/record/7890956/files/XCMS_object.rdata.xcms.fillpeaks
> >    ```
> >
> >    {% snippet faqs/galaxy/datasets_import_via_link.md %}
> >
> >    The format of uploaded file containing **XCMS** object should be `rdata.xcms.fillpeaks`.
> >
> >    {% snippet faqs/galaxy/datatypes_understanding_datatypes.md %}
> >
> {: .hands_on}
>
{: .details}

## Peak picking

The first step is to extract peaks from each of your data files independently. For this purpose, we use the _centWave_ chromatographic peak detection algorithm implemented in {% tool [xcms findChromPeaks (xcmsSet)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_xcmsset/abims_xcms_xcmsSet/3.12.0+galaxy0) %}.

> <hands-on-title> Peak picking </hands-on-title>
>
>  {% tool [xcms findChromPeaks (xcmsSet)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_xcmsset/abims_xcms_xcmsSet/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"RData file"*: `input.raw.RData` (output of **MSnbase readMSData** {% icon tool %})
>    - *"Extraction method for peaks detection"*: `CentWave - chromatographic peak detection using the centWave method`
>        - *"Max tolerated ppm m/z deviation in consecutive scans in ppm"*: `3.0`
>        - *"Min,Max peak width in seconds"*: `1,30`
>        - In *"Advanced Options"*:
>            - *"Prefilter step for for the first analysis step (ROI detection)"*: `3,500`
>            - *"Noise filter"*: `1000`
>  
>    You can leave the other parameters with their default values.
>
{: .hands_on}

## Determining shared ions

At this step, you obtain a dataset collection containing one `RData` file per sample, with independent lists of ions. Next, we want to identify the ions shared between samples. To do so, first, you need to group your individual `RData` files into a single one.

> <hands-on-title> Merging files </hands-on-title>
>
>  {% tool [xcms findChromPeaks Merger](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_merge/xcms_merge/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"RData file"*: `input.raw.xset.RData` (output of **xcms findChromPeaks (xcmsSet)** {% icon tool %})
>    - {% icon param-file %} *"Sample metadata file"*: `sample_metadata.tsv` (Input dataset)
>
>    You can leave the other parameters with their default values.
>
{: .hands_on}

Now we can proceed with the grouping and determining shared ions among samples. The aim of this step, called _grouping_, is to obtain a single matrix of ions’ intensities across all samples.

> <hands-on-title> Grouping peaks </hands-on-title>
>
>  {% tool [xcms groupChromPeaks (group)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"RData file"*: `xset.merged.RData` (output of **xcms findChromPeaks Merger** {% icon tool %})
>    - *"Method to use for grouping"*: `PeakDensity - peak grouping based on time dimension peak densities`
>        - *"Bandwidth"*: `3.0`
>        - *"Minimum fraction of samples"*: `0.9`
>        - *"Width of overlapping m/z slices"*: `0.01`
>
>    You can leave the other parameters with their default values.
>
{: .hands_on}

## Retention time correction

A deviation in retention time occurs from one sample to another, especially when you inject large sequences of samples. This step aims at correcting retention time drift for each peak among samples.

> <hands-on-title> Retention time correction </hands-on-title>
>
>  {% tool [xcms adjustRtime (retcor)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_retcor/abims_xcms_retcor/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"RData file"*: `xset.merged.groupChromPeaks.RData` (output of **xcms groupChromPeaks (group)** {% icon tool %})
>    - *"Method to use for retention time correction"*: `PeakGroups - retention time correction based on aligment of features (peak groups) present in most/all samples.`
>        - *"Minimum required fraction of samples in which peaks for the peak group were identified"*: `0.7`
>
>    You can leave the other parameters with their default values.
>
{: .hands_on}

## Second round of determining shared ions

By applying retention time correction, the used retention time values were modified. Consequently, applying this step to your data requires completing it with an additional _grouping_ step.

> <hands-on-title> Grouping peaks </hands-on-title>
>
>  {% tool [xcms groupChromPeaks (group)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"RData file"*: `xset.merged.groupChromPeaks.adjustRtime.RData` (output of **xcms adjustRtime (retcor)** {% icon tool %})
>    - *"Method to use for grouping"*: `PeakDensity - peak grouping based on time dimension peak densities`
>        - *"Bandwidth"*: `3.0`
>        - *"Minimum fraction of samples"*: `0.9`
>        - *"Width of overlapping m/z slices"*: `0.01`
>    - *"Get the Peak List"*: `Yes`
>        - *"Convert retention time (seconds) into minutes"*: `Yes`
>        - *"If NA values remain, replace them by 0 in the dataMatrix"*: `No`
>
>    You can leave the other parameters with their default values.
>
{: .hands_on}

## Integrating areas of missing peaks

At this point, the peak list may contain `NA` values when peaks were not considered in only some of the samples in the first peak-picking step. In this step, we will integrate the signal in the m/z-rt area of an ion (chromatographic peak group) for samples in which no chromatographic peak for this ion was identified.

> <hands-on-title> Integrating areas of missing peaks </hands-on-title>
>
>  {% tool [xcms fillChromPeaks (fillPeaks)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_fillpeaks/abims_xcms_fillPeaks/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"RData file"*: `xset.merged.groupChromPeaks.adjustRtime.groupChromPeaks.RData` (output of **xcms groupChromPeaks (group)** {% icon tool %})
>    - In *"Peak List"*:
>        - *"Convert retention time (seconds) into minutes"*: `Yes`
>
>    You can leave the other parameters with their default values.
>
>    > <comment-title> Output </comment-title>
>    >
>    > After the `fillChromPeaks` step, you obtain your final intensity table. At this step, you have everything mandatory to begin analysing
>    > your data:
>    >  - A *sampleMetadata* file (if not done yet, to be completed with information about your samples)
>    >  - A *dataMatrix* file (with the intensities)
>    >  - A *variableMetadata* file (with information about ions such as retention times, m/z) 
>    {: .comment}
>
{: .hands_on}

# Peak deconvolution

The next step is deconvoluting the detected peaks in order to reconstruct the full spectra of the chemical compounds present in the sample, separated by the chromatography and ionized in the mass spectrometer. {% tool [RAMClustR](toolshed.g2.bx.psu.edu/repos/recetox/ramclustr/ramclustr/1.3.0+galaxy0) %} is used to group features based on correlations across samples in a hierarchy, focusing on consistency across samples. While each feature is typically derived from a single compound and represents a fragment, the whole mass spectrum can be used to more accurately identify the precursor compound or molecular ion. **RAMClustR** uses a novel grouping method that operates in an unsupervised manner to group peaks into spectra without relying on the predicting in-source phenomena (in-source fragments or adduct formation) or fragmentation mechanisms.

> <details-title> RAMClust method </details-title>
> 
> The RAMclust similarity scoring utilises a Gaussian function, allowing flexibility in tuning correlational and retention time similarity decay rates independently, based on the dataset and the acquisition instrumentation. The correlational relationship between two features can be described by either MS-MS, MS-idMS/MS, or idMS/MS-idMS/MS values, and we support several correlation methods (e.g. Pearson’s method) to calculate similarity. These are parametrised by `sigma_t` and `sigma_r`, Gaussian tuning parameters of retention time similarity and correlational score, respectively, between feature pairs.
> 
> Similarities are then converted to dissimilarities for clustering. The similarity matrix is then clustered using one of the available methods, such as average or complete linkage hierarchical clustering. The dendrogram is cut using the `cutreeDynamicTree` function from the package `dynamicTreeCut`. For this application, the minimum module size is set to 2, dictating that only clusters with two or more features are returned, as singletons are impossible to interpret intelligently.
> 
> Cluster membership, in conjunction with the abundance values from individual features in the input data, is used to create spectra. The mass to charge ratio is derived from the feature m/z, and the abundance for each m/z in the spectrum is derived from the weighted mean of the intensity values for that feature. These spectra are then exported as an `.msp` formatted file, which can be directly imported by **NIST MSsearch**, or used as input for **MassBank** or **NIST msPepSearch** batch searching.
>
{: .details}

> <hands-on-title> Peak deconvolution </hands-on-title>
>
> 1. {% tool [RAMClustR](toolshed.g2.bx.psu.edu/repos/recetox/ramclustr/ramclustr/1.3.0+galaxy0) %} with the following parameters:
>    - *"Choose input format:"*: `XCMS`
>        - In *"Input MS Data as XCMS"*:
>            - {% icon param-file %} *"Input XCMS"*: `xset.merged.groupChromPeaks.adjustRtime.groupChromPeaks.fillChromPeaks.RData` (output of **xcms fillChromPeaks (fillPeaks)** {% icon tool %})
>        - In *"General parameters"*:
>            - *"Sigma r"*: `0.7`
>            - *"Maximum RT difference"*: `10.0`
>    - In *"Clustering"*:
>        - *"Minimal cluster size"*: `5`
>        - *"Maximal tree height"*: `0.9`
>
>    You can leave the other parameters with their default values.
>
{: .hands_on}

The spectral data comes as an `.msp` file, which is a text file structured according to the **NIST MSSearch** spectra format. `.msp` is one of the generally accepted formats for mass spectral libraries (or collections of unidentified spectra, so called spectral archives), and it is compatible with lots of spectra processing programmes (MS-DIAL, NIST MS Search, AMDIS, etc.). Because `.msp` files are text-based, they can be viewed as simple `txt` files:

> <hands-on-title> Data Exploration </hands-on-title>
>
> Click *"View data"* {% icon galaxy-eye %} icon next to the dataset in the Galaxy history. The contents of the file would look like this:
> 
> 
> {% snippet faqs/galaxy/datasets_icons.md %}
> 
> 
> ```
> NAME:C001
> IONMODE:Negative
> SPECTRUMTYPE:Centroid
> RETENTIONTIME:383.27
> Num Peaks:231
> 217.1073 64041926
> 243.0865 35597866
> 257.1134 31831229
> 224.061 27258239
> 258.11 24996353
> 241.0821 23957171
> 315.1188 13756744
> ...
>
> NAME:C002
> IONMODE:Negative
> SPECTRUMTYPE:Centroid
> RETENTIONTIME:281.62
> Num Peaks:165
> 307.1573 299174880
> 147.0654 298860831
> 149.0447 287809889
> 218.1066 118274758
> 189.076 112486871
> 364.1787 75134143
> 191.0916 52526567
> 308.1579 52057158
> ...
> ```
>
> > <details-title> Negative ion mode </details-title>
> >
> > You might wonder how can the ionisation mode (_IONMODE_) for GC-MS data be negative when using electron impact (EI+) ionization. This is, of course, incorrect. This is actually just a default behaviour of **RAMClustR**. We can optionally change this by providing **RAMClustR** an experiment definition file. This file can be created manually or using the {% tool [RAMClustR define experiment](toolshed.g2.bx.psu.edu/repos/recetox/ramclustr_define_experiment/ramclustr_define_experiment/1.0.2) %} tool. There we can specify annotations such as what instrument we used or ionisation mode (which was EI+ in our case), and this will be transfered to the `.msp` file. Finally, we can provide such a file as an input to **RAMClustR** in the _Extras_ inputs section.
> >
> {: .details}
>
{: .hands_on}

`.msp` files can contain one or more mass spectra, these are split by an empty line. The individual spectra essentially consist of two sections: metadata and peaks. The metadata consists of compound name, spectrum type (which is centroid in this case), ion mode, retention time, and the number of *m/z* peaks. If the compound has not been identified, as in our case, the **NAME** can be any arbitrary string. It is best, however, if that string is unique within that `.msp` file. The metadata fields are usually unordered, so it is quite common for one `.msp` file to contain **NAME** as the first metadata key and for another `.msp` to have this key somewhere in the middle. The keys themselves also aren't strictly defined and can have different names (e.g., **compound_name** instead of **NAME**). The `msp` format is inherently not standardized and various flavours exist.

However, as we can observe, the metadata part is rather incomplete. We would like to gather more information about the detected spectra and identify the specific compounds corresponding to them.

The second output file is the so called **Spec Abundance** table, containing the expression of all deconvoluted spectra across all samples. It is the corresponding element to the peak intensity table obtained from **XCMS**, only for the whole deconvoluted spectrum. This file is used for further downstream processing and analysis of the data and eventual comparisons across different sample types or groups. For more information on this downstream data processing see this [tutorial]({{ site.baseurl }}/topics/metabolomics/tutorials/lcms-dataprocessing/tutorial.html).

# Retention index calculation

The retention index ({% cite van1963generalization %}) is a way how to convert equipment- and experiment-specific retention times into system-independent normalised constants for GC-based experiments. The retention index of a compound is computed from the retention time by interpolating between the retention times adjacent alkanes which are assigned a fixed retention index. This can be different for the individual chromatographic system, but the derived retention indices are independent and allow comparing values measured by different analytical laboratories.

We use the package {% tool [RIAssigner](toolshed.g2.bx.psu.edu/repos/recetox/riassigner/riassigner/0.3.4+galaxy1) %} to compute retention indices for files in the `.msp` format using an indexed reference list of alkanes in `.csv` or `.msp` format. The output follows the same format as the input but with added retention index values. These can be used at a subsequent stage to improve compound identification ({% cite kumari2011applying %}). Multiple computation methods (e.g. piecewise-linear or cubic spline) are supported by the tool.

> <hands-on-title> Retention index calculation </hands-on-title>
>
> 1. {% tool [RIAssigner](toolshed.g2.bx.psu.edu/repos/recetox/riassigner/riassigner/0.3.4+galaxy1) %} with the following parameters:
>    - In *"Query dataset"*:
>        - {% icon param-file %} *"Query compound list"*: `Mass spectra from RAMClustR` (output of **RAMClustR** {% icon tool %})
>    - In *"Reference dataset"*:
>        - {% icon param-file %} *"Reference compound list"*: `reference_alkanes.csv` (Input dataset)
>
>    You can leave the other parameters with their default values.
>
>    > <comment-title> Minutes vs. seconds </comment-title>
>    >
>    > You might notice that in the last **XCMS** step, we converted the retention times to minutes. However, in this step, we are using seconds again. The reason is that **RAMClustR** converted them internally again to seconds. Regarding the reference compound list, this database already has its retention times in seconds.
>    {: .comment}
>
>    > <details-title> Kováts method </details-title>
>    >
>    > The Kováts retention index (or Kováts index) of a compound is its retention time normalized to the retention times of adjacently eluting n-alkanes. It is based on the fact that the logarithm of the retention time converges to the number of carbon atoms in the alkane. For an isothermal chromatogram, you use the following equation to calculate the Kováts index: 
>    > 
>    > $$I = 100 \times \frac{\mathtt{log}~t_x - \mathtt{log}~t_n}{\mathtt{log}~t_{n+1} - \mathtt{log}~t_n}$$ 
>    > 
>    > where $$t_n$$ and $$t_{n+1}$$ are retention times of the reference n-alkane hydrocarbons eluting immediately before and after chemical compound *X* and $$t_x$$ is the retention time of compound *X*.
>    > 
>    > In practice, first, you run a chromatogram of a standard alkane mixture in the range of interest. Then you do a co-injection of your sample with the standard alkanes. The main advantage of this method is reproducibility.
>    >
>    {: .details}
>
{: .hands_on}

# Identification

To annotate and putatively identify the deconvoluted spectra, we compare them with a reference spectral library. This library contains spectra of standards measured on the same instrument for optimal comparability. The **matchms** package is used for spectral matching. It was built to import and apply different similarity measures to compare large numbers of spectra. This includes common cosine scores but can also easily be extended with custom similarity measures. For more information on how to use matchms from Python check out this [blog](https://blog.esciencecenter.nl/build-your-own-mass-spectrometry-analysis-pipeline-in-python-using-matchms-part-i-d96c718c68ee).

> <details-title> Reference spectral library </details-title>
> 
> Fragmentation patterns of ions (spectra) are widely used in the process of compound identification for EI+ ionization in complex mixtures. The number of identifiable compounds and associated data keeps growing with the increasing sensitivity and resolution of mass spectrometers. These data are clustered into collections of chemical structures and their spectra, also called spectral libraries ({% cite stein1995chemical %}, {% cite stein2012mass %}), which can be used for fast, reliable identifications for any compound whose fragmentation pattern is measured by the instrument. 
> 
> To identify the compounds after GC/MS analysis, the *library searching* is performed for every detected spectrum. This locates the most similar spectra in the reference library, providing a list of the potential identifications sorted by computed similarity. Indeed, a library search should also yield the confidence of compound identification, often expressed as a similarity score.
>
> Reference libraries' users usually expect their reliability to be quite high. However, there are many sources of error which we need to be aware of. A particularly error-prone aspect is annotation. Even an excellent spectrum becomes worthless for a wrongly annotated compound. A variety of computer-aided quality control measures can help to improve these issues, but libraries always require some level of expert human decision-making before a spectrum is added to the library. This is also important for redundant and unidentified spectra.
>
{: .details}


## Compute similarity scores

We use the cosine score with a greedy peak pairing heuristic to compute the number of matching ions with a given tolerance and the cosine scores for the matched peaks. Compared to more traditional methods of forward and reverse matching, **matchms** is performing matching on peaks present both in the measured sample and the reference data. This inherently yields higher scores than the forward and reverse scoring used by NIST.

> <details-title> Cosine score methods </details-title>
>
> One of the most common methods to compute the similarity score between two spectra is computing the cosine of the *angle* between them. Each spectrum can be represented as a vector with an axis along each of its *m/z* values. Then the cosine of the angle is basically a normalised *dot product* of these two vectors. The intensity and *m/z* weighting corrections have been used to optimise performance. For example, higher mass peaks can be given greater weight since they are more discriminating than lower mass peaks. *Reverse* scores assign no weight to peaks present only in the measured sample spectrum under the assumption that they arise from impurities.
>
{: .details}

> <hands-on-title> Compute similarity scores </hands-on-title>
>
> 1. {% tool [matchms similarity](toolshed.g2.bx.psu.edu/repos/recetox/matchms_similarity/matchms_similarity/0.20.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Queries spectra"*: `RI using kovats of Mass spectra from RAMClustR` (output of **RIAssigner** {% icon tool %})
>    - *"Symmetric"*: `No` (if we were to query our spectra agains itself, we would select `Yes`)
>        - {% icon param-file %} *"Reference spectra"*: `reference_spectral_library.msp` (downloaded file from Zenodo)
>    - In *"Algorithm Parameters"*:
>        - *"tolerance"*: `0.03`
>    - *"Apply RI filtering"*: `Yes` (we use this since our data is GC)
>
>    You can leave the other parameters with their default values.
>
>    > <comment-title> Used reference spectra </comment-title>
>    > The RECETOX Metabolome HR-[EI+]-MS library is a collection of mostly endogenous compounds from the MetaSci Human Metabolite Library. Analytes underwent methoximation/silylation prior to acquisition. Spectra were acquired at 70 eV on Thermo Fisher Q Exactive™ GC Orbitrap™ GC-MS/MS at 60000 resolving power.
>    {: .comment}
>
{: .hands_on}

> <details-title> Overview of the spectral similarity scores </details-title>
>
> ## Cosine Greedy
> The cosine score, also known as the dot product, is based on representing the similarity of two spectra through the cosine of an angle between the vectors that the spectra produce. Two peaks are considered as matching if their *m/z* values lie within the given tolerance. Cosine greedy looks up matching peaks in a "greedy" way, which does not always lead to the most optimal alignments.
>
> This score was among the first to be used for looking up matching spectra in spectral libraries and, to this day, remains one of the most popular scoring methods for both library matching and molecular networking workflows.
>
> ## Cosine Hungarian
> This method computes the similarities in the same way as the *Cosine Greedy* but with a difference in *m/z* peak alignment. The difference lies in that the Hungarian algorithm is used here to find matching peaks. This leads to the best peak pairs match but can take significantly longer than the "greedy" algorithm.
>
> ## Modified Cosine
> Modified Cosine is another, as its name states, representative of the family of cosine-based scores. This method aligns peaks by finding the best possible matches and considers two peaks a match if their *m/z* values are within tolerance before or after a mass-shift is applied. A mass shift is essentially a difference of precursor-*m/z* of two compared spectra. The similarity is then again expressed as a cosine of the angle between two vectors.
>
> ## Neutral Losses Cosine
> Neutral Loss metric works similarly to all described above with one major difference: instead of encoding the spectra as "intensity vs *m/z*" vector, it encodes it to an "intensity vs *Δm/z*", where delta is computed as an *m/z* difference between precursor and a fragment *m/z*. This, in theory, could better capture the underlying structural similarities between molecules.
>
{: .details}

## Format the output

The output of the previous step is a `json` file. This format is very simple to read for our computers, but not so much for us. We can use the {% tool [matchms scores formatter](toolshed.g2.bx.psu.edu/repos/recetox/matchms_formatter/matchms_formatter/0.20.0+galaxy0) %} to convert the data to a tab-separated file with a scores matrix.

The output table contains the scores and number of matched ions of the deconvoluted spectra with the spectra in the reference library. The raw output can be filtered to only contain the top matches (3 by default) and/or to contain only pairs with a score and number of matched ions larger than provided thresholds.

> <hands-on-title> Format the output </hands-on-title>
>
> 1. {% tool [matchms scores formatter](toolshed.g2.bx.psu.edu/repos/recetox/matchms_formatter/matchms_formatter/0.20.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Scores object"*: `CosineGreedy scores` (output of **matchms similarity** {% icon tool %})
>
>    You can leave the other parameters with their default values.
>
{: .hands_on}

> | query | reference | CosineGreedy_score | CosineGreedy_matches |
> |-------|-----------|---------|--------|
> | C044  | Guanine_3TMS    | 0.987    | 19    |
> | C068  | Tryptophan_3TMS | 0.981    | 9   |
> | C053  | Norleucine_2TMS | 0.980    | 12   |
> | ...   | ...             | ... | ...  |
{: .matrix}

At this stage, all steps are complete: we have the list of identified spectra corresponding to a compound from the reference database. Since this process is dependent on the parameters we used along the way, the result should be considered dependent on them. To enumerate this dependency, each potential compound has been assigned a confidence score.

# Conclusion

In this tutorial, we show how untargeted GC-MS data can be processed to obtain tentative identification of compounds contained in the analyzed samples.
Using **XCMS**, we obtain a peak and intensity table from our files which contains information about the detected ions (such as m/z and retention time) and their intensities across all samples.
Afterwards, we used **RAMClustR** to reconstruct the fragmentation spectra which we assume to originate from the analyzed compounds contained within our samples.
We use a correlation based approach which allows us to reconstruct also low-abundant features as long as their intensities are correlating across samples.
This method is different than the sample-wise deconvolution using **CAMERA** ({% cite Kuhl2012 %}).
To have more information available for compound identification, we computed the retention index using **RIAssigner** and added it to our deconvoluted spectra.
We then finally compare the deconvoluted spectra against the reference library using **matchms**, using a retention index threshold to pre-filter our matches and then computing the cosine similarity for the spectra within the retention index tolerance.

This workflow leverages the correlation of features across samples to improve the annotation performance, trying to capture even low abundant signals.
It is suitable for both high- and low-resolution GC-MS data acquired in either profile or centroid mode by adapting the individual tool parameters or in-/excluding individual steps which are specific to a certain data format.

Before further statistical analysis, the information from the scores table can be included in the library of deconvoluted spectra manually to add the putative identifications. The further processing can then leverage this information and the intensity information contained in the **SpecAbundance** output from **RAMclustR**.

Note that this tutorial doesn't cover all steps which are possible or compatible with this workflow. Other options include the {% tool [matchms filtering](toolshed.g2.bx.psu.edu/repos/recetox/matchms_filtering/matchms_filtering/0.17.0+galaxy0) %} tool with which you can normalize your spectra or remove low-abundant ions before matching.
Another option for normalization outside of the built-in normalization tools of **RAMClustR** is the {% tool [WaveICA](toolshed.g2.bx.psu.edu/repos/recetox/waveica/waveica/0.2.0+galaxy2) %} tool for correction of batch effects.

tools_json:
["msconvert", "toolshed.g2.bx.psu.edu/repos/galaxyp/msconvert/msconvert/3.0.20287.2", "MSnbase readMSData", "toolshed.g2.bx.psu.edu/repos/lecorguille/msnbase_readmsdata/msnbase_readmsdata/2.16.1+galaxy0", "xcms findChromPeaks (xcmsSet)", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_xcmsset/abims_xcms_xcmsSet/3.12.0+galaxy0", "xcms findChromPeaks Merger", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_merge/xcms_merge/3.12.0+galaxy0", "xcms groupChromPeaks (group)", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy0", "xcms adjustRtime (retcor)", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_retcor/abims_xcms_retcor/3.12.0+galaxy0", "xcms fillChromPeaks (fillPeaks)", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_fillpeaks/abims_xcms_fillPeaks/3.12.0+galaxy0", "RAMClustR", "toolshed.g2.bx.psu.edu/repos/recetox/ramclustr/ramclustr/1.3.0+galaxy0", "RAMClustR define experiment", "toolshed.g2.bx.psu.edu/repos/recetox/ramclustr_define_experiment/ramclustr_define_experiment/1.0.2", "RIAssigner", "toolshed.g2.bx.psu.edu/repos/recetox/riassigner/riassigner/0.3.4+galaxy1", "matchms similarity", "toolshed.g2.bx.psu.edu/repos/recetox/matchms_similarity/matchms_similarity/0.20.0+galaxy0", "matchms scores formatter", "toolshed.g2.bx.psu.edu/repos/recetox/matchms_formatter/matchms_formatter/0.20.0+galaxy0", "matchms filtering", "toolshed.g2.bx.psu.edu/repos/recetox/matchms_filtering/matchms_filtering/0.17.0+galaxy0", "WaveICA", "toolshed.g2.bx.psu.edu/repos/recetox/waveica/waveica/0.2.0+galaxy2"]


topic: metabolomics
tutorial: lcms-dataprocessing
title: Mass spectrometry: LC-MS data processing
difficulty: introductory
time_estimation: 2H
requirements_json: []
tags_json: []
content_hash: c0c2cd0623af5aab787af13f915f965e
last_modified: 2026-05-15T22:28:13.786567
zenodo_link: https://zenodo.org/record/5179809
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/lcms-dataprocessing/tutorial.html

content:
Metabolomics is a *-omic* science known for being one of the most closely related to phenotypes.
It involves the study of different types of matrices, such as blood, urine, tissues, in various organisms including plants.
It focuses on studying the very small molecules which are called *metabolites*, to better understand matters linked to the metabolism.

Metabolomics analyses can be quite complex to conduct, especially when dealing with untargeted approaches.
**Liquid-Chromatography Mass Spectrometry** (LC-MS) is one of the three main technologies used to perform this kind of approach.
Data analysis for this technology requires a large variety of steps, ranging from extracting information from the raw data to statistical analysis and annotation.
One of these steps is called "data processing". It takes place after the pre-processing step (extraction of the peak list from raw data) and before any statistical analysis.
You can get an overview of a complete LC-MS untargeted metabolomic workflow by following [the dedicated training material]({% link topics/metabolomics/tutorials/lcms/tutorial.md %}).

After the pre-processing step, what you have at your disposal is a list of ions (hereafter in the form of a "variableMetadata" file) and the corresponding intensities ("dataMatrix" file).
What you may want now is to get some relevant information from your tables. However, your data may not be suitable yet for statistical analysis.
What should you do to ensure the quality of your tables? This tutorial will show you what the usual quality steps are.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Overview and data upload

Data processing covers a large range of actions. They can generally be described as transformation (*e.g.* normalisation) or filter (*e.g.* removal of unwanted ions).
In this tutorial we will focus on three main types of processing:
- Removing "trash" signals
- Correcting intensities
- Removing signals of insufficient quality

## Galaxy tools' workflow

The different Galaxy tools that will be used in this tutorial are given in the following workflow picture.

![The picture is composed of 3 main boxes linked by arrows from left to right. In each of the first and last boxes, labelled '3 tabulars', are found 3 files named sampleMetadata, variableMetadata and dataMatrix. In the middle box labelled 'Quality Control' is found the workflow used in this tutorial, represented as tiny boxes with tool names, linked with arrows following the tutorial order.](../../images/tutorial-lcms-proc-wf.png "The full tutorial workflow")

All these modules are part of the [Wokflow4Metabolomics](http://workflow4metabolomics.org/) tool suit ({% cite Giacomoni2014 %}, {% cite Guitton2017 %}).
They are compatible with the whole data analysis solution maintained by the W4M team.

> <comment-title>Workflow4Metabolomics public history</comment-title>
>
> This training material can be followed running it on any Galaxy instance holding the Galaxy modules needed.
> Nonetheless, if you happen to be a W4M user and do not want to run the hands-on yourself, please note that
> you can find the entire history in the 'published histories' section of usegalaxy.fr:
> [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing)
>
{: .comment}

## Dataset description

To illustrate the steps in this tutorial, a dataset has been built purposely.
It is composed of 3 files. They are text files of tables with tabulation as separator.

The *dataMatrix* file is a table containing the intensities of measured variables (ions, in lines) for every samples (in column).
The first column is for ions' identifiers while the first line is for samples' identifiers.

The *variableMetadata* file is a table containing information about the ions.
The first column is for ions' identifiers while the other columns gather information about m/z and retention time (rt).

The *sampleMetadata* file is a table containing information about the samples.
The first column is for samples' identifiers while the other columns gather analytical and biological information
such as the order of injection in the analytical sequence and the biological groups of interest for the supposed study.

The simulated design is composed of 30 biological samples (tagged "sample" in the *sampleType* column),
completed with 8 quality-control pooled samples (tagged "pool" in the *sampleType* column)
and 6 extraction solvent samples (tagged "blank" in the *sampleType* column).
The samples have been supposedly injected in two distinct sequences (tagged "B1" and "B2" in the *batch* column),
the injection order being given in the *injectionOrder* column.
Two sample characteristics are given:
- The *Group* column represents two groups "A" and "B", supposedly two biological groups (*e.g.* phenotypes, treatment groups...).
- The *Osmo* column represents a measurement of supposed osmolarity, imagining that the samples may be urine samples.

## Data upload

To perform the different exercices of this tutorial, you need to create a new history and upload the dedicated dataset.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the 3 starting files in your history. Two possibilities:
>    - Option 1: from a shared data library (ask your instructor)
>    - Option 2: from [Zenodo](https://zenodo.org/record/5179809) using the URLs given below:
>
>    ```
> https://zenodo.org/record/5179809/files/Dataprocessing_dataMatrix.txt
> https://zenodo.org/record/5179809/files/Dataprocessing_sampleMetadata.txt
> https://zenodo.org/record/5179809/files/Dataprocessing_variableMetadata.txt
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the datasets for their names to be shorter: "dataMatrix", "sampleMetadata", "variableMetadata".
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Check that the datatype is "tabular". If not, you may change it.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history,
> > this step corresponds to the datasets number 1 to 3.
> {: .tip}
>
{: .hands_on}

# Removing "trash" signals

Data are often affected by various sources of unwanted variability. This can be found in your tables in different ways,
two common ones being the presence of unwanted ions, and the effect of biological or analytical variables on intensity measures.
This unwanted information can limit the effectiveness of statistical methods, leading sometimes to difficulties in revealing investigated effects.
Thus, identifying such variability can help analysing your data at its full potential.
Yet, getting rid of such information may not be a trivial problem. It requires different elements to be completed successfully.

In this section we will adress the question of "trash" signal filtering.
By this we mean ions that are present in the extracted peak list we have, but that do not correspond to relevant compounds to analyse.
This can be for example noise, or ions from compounds that are not present in the original biological samples.

To make it clearer, we will illustrate this with two examples: a "simple" one first and a more advanced one in a second time.

## Filtering signals at given retention times

When using a chromatography column for MS analysis, you may want to exclude some time ranges where you know the information found there is not of interest.
For example, you may want to exclude the dead volume, a calibration zone at the begining or the end, or to exclude a column flush.

In this tutorial, let's suppose the data are from some LC-QTOF analysis with a dead volume between 0 and 0.4 minutes and a column flush from 16 minutes.
Then we may want to exclude ions that may be found at theses specific retention time (rt) ranges.
A quick check at the variableMetadata file reveals that a retention time column is available ("rt") with values in minutes.
We can then use this column to filter the dataset.

> <hands-on-title>Using <b>Generic_filter</b> to filter ions found at specific retention times</hands-on-title>
>
> 1. {% tool [Generic_Filter](toolshed.g2.bx.psu.edu/repos/melpetera/generic_filter/generic_filter/2020.01) %} with the following parameters:
>    - {% icon param-file %} *"Data Matrix file"*: the `dataMatrix` file
>    - {% icon param-file %} *"Sample metadata file"*: the `sampleMetadata` file
>    - {% icon param-file %} *"Variable metadata file"*: the `variableMetadata` file
>    - *"Deleting samples and/or variables according to Numerical values"*: `yes`
>        - In *"Identify the parameter to filter "*:
>            - *"1: Identify the parameter to filter "*
>                - *"On file"*: `Variable metadata`
>                - *"Name of the column to filter"*: `rt`
>                - *"Interval of values to remove"*: `extremity`
>                    - *"Remove all values lower than"*: `0.4`
>                    - *"And upper than"*: `16.0`
>    - *"Deleting samples and/or variables according to Qualitative values"*: `no`
>
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history,
>    > this step corresponds to the datasets number 4 to 6.
>    {: .tip}
>
{: .hands_on}

The **Generic Filter** {% icon tool %} tool generates 3 tables. They correspond to the 3 original tables,
except the content has been filtered according to the specified parameters.
By "filtering", it means removing from the dataset some variables (ions) and/or samples according to the defined filters.

> <question-title></question-title>
>
> What have changed between the 3 input tables and the 3 output ones?
>
> > <solution-title></solution-title>
> >
> >  The sampleMetadata output is identical to the input one since no filter on samples has been applied.
> >  The variableMetadata output contains less lines than the input one: some ions has been removed from the table.
> > The same happened with the dataMatrix table, corresponding ions being removed from the file.
> > Thus, these two files went from 201 lines (header included) to 172 lines, meaning only 171 ions remained in the dataset.
> >
> {: .solution}
>
{: .question}

This was a relatively easy task to do. The key point to perform the filter was to know the ranges of rt that needed to be filter,
since the information about rt values was already explicitly found in the dataset.
However, sometimes filtering requires more steps to be able to perform the wanted processing.
This will be illustrated with the next example.

## Using blanks to filter noise signals

As mentioned before, measured signals using mass spectrometry may not always be all of interest.
Some can be noise, or compounds not characteristic of the analysed biological samples.
There are several ways to reduce there impact on gathered data.
But one key point is always, as a starting point, to identify the issue.

In the previous example we knew there were retention time ranges of values where signals were not relevant.
But they may also be some signals that represent noise, found at no specific retention times.
So a question can be "how do we identify these signals?".

One possible procedure is the use of blanks to estimate the noise, as a reference.
The idea is to compare blanks’ intensities with other samples’ intensities.
If there is no subtantial difference, we can assume that the concerned signal is noise.

Of course, to be able to do so, you need to inject reference blanks along with your biological samples to get these intensities in your dataMatrix.
Thus, you need to anticipate it when you define your injection sequence.
Ideally, the blanks to use are extraction blanks, but you can also use injection solvent depending on your protocols.

When blanks are available in your dataset, another thing to consider is "how do I formalise the information needed to be able to filter?".
One common way to compare may be to set a minimum difference between means or medians, or to test for significant difference with a statistical test.
In this tutorial, we will choose to calculate a mean fold change between blanks and non-blank samples, and to set a threshold value for filtering.

The mean fold change ("fold") for each ion can be calculated using the **Intensity Check** {% icon tool %} tool.

> <hands-on-title>Using <b>Intensity Check</b> to generate the information needed to filter</hands-on-title>
>
> 1. {% tool [Intensity Check](toolshed.g2.bx.psu.edu/repos/melpetera/intensity_checks/intens_check/1.2.8) %} with the following parameters:
>    - {% icon param-file %} *"Data Matrix file"*: `GF_dataMatrix` (output of the previous **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `GF_sampleMetadata` (output of the previous **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `GF_variableMetadata` (output of the previous **Generic Filter** {% icon tool %} job)
>    - *"Computation method"*: `Between one class and all the remaining samples`
>        - *"Class column"*: `c5: sampleType`
>        - *"Selected class"*: `blank`
>        - *"Calculate the mean fold change"*: `Yes`
>            - *"Where should the class be placed for the mean fold change calculation?"*: `Denominator (Bottom)`
>
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history,
>    > this step corresponds to the datasets number 7 and 8.
>    {: .tip}
>
{: .hands_on}

This module generates two outputs: a pdf file for plots (that is not of interest in our example)
and a table corresponding to the variableMetadata file used as input, completed with new columns depending on the selected parameters.
In our case, it generated a column named *fold_Other_VS_blank* that we will use for filtering.

> <question-title></question-title>
>
> What does a value of "4" mean in the *fold_Other_VS_blank* column?
> Remember that we used as parameter *"Selected class"*=`blank` and
> *"Where should the class be placed for the mean fold change calculation?"*=`Denominator (Bottom)`.
>
> > <solution-title></solution-title>
> >
> > Since *"Selected class"*=`blank`, the samples are devided in two groups: the blank samples on one hand
> > and all the other samples on the other hand.
> > To calculate a mean fold change (*i.e.* a ratio of means) between the two classes, we need to define which mean will be used as numerator
> > and which one will be used as denominator. Since we defined that the selected class should be the denominator, the values we will get are
> > (mean of non-blank samples)/(mean of blank samples).
> > Thus, a value of "4" for a given ion means that the ion has a mean 4-times higher in non-blank samples compared to blank samples.
> >
> {: .solution}
>
{: .question}

The **Intensity Check** {% icon tool %} tool generates additional information about your dataset, but do not perform any filter.
We can now define a threshold for filtering, and use the **Generic Filter** {% icon tool %} tool again to remove noise signal.
Here we will use a threshold value of "4". What we want is to remove ions having a fold value lower than 4,
meaning that the mean values of biological samples' intensities for theses ions are not sufficiently high compared to blanks to be considered
to be resulting from relevant compounds.

> <hands-on-title>Using <b>Generic_filter</b> to filter ions with insuffisant mean contrast with blank samples</hands-on-title>
>
> 1. {% tool [Generic_Filter](toolshed.g2.bx.psu.edu/repos/melpetera/generic_filter/generic_filter/2020.01) %} with the following parameters:
>    - {% icon param-file %} *"Data Matrix file"*: `GF_dataMatrix` (output of the previous **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `GF_sampleMetadata` (output of the previous **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `IC_GF_variableMetadata` (output of the previous **Intensity Check** {% icon tool %} job)
>    - *"Deleting samples and/or variables according to Numerical values"*: `yes`
>        - In *"Identify the parameter to filter "*:
>            - *"1: Insert Identify the parameter to filter "*
>                - *"On file"*: `Variable metadata`
>                - *"Name of the column to filter"*: `fold_Other_VS_blank`
>                - *"Interval of values to remove"*: `lower`
>                    - *"Remove all values lower than"*: `4.0`
>    - *"Deleting samples and/or variables according to Qualitative values"*: `no`
>
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history, this step corresponds to the datasets number 9 to 11.
>    {: .tip}
>
{: .hands_on}

As for the previous use of **Generic Filter**, we now have a dataset filtered from noise signals.

> <question-title></question-title>
>
> What is the current stage of our training dataset?
>
> > <solution-title></solution-title>
> >
> > From a dataset containing originally 200 ions, the two successive filters lead to a dataset containing only 131 ions.
> > The number of samples did not change, so we still have 30 biological samples, 8 QC pools and 6 blanks.
> >
> {: .solution}
>
{: .question}

At that point we will not use the blank samples anymore. However, before removing them from the dataset, it is always interesting to
get an overview of the dataset including blanks.

## Filtered dataset overview

The idea here is to have a glance at what the dataset looks like at a macro scale.
For this tutorial, we consider that at this step what we have in our dataset are ions only resulting from compounds originally present in the biological samples.
Thus, there are some assumptions we can begin to make, that we can try to check using graphical tools.

Here, we will use a tool that is called **Quality Metrics** to have an overview of our dataset through the generation of a pdf file containing some plots.

> <hands-on-title>Using <b>Quality Metrics</b> to get an overview of the dataset</hands-on-title>
>
> 1. {% tool [Quality Metrics](toolshed.g2.bx.psu.edu/repos/ethevenot/qualitymetrics/quality_metrics/2.2.8) %} with the following parameters:
>    - {% icon param-file %} *"Data matrix file"*: `GF_GF_dataMatrix` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `GF_GF_sampleMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `GF_IC_GF_variableMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - *"Coefficient of Variation"*: `no`
>    - *"Advanced parameters"*: `Use default`
>
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history, this step corresponds to the datasets number 12 to 15.
>    {: .tip}
>
{: .hands_on}

This tool generates several outputs (some of them are going to be used in a later use of this module), however for now only the pdf output is of interest.

![Screenshot of the PDF output from Quality Metrics. It is a composition of several distinct plots. In particular there are a PCA plot and a total intensity per sample plot.](../../images/lcmsproc_QM.png "PDF output from Quality Metrics")

On the top left of the picture, we can see the two first components of a Principal Component Analysis (PCA) with the projection of samples on it, colored by sample type.
We can see that the main variability in the dataset distinguishes the blank samples (on the left in black) from the other ones (red for pools and green for samples).
This is awaited since blanks are supposed to have very low intensities compared to biological samples.
This observation is consistant with the top middle plot which represents the sum of intensities for each sample (plotting according to the injection order),
where blank samples have very low values.

This kind of plots is a good way to detect samples that may have abnormally low profiles:
if a sample is positioned at the same area as the blanks, it is suspicious and a special attention should be given to the concerned sample.

In this tutorial dataset, no atypical sample is observed, so no special attention needs to be paid on specific samples.

Since the blank samples are of no use anymore, we can remove them from the dataset.
Again, this can be done running the **Generic Filter** {% icon tool %} tool, using the *sampleType* column of the sampleMetadata table.

> <hands-on-title>Using <b>Generic_filter</b> to remove blank samples from the dataset</hands-on-title>
>
> 1. {% tool [Generic_Filter](toolshed.g2.bx.psu.edu/repos/melpetera/generic_filter/generic_filter/2020.01) %} with the following parameters:
>    - {% icon param-file %} *"Data Matrix file"*: `GF_GF_dataMatrix` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `GF_GF_sampleMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `GF_IC_GF_variableMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - *"Deleting samples and/or variables according to Numerical values"*: `no`
>    - *"Deleting samples and/or variables according to Qualitative values"*: `yes`
>        - In *"Removing a level in factor"*:
>            - *"1: Insert Removing a level in factor"*
>                - *"Name of the column to filter"*: `sampleType`
>                - *"Remove factor when"*: `blank`
>
>    > <comment-title></comment-title>
>    >
>    > The tabular outputs of the previous **Quality Metrics** job are not needed here.
>    > This is why what we use as input here are the outputs from the last **Generic Filter** job.
>    {: .comment}
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history, this step corresponds to the datasets number 16 to 18.
>    {: .tip}
>
{: .hands_on}

This step leads to a dataset containing 131 ions and 38 samples (from which 8 pools).

You may have noticed that from module to module, output names tend to become longer and longer.
To prevent very long and not-so-informative names due to successive use of modules, we highly recommand to regularly rename the outputs.
Here, we completed a first row of data processing with the filter of trash signals. Now is a good time for a little renaming.

> <hands-on-title>Rename the last 3 tables</hands-on-title>
>
> 1. Rename the tables for their names to be shorter: "RT_blank_Filter_dataMatrix", "RT_blank_Filter_sampleMetadata", "RT_blank_Filter_variableMetadata".
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
>
{: .hands_on}

Now we are ready to investigate another type of data processing: correcting intensities.

# Signal drift and batch effect correction

Here we will illustrate an example of data processing that is not about filtering, but about correcting.
Indeed, when it comes to performing statistics comparing biological samples, it is crucial for the intensity values used to be reflecting
relevant variability between samples.

In untargeted Metabolomics studies, we manipulate measures that are relative abundancies.
Although we have no unit attached to the intensities, we at least assume that, for a given ion, an intensity value for one sample being higher
than the one from another sample means that the compound from which the ion is generated is originally found in higher abundance in the biological sample from
the first sample compared to the other.
This assumption may seem trivial, but truth is it is not when dealing with LC-MS data.

It is known that when injecting successively a large number of samples, the LC-MS system tends to get dirty. This may cause a measure drift.
To prevent inability to catch signal anymore, in case of large injection series, the sequence is generally divided into several batches and the source is cleaned between batches.
Unfortunately, these signal drift and batch design can add significant variability in the data.
It makes sample comparison complicated, since the assumption stated previously about abundance in biological samples may not be true anymore.
In case data is impacted by these effects, we need a way to normalise the data to obtain something reliable for statistical analysis.

## [Optional step] Checking batch effect on data

You may have noticed in the **Quality Metrics** output PDF that appart from blanks, samples seemed to be seperated in two groups.
Truth is it is indeed. However, the best would be to confirm it, in particular by finding out what these groups are linked to.
Here dices are already rolled, and given the fact that in this section we are adressing signal drift and batch effects,
one could suppose that the groups may have something to do with it.

Let's confirm it by performing a PCA, with the specificity to colour the sample projections according to the supposed effect: the batch information.
This can be done using the **Multivariate** {% icon tool %} tool.

> <hands-on-title>Using <b>Multivariate</b> to get a coloured score plot from a PCA</hands-on-title>
>
> 1. {% tool [Multivariate](toolshed.g2.bx.psu.edu/repos/ethevenot/multivariate/Multivariate/2.3.10) %} with the following parameters:
>    - {% icon param-file %} *"Data matrix file"*: `RT_blank_Filter_dataMatrix` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `RT_blank_Filter_sampleMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `RT_blank_Filter_variableMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - *"Number of predictive components"*: `2`
>    - *"Advanced graphical parameters"*: `Full parameter list`
>        - *"Sample colors"*: `batch`
>        - *"Amount by which plotting text should be magnified relative to the default"*: `0.4`
>    - *"Advanced computational parameters"*: `Use default`
>
>    > <comment-title></comment-title>
>    >
>    > Changing the `0.8` value to `0.4` makes the text size of labels on the score plot smaller. Since the plot box size by default is tiny, this enhances the readability of the plot.
>    {: .comment}
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history, this step corresponds to the datasets number 19 to 22.
>    {: .tip}
>
{: .hands_on}

The output of **Multivariate** we will be interested in here is only the PDF file (Multivariate_figure.pdf).
Among the four plots displayed in the file, we can see at the bottom left corner that the first component clearly reveals two distinct groups.
These groups perfectly match the batch information we coloured the sample projections with.

This confirms that we indeed have at least a batch effect in the data.
To note, this comment is relevant thanks to the fact that supposedly we randomised the injection sequence.
This is essential to be able to efficiently separate any analytical effect from known biological variables when searching for effects.
If you look at the sampleMetadata file we originally imported, you can notice that the biological group (*A* and *B*) are equally found in the two batches,
with alternation all through the injection order.

## Performing the correction process

We previously saw that in data processing issues, it is sometimes crucial to anticipate what needs to be done before defining the injection sequence.
This is even more the case when dealing with batch effects and signal drifts.

There is a strategy, described initially by Van Der Kloet in 2009 ({% cite VdK2009 %}), that has made its way to nowadays procedures ({% cite Dunn2011 %}).
The idea is to model the signal drift and the batch effect level by using a reference that can be representative for every ions in the dataset.
Indeed, signal drift and batch effects can have very different impacts accross ions in a same dataset, so it is crucial to have a reference that is reliable
for each ion independantly.

Here the "universal" reference is obtained by using samples that are made by pooling together an extract of every samples in study.
Thus, these Quality-control pooled samples ('pools') contain all the compounds that are originally found in the samples, being a reference for a very wide range of ions.
By injecting these pools all through the injection sequence of the study samples, we obtain a reference for which the main variability observed is composed of
the analytical effects we want to correct, since biologically they are supposed to be identical.

> <details-title>More details about the theory</details-title>
>
> The procedure that is used to correct data in this tutorial is the following.
>
> For each ion independently, the normalisation process works as described in the following picture:
>
> ![An example plot with the normalisation formula](../../images/lcms_BC_theo.png "How this works")
>
> You can see a plot representing 6 sample measures (blue points) in a batch for a given extracted ion.
> The yellow line represents a model for the signal drift that can be used to normalise the data.
> This line is determined using the pools only (red squares).
> With the given formula, we can correct the signal drift.
>
> This work has to be done for each batch.
> Thus, if your sequence is divided into several batches, the idea is to obtain something similar to the following picture:
>
> ![A before/after plot showing an example of intensities before correction, with clear signal drift and batch effects, and after correction, where the effects have been erased thanks to the correction process](../../images/lcms_BC_theo2.png "Before/after picture")
>
{: .details}

In this tutorial, we have all the information we need to perform the correction:
- Pools representing the molecule diversity have been injected regularly all through the sequences.
- They are numerous enough in each batch for the regression to be reliable (well, at least we have enough of them to perform a linear modeling).
- The dataset contains the mandatory information needed in the sampleMetadata file: the injection order, the batches of analysis and the sample type (pool or sample).

We can then use the **Batch correction** {% icon tool %} tool to perform the correction.

> <hands-on-title>Using <b>Batch correction</b> to correct the data from signal drift and batch effet</hands-on-title>
>
> 1. {% tool [Batch_correction](toolshed.g2.bx.psu.edu/repos/melpetera/batchcorrection/Batch_correction/3.0.0) %} with the following parameters:
>    - {% icon param-file %} *"Data Matrix file"*: `RT_blank_Filter_dataMatrix` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `RT_blank_Filter_sampleMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `RT_blank_Filter_variableMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - *"Type of regression model "*: `linear`
>        - *"Factor of interest "*: `Group`
>        - *"Level of details for plots "*: `complete`
>
>    > <comment-title></comment-title>
>    >
>    > The tabular outputs of the previous **Multivariate** job are not needed here.
>    > This is why what we use as input here are the outputs from the last **Generic Filter** job.
>    {: .comment}
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history, this step corresponds to the datasets number 23 to 25.
>    {: .tip}
>
{: .hands_on}

The tool generates 3 outputs:
- a dataMatrix file containing the corrected intensities
- a variableMetadata file corresponding to the input one completed with additional columns
- a pdf file with graphics to visualise the impact of the correction process

Depending on the regression model option chosen (linear/lowess/loess/all_loess_pool/all_loess_sample), the output details differ slightly.
Indeed, the previously described algorithm has been adapted following two strategies: one for the linear/lowess/loess (3L) options,
another for the all_loess_pool/all_loess_sample (all_loess) ones.

The main differences between the two strategies are the following:
- one part allows regression type choice and applies the normalisation depending on internal quality metrics
- the other part offers the possibility to apply the correction without pool samples.
For more information, do not hesitate to read the help section provided with the Galaxy tool.

In this tutorial example, we used the *linear* option which is part of the 3L strategy.
We could not use any of the other options because the other ones would need more pools per batch than what we have here.
When using the 3L strategy, in the PDF file you can have various graphics depending on the *Level of details for plots* option you choose.
Here we generated all the plots that are available. This means that the PDF file contains first one page per ion with before/after plots,
plus 3 additional pages at the end that display overall before/after indicators (sum of intensities, PCA plots, pools' coefficients of variation).

> <question-title></question-title>
>
> Look at the penultimate page of the PDF output (page n°133).
> This page represents before/after PCA plots with various colouring.
> We will only focus on the two first lines, in particular looking at the first column.
> The two corresponding plots represent PCA score plots before (first line) and after (second line) the correction process.
> The two plots are coloured according to the batch information (B1 and B2).
> What can you say about the batch effect after the correction process compared to before?
>
> > <solution-title></solution-title>
> >
> > Compared to the batch effect that was highlighted as the main effect in the data before the correction process,
> > we can see that after correction, the batch effect is no longer explaining the first PCA component.
> > This result is awaited since the reduction of the batch effect is one of the aim of the Batch correction step.
> >
> {: .solution}
>
{: .question}

Now that we corrected the data from the signal drift and batch effect, we consider that differences in intensity between samples
can be attributed mostly to differences in concentration of original compounds.
However, we know that what we converted to intensity values in our table is initially signal measurements, thus we suspect that
intensities may be noisy at some point.

# Filtering signals of insufficient quality

Similarly to many measuring devices, values that are generated by LC-MS devices are subject to noise.
In the specific case of untargeted Metabolomics, the values are relative and can have, in a single dataset, a very large range of scales.
It is difficult to quantify the noise, even so it is crucial to have a minimum confidence when searching for subtile differences between groups of individuals.
Thus, having a way to put appart variables (ions) that may be of insufficient quality is key to limit the risk of wrong conclusion while analysing the data.

Beyond the issue of noise, the quality of the variables in one's dataset is a key question. In particular, in untargeted Metabolomics the data has particularities that
make the endpoint analyses complicated, such as the disproportion between sample size and variable number, the complex redundancy and the noise issue already mentioned.
When quality indicators can be computed before any statistical analysis, it is a real advantage: it can improve the efficiency of later statistical analysis and
also help to select good candidates in biomarker approaches.

One common and powerful way to assess measurements quality is the use of pools.
Indeed, now that the data is corrected for signal drift, we expect to have stable intensities within pools since there is no biological variability among them.
We can evaluate the remaining variability in pools for each ion, and consider it to be noise and/or analytical effects that we may not have managed to remove for specific ions.
This enables the use of quality thresholds to get rid of ions we consider not to be sufficiently reliable regarding intensity values.

In this tutorial, we will illustrate the use of pools by computing coefficients of variation (CV).
This metric allows to evaluate the variability accross samples independantly of the unit scale.
This advantage enables the definition of a unique threshold value that can be applied to every ions whatever the scales.
CV are computed for each ion in your dataset, so you can filter ions using the chosen threshold.
Note: the greater the CV value, the greater the variability.

The first use of CV we will illustrate here is the computation of CVs for the pools only. The idea here is to grasp the pool variability.
It is expected that pools do not vary much since they correspond to the same biological content.
Thus, if your pools vary too much, you can suspect that the corresponding ion is not reliable enough.

The second use of CV we will use here is a comparison between pool CVs and sample CVs.
Indeed, we expect sample CVs to be higher than pool CVs since they share the same noise and analytical variability but also carry biological variability.

The **Quality Metrics** {% icon tool %} tool provides different CV indicators depending on what is in your sample list.
In particular, in the present case-study, it computes pool CVs along with a ratio between pool CVs and sample CVs.
These two indicators are automatically calculated as long as the input datasets provide the needed information
(a *sampleType* column in the sampleMetadata file with the information of *pool* and *sample*).
Thus, for these two indicators the tool can be run without specific parameters to set.

> <hands-on-title>Using <b>Quality Metrics</b> to compute the CV indicators</hands-on-title>
>
> 1. {% tool [Quality Metrics](toolshed.g2.bx.psu.edu/repos/ethevenot/qualitymetrics/quality_metrics/2.2.8) %} with the following parameters:
>    - {% icon param-file %} *"Data matrix file"*: `BC_linear_RT_blank_Filter_dataMatrix` (output of the **Batch_correction** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `RT_blank_Filter_sampleMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `BC_linear_RT_blank_Filter_variableMetadata` (output of the **Batch_correction** {% icon tool %} job)
>    - *"Coefficient of Variation"*: `no`
>    - *"Advanced parameters"*: `Use default`
>
>
>    > <comment-title></comment-title>
>    >
>    > You may have noticed that we already used this tool previously in the tutorial.
>    > Although CV columns were already available at that time, here you need to use this tool again since this time indicators will be computed on intensities
>    > corrected from the signal drifts and batch effects.
>    > Note that what we are going to use this time is the tabular output, but while you are at it you can always check the pdf file if you want.
>    {: .comment}
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history, this step corresponds to the datasets number 26 to 29.
>    {: .tip}
>
{: .hands_on}

The tool provides a variableMetadata tabular output, containing all the computed CV values.
For now no filter has been applied on the dataset. To remove the ions of insufficient quality, you will need the **Generic Filter** {% icon tool %} tool again.

But first, you need to determine the thresholds you want to use.
Here we have two indicators we want to use. One is the pool CVs, the other is the ratio between pool CVs and sample CVs.
This means that we will need to set two distinct thresholds, and to add two numerical filters in **Generic Filter** {% icon tool %}.

> <hands-on-title>Using <b>Generic Filter</b> to filter the dataset</hands-on-title>
>
> 1. {% tool [Generic_Filter](toolshed.g2.bx.psu.edu/repos/melpetera/generic_filter/generic_filter/2020.01) %} with the following parameters:
>    - {% icon param-file %} *"Data Matrix file"*: `BC_linear_RT_blank_Filter_dataMatrix` (output of the **Batch_correction** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `QM_RT_blank_Filter_sampleMetadata` (output of the last **Quality Metrics** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `QM_BC_linear_RT_blank_Filter_variableMetadata` (output of the last **Quality Metrics** {% icon tool %} job)
>    - *"Deleting samples and/or variables according to Numerical values"*: `yes`
>        - In *"Identify the parameter to filter "*:
>            - *"1: Insert Identify the parameter to filter "*
>                - *"On file"*: `Variable metadata`
>                - *"Name of the column to filter"*: `poolCV_over_sampleCV`
>                - *"Interval of values to remove"*: `upper`
>                    - *"Remove all values upper than"*: `1.0`
>            - {% icon param-repeat %} *"2: Insert Identify the parameter to filter "*
>                - *"On file"*: `Variable metadata`
>                - *"Name of the column to filter"*: `pool_CV`
>                - *"Interval of values to remove"*: `upper`
>                    - *"Remove all values upper than"*: `0.3`
>    - *"Deleting samples and/or variables according to Qualitative values"*: `yes`
>        - In *"Removing a level in factor"*:
>            - *"1: Insert Removing a level in factor"*
>                - *"Name of the column to filter"*: `sampleType`
>                - *"Remove factor when"*: `pool`
>
>
>    > <comment-title></comment-title>
>    >
>    > Here, we took the opportunity of this filter to remove the pools from the dataset.
>    > Indeed, pools will be no longer used in this tutorial's steps, so we can remove them from the dataset.
>    {: .comment}
>
>
> 2. Rename the output tables for their names to be shorter: "Filtered_dataMatrix", "Filtered_sampleMetadata", "Filtered_variableMetadata".
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history, this step corresponds to the datasets number 30 to 32.
>    {: .tip}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What does the *0.3* threshold mean in the hands-on exercise you just executed?
> 2. What does the *1.0* threshold mean in the hands-on exercise you just executed?
> 3. How many variables are left in your dataset? How many samples?
>
> > <solution-title></solution-title>
> >
> > 1. The *0.3* value corresponds to the maximum value kept in the dataset ('Interval of values to remove: *upper*') regarding the
> >    *pool_CV* column in your *Variable metadata* file.
> >    Thus, ions with pool CV values superior to 0.3 are excluded from the dataset.
> >    To note, pool CV values are commonly considered as reflecting unstable ions when superior to 0.3.
> > 2. The *1.0* value corresponds to the maximum value kept in the dataset ('Interval of values to remove: *upper*') regarding the
> >    *poolCV_over_sampleCV* column in your *Variable metadata* file. This means that any ion with a pool CV / sample CV ratio above 1
> >    (*i.e.* a pool CV greater than the sample CV) is discarded from the dataset.
> > 3. Filtering led to 102 ions and 30 samples.
> >
> {: .solution}
>
{: .question}

Presented threshold values here are common thresholds in Metabolomics studies.
Please note that thresholds should be chosen consistently regarding the LC-MS device caracteristics and the study objectives.
Thus you may need to be more strict in some cases and less strict in others.
However, being less strict than usual should always be appropriately justified.

# Study-specific data processing: example normalising data according to a non-analytical effect

The previous steps presented in this tutorial illustrated categories of data processing that need to be consider whatever the study in
untargeted LC-MS Metabolomics projects. In some cases the decision may be not to take actions, but in all cases the issue has to be considered
to apprehend the reliability of the data for further analyses.

Nonetheless, each study is unique and specific issues can arise that would need some additional data processing steps that are very specific to the study.
Thus, sticking to a standard data processing workflow may not be sufficient in some cases.
For example, one commonly considered data processing step in biomarker discovery studies is the reduction of redundancy,
which is a complex question that can be addressed using various strategies depending on the study objectives.

In this section, we will illustrate an example of study-specific data processing that involves a normalisation of the dataset.
This will follow the standard *Identify->Formalise->Handle* design that is common to any data processing approach.

## Identifying unwanted variability

First thing is of course to identify an issue in the data that needs to be handled.
In the previous steps, this was the "easy" part since we already had knowledge about what needed to be handled.
We adressed well-known problems, for which solutions may not always be easy, but at least we know there is something.

In study-specific data processing, it may not be that simple.
Usually, to see something you need to search for it. This means you have hypotheses prior to the data analysis.
However, sometimes unwanted variability is unexpected, making the search process difficult.

Fortunately, whether you have hypotheses or not, there is a common starting point in Metabolomics that can help you see whether an issue seems to need special attention.
This is the use of PCA, that we already saw in previous steps to highlight awaited effects.

In this tutorial, we constructed the dataset so we already know what is the variability to identify.
In a real project it can also be the case, or at least we can have hypotheses about effects we would not be supprised to observe.
In this tutorial example, we considered samples as being urine samples.
It is known that urine samples can vary a lot in total solute concentration, potentially limiting the possibility to detect specific biological effects.
Thus, even if it may not be the case, we can wonder whether the variability of concentration may have a huge effect on the metabolomic dataset we have.
This is why, while making a PCA, we may want to plot the first components and see whether the variability displayed seems to match the concentration.

In the sampleMetadata file provided in this tutorial, you can find a column named *Osmo* that is meant to correspond to measures of osmolality,
one way to represent the overall concentration of urine samples.
Let's try to colour the PCA score plots according to that variable. For this, we will use the **Multivariate** {% icon tool %} Galaxy module.

> <hands-on-title>Using <b>Multivariate</b> to visualise the two first components of a PCA</hands-on-title>
>
> 1. {% tool [Multivariate](toolshed.g2.bx.psu.edu/repos/ethevenot/multivariate/Multivariate/2.3.10) %} with the following parameters:
>    - {% icon param-file %} *"Data matrix file"*: `Filtered_dataMatrix` (output of the last **Generic_Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `Filtered_sampleMetadata` (output of the last **Generic_Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `Filtered_variableMetadata` (output of the last **Generic_Filter** {% icon tool %} job)
>    - *"Number of predictive components"*: `2`
>    - *"Advanced graphical parameters"*: `Full parameter list`
>        - *"Sample colors"*: `Osmo`
>        - *"Amount by which plotting text should be magnified relative to the default"*: `0.4`
>    - *"Advanced computational parameters"*: `Use default`
>
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history, this step corresponds to the datasets number 33 to 36.
>    {: .tip}
>
{: .hands_on}

Among the four output files generated by the tool, the one we are interested in is the *Multivariate_figure.pdf* file.
The PCA scores plot representing the projection of samples on the two first components is located at the bottom left corner:

![The picture represents the PCA scores plot of components t1 (32% of total variability) and t2 (9% of total variability) with sample labels coloured according to the values from the Osmo column of the sampleMetadata file. We observe a gradation in the colours of samples following the first bisector of the graphic.](../../images/lcmsproc_osmo.png "PCA scores plot")

Taking into account that the PCA was computed on Unit-Variance-scaled data (see the **Multivariate** {% icon tool %} tool's help section for 

tools_json:
["Generic_Filter", "toolshed.g2.bx.psu.edu/repos/melpetera/generic_filter/generic_filter/2020.01", "Intensity Check", "toolshed.g2.bx.psu.edu/repos/melpetera/intensity_checks/intens_check/1.2.8", "Quality Metrics", "toolshed.g2.bx.psu.edu/repos/ethevenot/qualitymetrics/quality_metrics/2.2.8", "Multivariate", "toolshed.g2.bx.psu.edu/repos/ethevenot/multivariate/Multivariate/2.3.10", "Batch_correction", "toolshed.g2.bx.psu.edu/repos/melpetera/batchcorrection/Batch_correction/3.0.0", "Normalization", "toolshed.g2.bx.psu.edu/repos/marie-tremblay-metatoul/normalization/normalization/1.0.7"]


topic: synthetic-biology
tutorial: basic_assembly_analysis
title: Designing plasmids encoding predicted pathways by using the BASIC assembly method
difficulty: intermediate
time_estimation: 15M
requirements_json: []
tags_json: []
content_hash: 15d0180d372e96245fdc2717e64df7ee
last_modified: 2026-05-15T22:28:17.273329
zenodo_link: https://zenodo.org/record/6123887
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/synthetic-biology/tutorials/basic_assembly_analysis/tutorial.html

content:
Synthetic biology is a novel engineering discipline which requires computational tools for the design of metabolic pathways for the production of chemicals such as SynBioCAD portal which is the first Galaxy set of tools for synthetic biology and metabolic engineering ({% cite Hrisson2022 %}).

In this tutorial, we will use a set of tools from the **Genetic Design - BASIC Assembly Workflow** (https://galaxy-synbiocad.org) which will enable you to design plasmids implementing metabolic pathways for the bioproduction of lycopene in _E.coli_ (one of the preferred host cell for microbial biochemicals production).

Lycopene is a potent antioxidant and has been widely used in the fields of pharmaceuticals, nutraceuticals, and cosmetics. It's widely found in fruits including tomato, watermelon, guava, and papaya but the extraction method of lycopene from these natural sources is expensive, complicated and cannot match the large market demand.

To address this demand, synthetic biology and metabolic engineering have been employed to develop microbial cell factories (e.g. _E.coli_ strains) for lycopene production.

To design plasmids encoding lycopene bioproducing pathways, we will use the BASIC assembly method (as described in {% cite Storch2015 %}) which relies on orthogonal linkers and type IIs restriction enzyme cleavage to provide a robust and accurate assembly of DNA parts into plasmid constructs. From these construct definition, the workflow will generate scripts enabling the automatic build of the plasmids as well as the transformation of strains using an Opentrons liquid handler robot. After downloading these scripts onto a computer connected to an Opentrons, one can perform the automated construction of the plasmids at the bench.

The workflow scheme we will use is shown below. First, we will run the steps of this workflow individually so as not to neglect the understanding of the intermediate steps as well. Then, we will run the workflow automatically so that it itself retrieves the outputs from the previous step and gives them as input to the next tool.

![Genetic Design - BASIC Assembly Workflow](../../images/basic_assembly_workflow.png)

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data Preparation

First we need to upload and prepare the following inputs to analyze:

- One SBML (Systems Biology Markup Language) file modeling a heterologous pathway producing lycopene such as those produced by the Pathway Analysis Workflow (https://galaxy-synbiocad.org).

- The `parts_for_lycopene.csv` file listing the parts to be used (linkers, backbone and promoters) in the constructions.

- Two YAML files providing two examples of settings, i.e. providing the identifiers of the laboratory equipment and the parameters to be used in the Opentrons scripts.

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial named *Genetic Design - BASIC Assembly Analysis*.
> 2. Import the input files from [Zenodo]({{ page.zenodo_link }}):
>
>    ```
>    https://zenodo.org/record/6123887/files/dnabot_london_settings.yaml
>    https://zenodo.org/record/6123887/files/dnabot_paris_settings.yaml
>    https://zenodo.org/record/6123887/files/parts_for_lycopene.csv
>    https://zenodo.org/record/6123887/files/rp_002_0011.xml
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the datasets if you want to shorten the file names.
>    Rename `rp_002_0011.xml` to `pathway.xml`.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

# Find enzymes using **Selenzyme**

At first, a pathway generated by the **Pathway Analysis workflow** (https://galaxy-synbiocad.org) is provided as input to the Selenzyme tool ({% cite Carbonell2018 %}). Selenzyme searches for enzymes corresponding to each reaction of the pathway. It performs a reaction similarity search in the reference reaction database [MetaNetX](https://www.metanetx.org) and outputs an updated SBML file annotated with the enzyme UniProt IDs.
The tool provides several scores that can be combined in order to define an overall score. Scores are given for reaction similarity, conservation based on a multiple sequence alignment, phylogenetic distance between source organism and host, and additional scores calculated from sequence properties as shown in the example bellow.

![Selenzyme Results](../../images/selenzy_tool.png)

> <hands-on-title>Annotate enzymes with Uniprot IDs from a SBML pathway</hands-on-title>
>
> 1. Run {% tool [Selenzyme](toolshed.g2.bx.psu.edu/repos/tduigou/selenzy/selenzy-wrapper/0.2.0) %} with the following parameters:
>    - {% icon param-file %} *"Pathway (SBML)"*: Select `pathway.xml` from your current history
>    - In *"Advanced Options"*:
>        - *"Host taxon ID"*: Leave the default value `83333`. This stands for using E. coli as the chassis host.
>        - *"Comma separated taxon IDs of output enzyme sequences"*: *enter `553` which is the taxon ID of the Pantoea ananatis strain from which we want to extact enzymes*
>
>    > <comment-title></comment-title>
>    >
>    > This tool will produce annotated pathway with UniProt IDs in XML and CSV format.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How are identified the enzyme in the SBML file?
>
> > <solution-title></solution-title>
> >
> > 1. Enzyme are identified by their UniProt ID
> >
> {: .solution}
>
{: .question}

# Generate potential constructs from SBML file using **BasicDesign**

Second, the BasicDesign tool extracts enzyme IDs contained in the SBML (previously produced by Selenzyme) to generate genetic constructs compliant with the BASIC assembly approach. It uses as input an SBML file annotated with enzyme IDs for each reaction, and optionally one or several CSV files listing by their IDs the linkers, the promoters and the backbone used (`parts_for_lycopene.csv`). Example, below:

*id,type,sequence,comment*\
*L1,neutral linker,,*\
*L2,neutral linker,,*\
*L3,neutral linker,,*

For **linkers**, the type annotation should be one of *neutral linker*, *methylated linker*, *peptide fusion linker* or *RBS linker*. For **user parts**, type should be one of *backbone* or *constitutive promoter*. Other type will raise a warning and will be omited.

BasicDesign converts the SBML file into CSV files describing the DNA-parts to be included into each construct (in an operon format, i.e. with only one promoter) and enumerate possible combinations of promoters, RBSs and enzymes into constructs. Depending on the numbers of enzymes per reaction, of RBSs and promoters available, and whether or not to perform CDS permutation within the operon, the number of constructs may vary.

> <hands-on-title>Generate genetic constructs from the previously annotated SBML file</hands-on-title>
>
> 1. Run {% tool [BasicDesign](toolshed.g2.bx.psu.edu/repos/tduigou/rpbasicdesign/rpbasicdesign/0.3.4) %} with the following parameters:
>    - {% icon param-file %} *"rpSBML file": `uniprot_ids` (output of **Selenzyme** {% icon tool %} in xml format)*
>    - In *"Advanced Options"*:
>        - {% icon param-file %} *"Linkers and user parts": Select `parts_for_lycopene.csv` from your current history in csv format*
>
>    > <comment-title></comment-title>
>    >
>    > This tool will output a CSV constructs file (listing the constructs to be built), where each row corresponds to one construct and consists of a sequence of BASIC linker and DNA part IDs. The two other outputted CSV files provide the plate coordinates of (i) the BASIC linkers and (ii) the DNA-parts that the user will need to provide. Additionally, one SBOL (Synthetic Biology Open Language) file is produced for each construct generated. These files can be downloaded and visualized using online tools such as [VisBOL](https://visbol.org/).
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many constructs did you get ?
>
> > <solution-title></solution-title>
> >
> > 1. 88 construct designs were generated in CSV and SBOL format.
> >
> {: .solution}
>
{: .question}

# Generating python scripts using **DNA-Bot**

 In the last step, the DNA-Bot tool ({% cite Storch2020 %}) reads the list of constructs (previously produced by BasicDesign) and the DNA-parts position on the source plates and generates a set of python scripts to drive an Opentrons liquid handling robot for building the the plasmids. Optional parameters can be set by the user to define the plastic labwares to be used, and set protocol parameters such as washing or incubation times for purification step (`dnabot_paris_settings.yaml`).

> <hands-on-title>Generate DNA Bot python scripts</hands-on-title>
>
> 1. Run {% tool [DNA-Bot](toolshed.g2.bx.psu.edu/repos/tduigou/dnabot/dnabot/3.1.0) %} with the following parameters:
>    - {% icon param-file %} *"Source Construct"*: `Constructs` (output of **BasicDesign** {% icon tool %})
>    - {% icon param-files %} *"Plate files"*: Select these two files: `User parts plate` (output of **BasicDesign** {% icon tool %}), `Biolegio plate` (output of **BasicDesign** {% icon tool %}) with **Multiple datasets**.
>    - In *"Advanced Options"*:
>        - {% icon param-file %} *"Yaml file providing labware IDs and parameter to be used (Optionally)"*: Select either `dnabot_paris_settings.yaml` or `dnabot_london_settings.yaml` from your history)
>
>{% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>
>    > <comment-title></comment-title>
>    >
>    > This tool will output DNA-Bot scripts in tar format. You need to download it and decompress the archive. After downloading these scripts onto a computer connected to an Opentrons, one can perform the automated construction of the plasmids at the bench. Additional metadata meaningful to keep track of parameters are also outputted by the tool.
>    > ![Dnabot Outputs](../../images/dnabot_outputs.png)
>    > ![BASIC assembly workflow](../../images/BASIC_ASSEMBLY_WF.png)
>    {: .comment}
{: .hands_on}

> <question-title></question-title>
>
> 1. Looking at the scripts names, can you figure out the main steps of DNA-Bot ?
>
> > <solution-title></solution-title>
> >
> > 1. DNA-Bot outputs python scripts that implement the 4 assembly steps: clip reactions, purification, assembly and strain transformation. In short, the **Clip reactions** step prepares the mixes for the ligation of the individual DNA parts with the linkers; the **Purification** step purifies the linker-ligated DNA parts using magnetic beads and the Opentrons magnetic module; the **Assembly** step mixes the DNA purified parts to build the final constructs; while the **Transformation** step transforms the chassis micro-organism with the plasmid and inoculates onto agar.
> >
> {: .solution}
>
{: .question}

# Run the **Genetic Design - BASIC Assembly Workflow**

In this section, you can run the Genetic Design - BASIC Assembly Workflow more easily and fastly following these instructions:

> <hands-on-title>Execute the entire workflow in one go.</hands-on-title>
>
> 1. Import the workflow into Galaxy
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/synthetic-biology/tutorials/basic_assembly_analysis/workflows/Genetic_Design_BASIC_Assembly.ga" title="Genetic Design - Basic Assembly Workflow" %}
>
> 2. Click on *Workflow* on the top menu bar of Galaxy. You will see **Genetic Design - Basic Assembly Workflow**
> 3. Click on the {% icon workflow-run %} (*Run workflow*) button next to your workflow
> 4. Provide the workflow with the following parameters:
>    - {% icon param-file %} *"Pathway (SBML)":* Select `pathway.xml` from your current history
>    - *"Host taxon ID"*: Leave the default value `83333`. This stands for using E. coli as the chassis host.
>    - *"Comma separated taxon IDs of output enzyme sequences"*: enter `553` which is the taxon ID of the Pantoea ananatis strain from which we want to extact enzymes
>    - {% icon param-file %} *"Linkers and user parts": Select `parts_for_lycopene.csv` from your current history in csv format*
>    - {% icon param-file %} *"DNA-Bot settings (Optionally)"*: Select either `dnabot_paris_settings.yaml` or `dnabot_london_settings.yaml` from your history)
>
>    > <comment-title></comment-title>
>    >
>    > All the outputs will be automatically generated and identical to the previous ones.
>    {: .comment}
{: .hands_on}

tools_json:
["Selenzyme", "toolshed.g2.bx.psu.edu/repos/tduigou/selenzy/selenzy-wrapper/0.2.0", "BasicDesign", "toolshed.g2.bx.psu.edu/repos/tduigou/rpbasicdesign/rpbasicdesign/0.3.4", "DNA-Bot", "toolshed.g2.bx.psu.edu/repos/tduigou/dnabot/dnabot/3.1.0"]


topic: synthetic-biology
tutorial: pathway_analysis
title: Evaluating and ranking a set of pathways based on multiple metrics
difficulty: intermediate
time_estimation: 20M
requirements_json: []
tags_json: []
content_hash: 72f1eb4e6bd7a4cee768c861c2947ad1
last_modified: 2026-05-15T22:28:17.274472
zenodo_link: https://zenodo.org/record/6628296
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/synthetic-biology/tutorials/pathway_analysis/tutorial.html

content:
Progress in synthetic biology is enabled by powerful bioinformatics tools such as those aimed to design metabolic pathways for the production of chemicals. These tools are available in SynBioCAD portal which is the first Galaxy set of tools for synthetic biology and metabolic engineering ({% cite Hrisson2022 %}).

In this tutorial, we will use a set of tools from the **Pathway Analysis workflow** which will enable you to evaluate a set of heterelogous pathways previously produced by the RetroSynthesis workflow in a chassis organism (_E. coli_ iML1515). These workflows are available in the [Galaxy SynbioCAD platform](https://galaxy-synbiocad.org). The goal is to inform the user of the theoretically best performing pathways by ranking them based on the four following criteria: target product flux, thermodynamic feasibility, pathway length and enzyme availability.

We recommend that you follow the Retrosynthesis tutorial before starting the current tutorial which will enable you to find pathways to synthesize heterologous compounds producing Lycopene in the _E. coli_ chassis organism.

Four main steps will be run using the following workflow:

To rank the computed heterologous pathways, we need to calculate some metrics. This is why an in-house Flux Balance Analysis (FBA) was developed to calculate the production flux of a given target (e.g. lycopene). The method forces a fraction of its maximal flux through the biomass reaction while optimizing for the target molecule. This is achieved by the _Flux Balance Analysis_ tool.

Secondly, we will use the _Thermo_ tool to estimate thermodynamics values (based on Gibbs free energies) for each pathway to know whether a producing pathway is feasible in physiological conditions. The contribution of individual reactions to the final pathway thermodynamic is balanced solving a linear equation system.

After that, the _Score Pathway_ tool is used to calculate a global score combining target flux, pathway thermodynamics, pathway length and enzyme availability.

Finally, the pathway are ranked based on the global score using the _Rank Pathways_ tool.

![This image shows the pathway analysis workflow viewed from the Galaxy editor interface. The pathway analysis workflow will evaluate and rank pathways based on multiple metrics. First, Flux Balance Analysis (FBA) node will take as inputs a collection of SBMLs representing the heterologous pathways to be evaluated, another SBML representing the chassis, and 2 additional inputs are expected for indicating the biomass reaction ID and the cell compartment ID that will be used during the FBA analysis. The FBA method will optimize the production of the target under the constraint that the model should still produce some level biomass. Then, the Thermo tool is used to estimate thermodynamics values for each pathway to know whether it is favorable towards target production in physiological conditions. After that the Score Pathway tool will calculate a global score combining target flux, pathway thermodynamics, pathway length and enzyme availability. Finally, the pathways are ranked based on the global score using the Rank Pathways tool. All exchange files are SBMLs. The output is a CSV file which contains the pathway IDs and their corresponding global score.](../../images/pathway_analysis_workflow.jpg)

Note that we will run the steps of this workflow individually so as not to neglect the understanding of the intermediate steps. Then, we will run the workflow automatically so that it itself retrieves the outputs from the previous step and gives them as input to the next tool.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data Preparation

First we need to upload and prepare the following inputs to analyze:

- A set of pathways provided in the SBML format (Systems Biology Markup Language) to be ranked, modeling heterologous pathways such as those outputted by the **RetroSynthesis workflow** (available in [Galaxy SynbioCAD platform](https://galaxy-synbiocad.org)).

- The GEM (Genome-scale Metabolic Models) which is a formalized representation of the metabolism of the host organism (the model is _E. coli_ iML1515), provided in the SBML format.

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial named *Pathway Analysis*.
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) :
>
>    ```
>    https://zenodo.org/record/6628296/files/rp_001_0001.xml
>    https://zenodo.org/record/6628296/files/rp_001_0006.xml
>    https://zenodo.org/record/6628296/files/rp_001_0011.xml
>    https://zenodo.org/record/6628296/files/rp_002_0001.xml
>    https://zenodo.org/record/6628296/files/rp_002_0006.xml
>    https://zenodo.org/record/6628296/files/rp_002_0011.xml
>    https://zenodo.org/record/6628296/files/rp_003_0001.xml
>    https://zenodo.org/record/6628296/files/rp_003_0116.xml
>    https://zenodo.org/record/6628296/files/rp_003_0231.xml
>    https://zenodo.org/record/6628296/files/SBML_Model_iML1515.xml
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Create a list or collection named `Heterologous pathways` and composed of the 9 rpSBML pathways.
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
{: .hands_on}

# Compute the target product flux

Notice that the starting compounds (in other words, _the precursors_) of the predicted pathways (also referred as the _heterologous pathways_) are compounds that have been initially extracted from the genome-scale metabolic model (GEM) of the organism we are interested in (also referred as _chassis_). While this step is out of the scope of the present Pathway Analysis tutorial, this means that the precursors of predicted pathways are also present in the chassis model. Hence, predicted pathways and the chassis organism model can be merged to construct "augmented" whole-cell models, enabling flux analysis of these metabolic systems. This is what we'll do here to predict the production flux of a compound of interest.

Within the frame of this tutorial, we'll use the _E. coli_ iML1515 GEM (downloaded from the [BiGG database](http://bigg.ucsd.edu/)) to model the chassis metabolism of _E. coli_ and the target compound is the lycopene. The provided _E. coli_ model is in the SBML. The extraction of precursor compounds and the pathway prediction have already been performed during the RetroSynthesis workflow (available in [Galaxy SynbioCAD platform](https://galaxy-synbiocad.org)).

The FBA (Flux Balance Analysis) method used to calculate the flux is a mathematical approach (as decribed in section Methods in {% cite Hrisson2022 %}) which uses the COBRApy package ({% cite Ebrahim2013 %}) and proposes 3 different analysis methods (standard FBA, parsimonious FBA, fraction of reaction). The first two methods are specific to the COBRApy package and the last one `Fraction of Reaction` is an in-house analysis method (as decribed in section Methods in {% cite Hrisson2022 %}) to consider the cell needs for its own maintenance while producing the target compound.

Within the workflow, the purpose of the _Flux Balance Analysis_ tool is to predict the production flux of the targeted compound, while considering the cellular needs. Under such simulation conditions, the analysis that returns a low production flux may be due to some precursor compounds having a limiting production flux, nor cofactor fluxes involved not being sufficiently balanced by the chassis native metabolism. Pathways with high flux would be caused by both the precursor compounds and the cofactors being in abundance. In either case, bottlenecks that limit the flux of the pathway may be investigated (this is outside of the scope of the workflow) and pathways that do not theoretically generate high yields can be filtered out.

We first perform an FBA (with COBRApy) optimizing the biomass reaction and record its maximal theoretical flux. The upper and lower bounds of the biomass reaction are then set to a same amount, equals to a fraction of its previously recorded optimum (default is 75% of its optimum). The method then performs a second FBA where biomass flux is enforced to this fraction of its optimum while optimizing the target production flux. Simulated fluxes are recorded directly into the SBML file and all changed flux bounds are reset to their original values before saving the output file.

![This picture describes the process to obtain annotated SBML pathways with calculated fluxes. First, FBA tool takes as input one SBML representing the heterologous pathway and another SBML representing the chassis. The two SBMLs are merged to render an augmented model containing both the reactions of the heterologous pathway and the chassis. Then FBA tool uses the COBRApy package to optimize the producing flux of the target reaction, under the constraint that the flux of the biomass reaction should be equals to 75% of its maximal theoretical value. At the end, the calculated fluxes are recorded as annotations into the SBML of the heterologous pathway.](../../images/fba_calculation.png)

> <details-title>Comment</details-title>
>
> Blocking compounds that cannot provide any flux are temporarily removed from heterologous reactions for the FBA evaluation. Such cases can happen due to side substrates or products of predicted reactions that do not match any chassis compound, representing dead-end paths.
>
{: .details}

> <hands-on-title>Calculating the flux of a target using Flux Balance Analysis (FBA)</hands-on-title>
>
> 1. Run {% tool [Flux balance analysis](toolshed.g2.bx.psu.edu/repos/iuc/rpfba/rpfba/5.12.1) %} with the following parameters:
>    - {% icon param-collection %} *"Pathway (rpSBML)"*: Select `Heterologous pathways` (Input dataset collection) from your current history.
>    - {% icon param-file %} *"Model (SBML)"*: Select `SBML_Model_iML1515.xml` (Input dataset) from your current history.
>    - *"SBML compartment ID"*: Leave the default value `c`.
>
>    > <comment-title>Choose a compartment corresponding to your model</comment-title>
>    >
>    > You can specify the compartment from which the chemical species were extracted.
>    > The default is `c`, the BiGG code for the cytoplasm.
>    {: .comment}
>
>    - *"biomass reaction ID"*: Specify the biomass reaction ID that will be used for the "fraction" simulation, type `R_BIOMASS_Ec_iML1515_core_75p37M`.
>
>    > <comment-title>How to select the biomass reaction ID ?</comment-title>
>    >
>    > The biomass reaction ID objective is extracted from the current model *E.Coli iML1515*. You can search the term `biomass` in your XML model and pick the ID where the term `core` appears.
>    {: .comment}
>
>    - *"Constraint based simulation type"*: `Fraction of Reaction`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What is the FBA score for `rp_003_0001` pathway ?
>
> > <solution-title></solution-title>
> >
> > 1. View the SBML rp_003_0001 file and look for `fba_fraction` value in `<groups:listOfGroups>` section: value= `0.23693089430893874`.
> >
> {: .solution}
>
{: .question}

## Compute thermodynamics values

The goal of the thermodynamic analysis is to estimate the feasibility of the predicted pathways toward target production, in physiological conditions. The eQuilibrator libraries ({% cite Flamholz_2011 %}) are used to calculate the formation energy of compounds by either using public database IDs (when referenced within the tools internal database) or by decomposing the chemical structure and calculating its energy of formation using the component contribution method.

The reaction Gibbs energy is estimated by combining the energy of formation of the compounds involved in the reaction (with consideration for the stoichiometric coefficients).

Finally, the thermodynamic of a pathway is estimated by combining the Gibbs energy of reactions involved in it. The contribution of individual reactions to the final pathway thermodynamic is balanced using a linear equation system, according to the relative uses of intermediate compounds across the pathway (See Thermodynamics in Methods section for further details: {% cite Hrisson2022 %}). A pathway Gibbs energy below zero indicates that the thermodynamic is favorable toward the production of the target.

![This image describe the process of computing thermodynamics values of a pathway, to assess whether a pathway is favorable towards target production in physiological conditions. rpThermo tool takes as input the output of rpFBA, which corresponds to the pathway to be analyzed (as an SBML file). rpThermo relies on eQuilibrator libraries to calculate the formation energy of compounds. The reaction Gibbs energy is estimated by combining the energy of formation of the compounds involved in the reaction (with consideration for the stoichiometric coefficients). From this, the thermodynamic of a pathway is estimated by combining the Gibbs energy of all reactions taking part in it. The individual contribution of reactions to the final pathway thermodynamic is weighted using a linear equation system, to take into consideration of the relative needs of intermediate compounds across the pathway. For the record, a pathway Gibbs energy below zero indicates that the thermodynamic is favorable toward the production of the target.](../../images/rpthermo.png)

Secondly, we will use the _Thermo_ tool to estimate thermodynamics values (based on Gibbs free energies) for each pathway to know whether a producing pathway is feasible in physiological conditions

> <hands-on-title>Compute thermodynamics values for each pathway using rpThermo tool</hands-on-title>
>
> 1. {% tool [Thermo](toolshed.g2.bx.psu.edu/repos/tduigou/rpthermo/rpthermo/5.12.1) %} with the following parameters:
>    - {% icon param-file %} *"Input File"*: `pathway_with_fba` (output of **Flux balance analysis** {% icon tool %})
>
>    > <comment-title></comment-title>
>    >
>    > The tool takes as input pathways in SBML format and returns annotated pathways (with thermodynamics information for each reaction) in SBML format too.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What is the thermodynamic score attributed to the reaction with the following *EC (Enzyme Commission) number* *2.5.1.29* for `rp_001_0001` pathway ?
>
> > <solution-title></solution-title>
> >
> > 1. View the SBML rp_001_0001 file and search the reaction ID `2.5.1.29` contained in `<listOfReactions>`. The corresponding value is indicated in `thermo_dGm_prime` field : `-242.348`.
> >
> {: .solution}
>
{: .question}

## Compute the global score of pathways

The _Pathway Score_ tool provides a global score for a given pathway previously annotated by the _Flux Balance Analysis_ and _Thermo_ tools. This score is computed by a machine learning (ML) model (cf. Machine Learning Global Scoring in {% cite Hrisson2022 %}). The model takes as input features describing the pathway (thermodynamic feasibility, target flux with fixed biomass, length) and the reactions within the pathway (reaction SMARTS, Gibbs free energy, enzyme availability score) and prints out the probability for the pathway to be a valid pathway. The ML model has been trained on literature data (cf. section Benchmarking with literature data in {% cite Hrisson2022 %}) and by a validation trial (cf. section Benchmarking by expert validation trial in {% cite Hrisson2022 %}).

> <hands-on-title>Compute the global score using the _Pathway Score_ tool</hands-on-title>
>
> 1. {% tool [Score Pathway](toolshed.g2.bx.psu.edu/repos/tduigou/rpscore/rpscore/5.12.1) %} with the following parameters:
>    - {% icon param-file %} *"Pathway (rpSBML)"*: `pathway_with_thermo` (output of **Thermo** {% icon tool %})
>
>    > <comment-title></comment-title>
>    >
>    > This tool will output a new annotated SBML file representing the pathway, containing the `global_score` annotation.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What is the computed global score for the `rp_001_0001` pathway ?
>
> > <solution-title></solution-title>
> >
> > 1. View the SBML file `rp_001_0001` and search for `global_score` : value=`0.975147980451584`.
> >
> {: .solution}
>
{: .question}

## Rank annotated pathways

Finally, _Rank Pathways_ ranks the previous set of heterologous pathways, based on their global score, to reveal what are the most likely pathways to produce the target molecule (here it is _lycopene_) in a given organism of interest (_E. coli_ in this tutorial).

> <hands-on-title>Rank annotated pathways using rpRanker tool</hands-on-title>
>
> 1. {% tool [Rank Pathways](toolshed.g2.bx.psu.edu/repos/tduigou/rpranker/rpranker/5.12.1) %} with the following parameters:
>    - {% icon param-file %} *"Pathways"*: `scored_pathway` (output of **Score Pathway** {% icon tool %})
>
>    > <comment-title></comment-title>
>    >
>    > This tool will output a CSV file which contains the pathway IDs and their corresponding global score.
>    {: .comment}
>
{: .hands_on}


> <question-title></question-title>
>
> 1. What are the 3 top-ranked pathways ?
>
> > <solution-title></solution-title>
> >
> > 1. `002_0011`, `001_0011` ,`002_0006`.
> >
> {: .solution}
>
{: .question}

# Run the **Pathway Analysis Workflow**

In this section, you can run the Pathway Analysis Workflow more easily and fastly following these instructions:

> <hands-on-title>Execute the entire workflow in one go.</hands-on-title>
>
> 1. Import the workflow into Galaxy
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/synthetic-biology/tutorials/pathway_analysis/workflows/main_workflow.ga" title="Pathway Analysis Workflow" %}
>
> 2. Click on *Workflow* on the top menu bar of Galaxy. You will see **Pathway Analysis Workflow**
> 3. Click on the {% icon workflow-run %} (*Run workflow*) button next to your workflow
> 4. Provide the workflow with the following parameters:
>    - {% icon param-file %} *"Heterologous pathways":* Select `Heterologous pathways` (Input dataset collection) from your current history.
>    - {% icon param-file %} *"Chassis where to produce target from"*: Select `SBML_Model_iML1515.xml` (Input dataset) from your current history.
>    - *"Cell compartment ID"*: Enter value `c`.
>    - *"biomass reaction ID"*: Specify the biomass reaction ID that will be restricted in the "fraction" simulation type `R_BIOMASS_Ec_iML1515_core_75p37M`.
>
>    > <comment-title></comment-title>
>    >
>    > All the outputs will be automatically generated and identical to the previous ones.
>    {: .comment}
{: .hands_on}

# Conclusion


To select the best pathways for producing the lycopene in *E. coli*, some metrics have to be estimated, namely production flux of the target and pathway thermodynamics. A global score is then computed by combining these criteria with others (pathway length, enzyme availability score, reaction SMARTS) using a machine learning model. These steps achieved using the tools of the presented Pathway Analysis workflow.

![This scheme represents the pathway analysis workflow enabling the identification of the best pathways for producing a molecule of interest. To do that the workflow takes as input the collection of pathways to be scored and the metabolic model of the chassis (all files are SBMLs). Iteratively, each pathway is merged with the chassis model, and several metrics are evaluated such as the target production flux (using the rpFBA tool) and the thermodynamics of the pathway (using the rpThermo tool). A global score of each pathway combining these metrics with others is computed using a machine learning method (with the rpScore tool). Finally, the pathways are ranked from best to worst according to their global score (using the rpRanker tool). During the workflow, all metrics are stored as SBML annotations. The final output is a CSV file which contains the pathway IDs and their corresponding global score.](../../images/pathway_analysis_scheme.png)

tools_json:
["Flux balance analysis", "toolshed.g2.bx.psu.edu/repos/iuc/rpfba/rpfba/5.12.1", "Thermo", "toolshed.g2.bx.psu.edu/repos/tduigou/rpthermo/rpthermo/5.12.1", "Score Pathway", "toolshed.g2.bx.psu.edu/repos/tduigou/rpscore/rpscore/5.12.1", "Rank Pathways", "toolshed.g2.bx.psu.edu/repos/tduigou/rpranker/rpranker/5.12.1"]


topic: synthetic-biology
tutorial: retrosynthesis_analysis
difficulty: intermediate
time_estimation: 30M
requirements_json: []
tags_json: []
content_hash: 33ba8d35278cb1c66221875913dc3548
last_modified: 2026-05-15T22:28:17.277991
workflows_json: []
title:
Generating theoretical possible pathways for the production of Lycopene in E.Coli using Retrosynthesis tools

url:
https://training.galaxyproject.org/training-material/topics/synthetic-biology/tutorials/retrosynthesis_analysis/tutorial.html

content:
[Galaxy-SynBioCAD portal](https://galaxy-synbiocad.org/) is the first toolshed for synthetic biology, metabolic engineering, and industrial biotechnology ({% cite Hrisson2022 %}). It provides a set of Retrosynthesis tools aimed at finding pathways to synthesize heterologous compounds in chassis organisms (**RetroRules** ({% cite Duigou2018 %}), **RetroPath2.0** ({% cite Delpine2018 %}), **RP2Paths**, **rpCompletion**).

Retrosynthesis is a concept originally proposed for synthetic chemistry where chemists have to work backwards, starting from a target product to reach precursors that are endogenous to the chassis (host organism).

Typically, the target compound, also named “source compound” is the compound of interest one wishes to produce, while the precursors are usually compounds that are natively present in a chassis strain.

<center><img src="../../images/chassis_target.png" width="500" alt="Chassis - target"/></center>

In this tutorial, we want to obtain the reactions producing the lycopene (source) into the iML1515 _Escherichia Coli_ strain (chassis).

To do that, we will use the following RetroSynthetis Workflow composed of 3 key steps.

First, we aggregate the metabolites present in the chassis and download reaction rules.

Then, RetroPath2.0 generates feasible metabolic routes between a collection of chemical species contained within a GEM SBML (Systems Biology Markup Language) file of the selected organism, a target molecule that the user wishes to produce, and reaction rules extracted from RetroRules.

Lastly, the metabolic network is then deconstructed into individual pathways using RP2paths and rpCompletion takes those individual metabolic pathways to filter them (duplicated pathways are removed), splits them into sub-pathways by adding the appropriate cofactors, and finally converted them to SBML files.

![Retrosynthesis Workflow](../../images/Retrosynthesis_Workflow.png)

Note that we will run the steps of this workflow individually so as not to neglect the understanding of the intermediate steps as well. Then, we will run the workflow automatically so that it itself retrieves the outputs from the previous step and gives them as input to the next tool.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data Preparation

RetroSynthesis workflow will be run with the following inputs:

1. The International Chemical Identifier (InChI) of the compound of interest to produce,
2. The structure of metabolites present in the chosen chassis (E. coli),
3. Reaction rules (generated by RRules Parser node that calls RetroRules).

The data used are pretty straight forward to obtain.
Firstly, we download an SBML model, then we select all sinks to use into the RetroPath2.0 software from this model.
Lastly, we request from RetroRules all possible reactions to find a chemical reaction cascade that produces the target.

## Download a model

> <hands-on-title>Select a model.</hands-on-title>
>
> 1. Run {% tool [Pick SBML Model](toolshed.g2.bx.psu.edu/repos/tduigou/get_sbml_model/get_sbml_model/0.0.1) %} with the following parameters:
>    - {% icon galaxy-dropdown %} *"Strain"*: `Escherichia coli str. K-12 substr. MG1655 (iML1515)`
>
>    > <comment-title>What does this tool do?</comment-title>
>    >
>    > The selected SBML model is downloaded from the [BiGG database](http://bigg.ucsd.edu/).
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What is the file format of the model?
>
> > <solution-title></solution-title>
> >
> > 1. The SBML is based on XML.
> >
> {: .solution}
>
{: .question}

## Create a sink file from the SBML model

> <hands-on-title>Generate a sink file.</hands-on-title>
>
> 1. Run {% tool [Sink from SBML](toolshed.g2.bx.psu.edu/repos/tduigou/rpextractsink/rpextractsink/5.12.1) %} with the following parameters:
>    - {% icon param-file %} *"Strain"*: `sbml_model` (output of **Pick SBML Model** {% icon tool %})
>    - *"SBML compartment ID"*: `c`
>
>    > <comment-title>Choose a compartment corresponding to your model</comment-title>
>    >
>    > You can specify the compartment from which the tool will extract the chemical species.
>    > The default is `c`, the BiGG code for the cytoplasm.
>    > If the user wishes to upload an SBML file from another source, then this value must be changed.
>    {: .comment}
{: .hands_on}

> <question-title></question-title>
>
> 1. What this tool does?
> 2. How many columns are in the file?
>
> > <solution-title></solution-title>
> >
> > 1. This tool creates a friendly CSV file format that can be used as sink input for RetroPath2.0.
> > 2. Click on {% icon galaxy-eye %}, you should see 2 columns: "Name" and "InChi"
> >
> {: .solution}
>
{: .question}

## Retrieve the reaction rules

> <hands-on-title>Generate a file with all reactions.</hands-on-title>
>
> 1. Run {% tool [RRules Parser ](toolshed.g2.bx.psu.edu/repos/tduigou/rrparser/rrparser/2.4.6) %} with the following parameters:
>    - *"Rule Type"*: `retro`
>    - *"Select the diameters of the reactions rules"*: `2`, `4`, `6`, `8`, `10`, `12`, `14` and `16`
>    - {% icon galaxy-toggle %} *"Compress output"*: `no`
>
>    > <comment-title>How to choose a right diameter ?</comment-title>
>    >
>    > The diameter of the sphere including the atoms around the reacting center.
>    > The higher is the diameter, the more specific are the rules.
>    {: .comment}
{: .hands_on}

> <question-title></question-title>
>
> 1. Does a low diameter select specific rules?
> 2. How many rows are in the file?
>
> > <solution-title></solution-title>
> >
> > 1. No, a low diameter selects more unspecific rules.
> > 2. More than 200 thousands!
> >
> {: .solution}
>
{: .question}

# Run the Retrosynthesis algorithm

RetroPath2.0 is an open-source tool for building retrosynthesis networks by combining reaction rules and a retrosynthesis-based algorithm to link the desired target compound to a set of available precursors.
The RetroPath2.0 tool is freely available at [myExperiment](https://www.myexperiment.org/workflows/4987.html).
The retrosynthesis network is outputted as a CSV file providing reactions in the reaction SMILES format and chemicals in both SMILES and InChI formats along with other information like the score for each reaction.

## Launch **RetroPath2.0**

> <hands-on-title>Build a reaction network</hands-on-title>
>
> 1. Run {% tool [RetroPath2.0](toolshed.g2.bx.psu.edu/repos/tduigou/retropath2/retropath2/2.3.0) %} with the following parameters:
>    - {% icon param-file %} *"Rules File"*: `out_rules` (output of **RRules Parser** {% icon tool %})
>    - {% icon param-file %} *"Sink File"*: `sink` (output of **Sink from SBML** {% icon tool %})
>    - *"InChI type"*: `By string`
>        - *"Source InChI"*: InChI=1S/C40H56/c1-33(2)19-13-23-37(7)27-17-31-39(9)29-15-25-35(5)21-11-12-22-36(6)26-16-30-40(10)32-18-28-38(8)24-14-20-34(3)4/h11-12,15-22,25-32H,13-14,23-24H2,1-10H3/b12-11+,25-15+,26-16+,31-17+,32-18+,35-21+,36-22+,37-27+,38-28+,39-29+,40-30+
>
>    > <tip-title>InChi format</tip-title>
>    >
>    > Be careful, you must have `InChi=` in front of your InChi key.
>    {: .tip}
{: .hands_on}

> <question-title></question-title>
>
> 1. What is the file format produce by RetroPath2.0?
>
> > <solution-title></solution-title>
> >
> > 1. A csv file.
> >
> {: .solution}
>
{: .question}

# Enumerate pathways with **RP2paths**

The RetroPath2.0 algorithm produces a reaction network, we want to have one pathway per file in SBML format.
We need to split the network and perform some adjustments to these pathways.

## Split the network
> <hands-on-title>Build a reaction network</hands-on-title>
>
> 1. Run {% tool [RP2paths](toolshed.g2.bx.psu.edu/repos/tduigou/rp2paths/rp2paths/1.5.0) %} with the following parameters:
>    - {% icon param-file %} *"RetroPath2.0 Pathways"*: `Reaction_Network` (output of **RetroPath2.0** {% icon tool %})
>
>    > <comment-title>Principle</comment-title>
>    >
>    > Extracts the set of pathways that lies in a metabolic space file output by the RetroPath2.0 workflow
>    {: .comment}
{: .hands_on}

> <question-title></question-title>
>
> 1. Why producing multiple pathways?
> 2. How many outputs are produced?
>
> > <solution-title></solution-title>
> >
> > 1. One network could represent several pathways, so several solutions.
> > 2. Two outputs are produced: one corresponding to the metabolites, the other one corresponding to the pathways.
> >
> {: .solution}
>
{: .question}

## Refine reactions

> <hands-on-title>Refine reactions</hands-on-title>
>
> 1. Run {% tool [Complete Reactions](toolshed.g2.bx.psu.edu/repos/tduigou/rpcompletion/rpcompletion/5.12.2) %} with the following parameters:
>    - {% icon param-file %} *"RP2paths pathways"*: `master_pathways` (output of **RP2paths** {% icon tool %})
>    - {% icon param-file %} *"RP2paths compounds"*: `compounds` (output of **RP2paths** {% icon tool %})
>    - {% icon param-file %} *"RetroPath2.0 metabolic network"*: `Reaction_Network` (output of **RetroPath2.0** {% icon tool %})
>    - {% icon param-file %} *"Sink from SBML"*: `sink` (output of **Sink from SBML** {% icon tool %})
>
>    > <comment-title>Principle</comment-title>
>    >
>    > Each reaction rule can correspond to several template reactions, the task here is to enumerate the different possible transformations according to these templates.
>    > Because the RetroRules reaction rules consider only one substrate at a time, some compounds are by purpose omitted: the task here is to complete the predicted reactions by putting back these omitted compounds (mostly  cofactors).
>    > The node converts each predicted pathway to distinct SBML files.
>    {: .comment}
{: .hands_on}

> <question-title></question-title>
>
> 1. How many solutions are found?
> 2. In which file format are the pathways?
> 2. Do these pathways are a good solution?
>
> > <solution-title></solution-title>
> >
> > 1. We have 9 candidates.
> > 2. The pathways are represented in a SBML format.
> > 3. It's a trap! We don't know if they represent a good solution. We need to evaluate them.
> >
> {: .solution}
{: .question}

# Run the **RetroSynthesis Workflow**

In this section, you can run the RetroSynthesis Workflow more easily and fastly following these instructions:

> <hands-on-title>Execute the entire workflow in one go.</hands-on-title>
>
> 1. Import the workflow into Galaxy
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/synthetic-biology/tutorials/basic_assembly_analysis/workflows/Genetic_Design_BASIC_Assembly.ga" title="RetroSynthesis workflow" %}
>
> 2. Click on *Workflow* on the top menu bar of Galaxy. You will see **RetroSynthesis** workflow.
> 3. Click on the {% icon workflow-run %} (*Run workflow*) button next to your workflow
> 4. Provide the workflow with the following parameters:
>    - *"Target to produce":* Provide the following Inchi source:  `InChI=1S/C40H56/c1-33(2)19-13-23-37(7)27-17-31-39(9)29-15-25-35(5)21-11-12-22-36(6)26-16-30-40(10)32-18-28-38(8)24-14-20-34(3)4/h11-12,15-22,25-32H,13-14,23-24H2,1-10H3/b12-11+,25-15+,26-16+,31-17+,32-18+,35-21+,36-22+,37-27+,38-28+,39-29+,40-30+`
>    - *"Strain":* Select `Escherichia coli str. K-12 substr. MG1655 (iML1515)` SBML model.
>
>    > <comment-title></comment-title>
>    >
>    > All the outputs will be automatically generated and identical to the previous ones.
>    {: .comment}
{: .hands_on}

# Conclusion


In this tutorial we produced candidates pathways to produce Lycopene in an Esherichia Coli strain.
Three main steps were involved:
1. Preprocessing data
2. Run Retrosynthesis algorithm with RetroPath2.0
3. Enumerate all solutions found by RetroPath2.0

tools_json:
["Pick SBML Model", "toolshed.g2.bx.psu.edu/repos/tduigou/get_sbml_model/get_sbml_model/0.0.1", "Sink from SBML", "toolshed.g2.bx.psu.edu/repos/tduigou/rpextractsink/rpextractsink/5.12.1", "RRules Parser", "toolshed.g2.bx.psu.edu/repos/tduigou/rrparser/rrparser/2.4.6", "RetroPath2.0", "toolshed.g2.bx.psu.edu/repos/tduigou/retropath2/retropath2/2.3.0", "RP2paths", "toolshed.g2.bx.psu.edu/repos/tduigou/rp2paths/rp2paths/1.5.0", "Complete Reactions", "toolshed.g2.bx.psu.edu/repos/tduigou/rpcompletion/rpcompletion/5.12.2"]


topic: materials-science
tutorial: muon-stopping-sites-muairss-uep
title: Finding the muon stopping site with pymuon-suite in Galaxy
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: []
content_hash: cd3ae6bdb0e522d1e3485604cb988281
last_modified: 2026-05-15T22:28:13.738701
zenodo_link: https://zenodo.org/records/10219558
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/materials-science/tutorials/muon-stopping-sites-muairss-uep/tutorial.html

content:
The problem of identifying the stopping site of the muon is often key in the
interpretation of muon experimental results. As it was mentioned in the lecture,
the problem of finding the muon stopping site has been previously
tackled using computational tools, and one of the most common methods is the so-called
{UEP} method, which identifies proposes that
the muon stops at the minima of the {UEP} of a crystalline host material.

pymuon-suite in Galaxy implements a variant of the {UEP}, that allows the
users to efficiently scan the {UEP} of the host material and use a clustering procedure
to classify those minima as potential stopping sites. But, before moving into the details
of how the method works, it is useful to revisit some of the key assumptions
behind what happens in a muon experiment.

At the atomic level, muon spectroscopy experiments go something like this:

- A spin-polarized muon beam of positive muons is fired at the sample
- Through inelastic collisions with the atoms of the crystalline host material,
 muons shed their kinetic energy, which turns into heat or causes small amounts
 of damage to the crystal. This continues until the muons' kinetic energy becomes low enough
that it is comparable to their potential energy in the electrostatic potential
generated by electrons and nuclei alike.
- At that point, Coulomb forces start driving the muon, until it eventually
  loses as much energy as possible and comes to a stop in a position that
  sometimes represents a local (and possibly global) minimum for the sample's
  {UEP} energy. Once it stopped, the muon may spend
  the rest of its time in that position until it decays.

![Graphical illustration of muons being implanted in a sample, then decaying to a positron and being detected](../../images/muon_spectroscopy_diagram.png)

Where these minima of the potential actually are isn't always obvious and,
although the our version of the {UEP} method has proven to be quite efficient and useful,
 there are certain crystalline materials where predicting the minima of the {UEP}
 potential can be a lot harder. For instance, in lithium fluoride {% cite Liborio2018 %},
 the potential's minima are flat and
wide and current computer simulations fail to predict the muon stopping site.
Hence, users must exercise scientific criteria when analyzing results form the method

This tutorial goes into the details of how to estimate the muon stopping
site in crystalline copper, as described in the lecture, and it assumes
 that some basic Galaxy concepts such as: how to upload a file;
how to use a Galaxy tool; how to create, view and share a history and how to extract
 and run a Galaxy workflow, are known. Hence,
as stated in the requirements at the beginning of this tutorial please, before starting
 have a look at this: [A short introduction to Galaxy]({% link topics/introduction/tutorials/galaxy-intro-short/tutorial.md %})

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Workflow for Finding the µ+ stopping site
As described in the lecture, the full workflow associated with finding
potential muon stopping sites in crystalline copper is shown in the figure below:

![Graphical illustration of the steps in finding the muon stopping site for a Silicon crystal lattice. (1) Start with a pure host, (2) randomly populate with muons, (3) relax electrostatic forces, (4) cluster structures to give the stopping site in either (5) a conventional unit cell or (6) a supercell.](../../images/Workflow_for_finding_mu+.png)

It comprises 6 steps that will be described in detail in the next sections.  Steps 1 and 6
are performed outside of the Galaxy platform, usually in an external computer cluster that can
run the necessary {DFT} simulations. Steps 2, 3, 4, and 5 of this workflow are are carried out
in Galaxy using two Galaxy tools: `PyMuonSuite AIRSS UEP Optimise` and `PyMuonSuite AIRSS Cluster`.

![Screenshots of the Galaxy UI showing the two tools, PyMuonSuite AIRSS UEP Optimise and PyMuonSuite AIRSS Cluster, and which steps of the described process they correspond to (1,2,3 and 4,5,6 respectively).](../../images/2GalaxyTools.png)

Hence, there is a difference between the full workflow and the specific Galaxy part of this workflow, which can be
saved, shared and edited in the Galaxy platform. For the case of finding potential muon stopping sites in
crystalline Cu, the components of Galaxy workflow are shown below:

![Screenshot of the Galaxy workflow editor showing the three input files (Copper-out.cell, Copper.den_fmt, Copper.castep) needed for the tool PyMuonSuite AIRSS UEP Optimise, and how that output connects to the PyMuonSuite AIRSS Cluster tool.](../../images/GalaxyWorkflow.png)

And this is the workflow that can be imported for this tutorial.

## Import workflow

> <hands-on-title>Import workflow</hands-on-title>
>
> 1. Import the workflow for this tutorial using [these instructions](workflows/).
> 2. Open the editor for the workflow, and verify that all the required tools
>    are installed on your Galaxy instance, and have the same version(s) as the
>    tools in workflow. If this is not the case a dialogue box will warn you.
>
{: .hands_on}

## {DFT} calculation of the Pure Host Material

 This method requires only a single {DFT} calculation for the host material.
 This calculation could be a geometry optimisation, which uses
 a self-consistent methodology to estimate the electronic
 structure of the material and relax the structure. Pymuon_suite uses the
 CASTEP code for running its {DFT} calculations, and some basic information
  about how to run a geometry optimisation is available in the [CASTEP documentation](https://castep-docs.github.io/castep-docs/).

Once the relaxed structure and its corresponding electronic structure are obtained,
a set of CASTEP output files is generated. We need to load just three of these files
into Galaxy.  These are the relaxed CASTEP structural file `*-out.cell`, the
charge density file `*.den_fmt` and the CASTEP output file `*.castep`.

The CASTEP files for this example have all been pre-generated for this tutorial,
but in general you would need to create these yourself by running CASTEP simulations
outside of Galaxy.

To run the first part of the workflow, on the left menu of the Galaxy web-page
we look for the `FIND MUON STOPPING SITES` header and first click on `UEP method` and then
select the `PyMuonSuite AIRSS UEP Optimise` Galaxy tool. This brings up the central panel of the Galaxy
web-page, where can load the CASTEP files; complete the *Generation* parameters to set up the
creation of randomly populated muonated structures; and complete the *Optimisation* parameters to set up the relaxation the
electrostatic forces of the muon in each of the randomly populated muonated structures.

### How to load CASTEP files into Galaxy

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}), from you computer, or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/10219558/files/Cu-out.cell
>    https://zenodo.org/records/10219558/files/Cu.castep
>    https://zenodo.org/records/10219558/files/Cu.den_fmt
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets to match their file names, if needed.
> 4. Check that the datatype is correct for each dataset.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

## Create Randomly Populated Muonated Structures

The first step within the Galaxy platform is the population of the
host material's interstitial spaces with muons that are
randomly distributed in accordance with certain physical constraints.

![Graphical illustration of how the parameters Gaussian Width affects the potential of atoms, Van der Waals Scale increases the distance between atoms and muons, and Poisson Radius increases the starting distance between muons.](../../images/Create-random-structures.png)

Essentially, the muons are randomly distributed while respecting
a minimum distance between them and between the atoms
in the host material. The `Poisson radius` parameter sets the minimum spacing
between the muons that are randomly implanted in the sample, while the
`Van der Waals scale` parameter sets the minimum spacing between randomly
implanted muons and the atoms in the host material.  Two of the images in the figure
above show how the density of distributed muons changes for different values of
the Poisson radius. These images are just for visualization purposes, what
the method really does is generate a set of muonated structures, where each one has a muon
in a different place, and place each muonated structure in a corresponding folder.

## Relax Electrostatic Forces on Muons

As stated in the lecture, the method minimizes the electrostatic forces acting on the muon in each of the
muonated structures generated in the previous step.  In this relaxation procedure,
the forces are calculated analytically using data obtained from the {DFT} simulation of the unperturbed host material:
the charge density file `*.den_fmt`, the relaxed CASTEP structural file `*-out.cell`, the and the CASTEP output file `*.castep`,
and then the muon is displaced in a direction
that minimizes the forces acting upon it. This is a self-consistent procedure, which is repeated until the
forces on each muon are below the value specified in the `geometry optimisation tolerance` parameter, or until
the maximum number of `geometry optimisation steps` is reached.

![Graphical illustration of how muon positions might move during geometry optimisation to concentrate in one or more areas in the structure, and the equation defining the electrostatics of this process.](../../images/relax_electrostatic_forces_on_muons.png)

This workflow's step could be computationally expensive. The larger the
the size of the host material's unit cell, and the smaller the values of `Poisson radius` and
`Van der Waals scale` parameters, the larger number of muonated structures created, and the
more expensive this step will be.  In any case: the relaxation procedure is run in the Galaxy
server, so the user can just start it and leave it running in the Galaxy instance. The status of
the relaxation will be indicated in the history panel on the right menu of the Galaxy web-page, where
all the processes run by the Galaxy instance are numbered.

Below, a hands-on example on how to run these workflow steps.

> <hands-on-title>PyMuonSuite AIRSS UEP Optimise</hands-on-title>
>
> 1. Run the {% tool [PyMuonSuite AIRSS UEP Optimise](toolshed.g2.bx.psu.edu/repos/muon-spectroscopy-computational-project/pm_uep_opt/pm_uep_opt/0.3.0+galaxy1) %} tool with the following parameters:
>    - *"Tool Parameters"*:
>      - {% icon param-file %} *"Structure file"*: `Cu-out.cell` relaxed CASTEP structural file
>      - {% icon param-file %} *"Charge density file"*: `Cu.den_fmt` CASTEP file with charge density of host material
>      - {% icon param-file %} *"CASTEP log file"*: `Cu.castep` The CASTEP output file for the initial {DFT} simulation
>    - *"Generation Parameters"*:
>      - *"Poisson radius"*: `0.6` Sets the minimum spacing between the muons that are randomly implanted in the sample
>      - *"Van der Waals scale"*: `0.5` Sets the minimum spacing between randomly implanted muons atoms in host material
>      - *"Random seed"*: `optional` this is if we want to re-use a set of generated structures.  Rarely used
>      - *"Format for file containing all muon positions (optional)"*: `cell` Generates an extra output file containing all the muon positions
>    - *"Optimisation parameters Parameters"*:
>      - *"Geometry optimization steps"*: `300` Maximum number of geometry optimisation steps
>      - *"Geometry optimization tolerance"*: `0.05` Tolerance on geometry optimisation in eV/AA
>      - *"Gaussian width factor"*: `5.0` Adjusts how atom potentials are modelled
{: .hands_on}

## Cluster Relaxed Structures - µ+ Stopping Sites

Once the relaxation is over, there will appear a new item on the history panel denominated
*PyMuonSuite AIRSS UEP Optimise on data X, data Y and data Z*. This item
contains a `*.zip` file with all the optimised muonated structures, where *X*, *Y* and *Z* are
the numbers of the processes, in the history panel, that were used for running the relaxation
of the electrostatic forces on muons.

![Graphical illustration of muons clustered in two possible sites for a structure, how this is represented in the tool output report, and the original structure within that cluster which has minimal energy.](../../images/cluster_relaxed_structures_mu+_stopping_sites.png)

The first step for running the clustering procedure is to select the `PyMuonSuite AIRSS Cluster` Galaxy tool on
the left menu of the Galaxy web-page . This brings up the corresponding central panel of the Galaxy
web-page, where can load the optimised muonated structures `*.zip` contained in the
*PyMuonSuite AIRSS UEP Optimise on data X, data Y and data Z* item.

Then we need to select the *clustering method* and it corresponding parameters. There are two clustering methodologies
available: `hierarchical` and `k-means` clustering. We recommend using the hierarchical clustering first, choosing different
values of the `t parameter for hierarchical clustering`, until a consistent number of clusters is found.  Once that
number is found, we can perform a k-means clustering analysis, using the number of hierarchical clustering clusters as
the value for the `Number of clusters for k-means clustering` parameter.

Finally, we need to choose the *Clustering Save Format*. This allows for the visualisation of the potential stopping sites
associated to each of the clusters found, but it can also be used to prepare the `*.cell` supercell file that is needed to
run the last step of the workflow: the {DFT} relaxation of each of the muonated structures associated to each of
the potential stopping sites.  The `*.cell` supercell file can then be downloaded from Galaxy and used to run
a CASTEP {DFT} simulation, which can be used to analyse things such as the distortions caused by the muon in the host
material.

![Graphical illustration of how a muon within a conventional unit cell can be converted to a larger 3 by 3 by 3 supercell, still containing only one muon but with more of the original atoms of the lattice.](../../images/DFT_supercell_muonated_structures.png)

Below, a hands-on example on how to run this workflow step.

> <hands-on-title>PyMuonSuite AIRSS Cluster</hands-on-title>
>
> 1. Run the {% tool [PyMuonSuite AIRSS Cluster](toolshed.g2.bx.psu.edu/repos/muon-spectroscopy-computational-project/pm_muairss_read/pm_muairss_read/0.3.0+galaxy0) %} tool with the following parameters:
>    - *"Tool Parameters"*:
>      - *"optimised muonated structures"*: `PyMuonSuite AIRSS UEP Optimise on data X, data Y and data Z` load the optimised muonated structures `*.zip` file
>      - *"Clustering method"*: `hierarchical` Performs hierarchical clustering
>      - *"t parameter for hierarchical clustering"*: `0.2` Threshold value for clusters
>      - *"Clustering method"*: `k-means` Performs k-means clustering.  Needs guess for number of clusters
>      - *"Number of clusters for k-means clustering "*: `2` Proposed number of clusters for k-means. Usually the clusters found with `hierarchical` clustering
>      - *"Clustering Save Format"*: `cell`  Creates a structural file for each cluster, corresponding to the structure in the cluster with minimal energy, with  the specified format
>      - *"Supercell"*: `3` Supercell size and shape to use when saving cluster structures
>      - *"Muon symbol"*: `H:mu` The symbol to use for the muon when writing out the CASTEP custom species. H:mu is the default.
>
{: .hands_on}


### Analysis of the Cluster report

Once the clustering procedure is over, there will appear a new item on the history panel denominated
*Cluster report for PyMuonSuite AIRSS UEP Optimise on data X, data Y, and data Z*, which contains a
human-readable summary of the clustering results, with the underlying raw data also provided. The top of the
report contains a summary of the settings used to run the clustering, and when it was run.

> <code-out-title>MuAirss Clusters report</code-out-title>
> ```
> ****************************
> |                          |
> |       PYMUON-SUITE       |
> |  MuAirss Clusters report |
> |                          |
> ****************************
>
> Name: struct
> Date: 2023-11-06 09:41:00.787599
> Structure file(s): input_structure.cell
> Parameter file: params.yaml
>
> Clustering method: Hierarchical
>     t = 0.4
>
> *******************
> ```
{: .code-out}

The remainder of the report details the clusters and the individual muonated
structures composing each cluster. In
addition to showing the relative numbers of structures forming each cluster,
we also report the structure with the minimum energy, the average energy for all the structures
and the corresponding standard deviation for that average energy. By analysing of these
quantities for different values of the `Poisson radius`, `Van der Waals scale` and
`t parameter for hierarchical clustering` parameters, we can determine which
are the "stable" clusters, which are the ones associated to potential stopping sites.

> <code-out-title>MuAirss Clusters report</code-out-title>
> ```
> Clusters for struct:
> CALCULATOR: uep
> 	2 clusters found
>
>
> 	-----------
> 	Cluster 1
> 	-----------
> 	Structures: 7
>
> 	Energy (eV):
> 	Minimum		Average		StDev
> 	-10.30		-10.30		0.00
>
> 	Minimum energy structure: struct_7
>
>
> 	Structure list:
> 	struct_1	struct_3	struct_5	struct_6
> 	struct_7	struct_8	struct_12
>
> 	-----------
> 	Cluster 2
> 	-----------
> 	Structures: 7
>
> 	Energy (eV):
> 	Minimum		Average		StDev
> 	-11.93		-11.93		0.00
>
> 	Minimum energy structure: struct_14
>
>
> 	Structure list:
> 	struct_2	struct_4	struct_9	struct_10
> 	struct_11	struct_13	struct_14
>
> 	----------
>
> 	Similarity (ranked):
> 	1 <--> 2 (distance = 1.681)
>
> --------------------------
> ```
{: .code-out}

The classical {UEP} method scans the electrostatic potential looking for minima in it, which
are associated to potential muon stopping sites. This method further classifies those minima
by estimating their "attractor size".  If you look at the picture in the figure below, the two
minima have similar values, but the capacity to "attract" a muon is greater for the minima on the left,
which makes it more likely to be a muon stopping site. The probability of a potential stopping site is
given by the size of the cluster that forms around its associated minima.

![Graphical illustration of how muons may be clustered in two different potential minima that have similar energies with a simple 1D plot, and two real examples (CaF and YF3) showing the potential energy in a 3D structure.](../../images/theoretical_basis_of_our_variant_of_the_UEP_method.png)

tools_json:
["PyMuonSuite AIRSS UEP Optimise", "toolshed.g2.bx.psu.edu/repos/muon-spectroscopy-computational-project/pm_uep_opt/pm_uep_opt/0.3.0+galaxy1", "PyMuonSuite AIRSS Cluster", "toolshed.g2.bx.psu.edu/repos/muon-spectroscopy-computational-project/pm_muairss_read/pm_muairss_read/0.3.0+galaxy0"]


topic: sequence-analysis
tutorial: mapping
title: Mapping
difficulty: introductory
time_estimation: 1h
requirements_json: ["type: internal", "quality-control"]
tags_json: ["deutsch", "espa\u00f1ol", "italiano"]
content_hash: 8496c2b15a04e3d733bc0bcae66bb182
last_modified: 2026-05-27T09:01:49.504188
zenodo_link: https://doi.org/10.5281/zenodo.1324070
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/sequence-analysis/tutorials/mapping/tutorial.html

content:
Sequencing produces a collection of sequences without genomic context. We do not know to which part of the genome the sequences correspond to. Mapping the reads of an experiment to a reference genome is a key step in modern genomic data analysis. With the mapping the reads are assigned to a specific location in the genome and insights like the expression level of genes can be gained.

The reads do not come with position information, so we do not know what part of the genome they came from. We need to use the sequence of the read itself to find the corresponding region in the reference sequence. But the reference sequence can be quite long (~3 billion bases for human), making it a daunting task to find a matching region. Since our reads are short, there may be several, equally likely places in the reference sequence from which they could have been read. This is especially true for repetitive regions.

In principle, we could do a BLAST analysis to figure out where the sequenced pieces fit best in the known genome. We would need to do that for each of the millions of reads in our sequencing data. Aligning millions of short sequences this way may, however, take a couple of weeks. And we do not care about the exact base to base correspondence (alignment). What we are interested in is "where these reads came from". This approach is called **mapping**.

In the following, we will process a dataset with the mapper **Bowtie2** and we will visualize the data with the program **IGV**.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare the data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import `wt_H3K4me3_read1.fastq.gz` and `wt_H3K4me3_read2.fastq.gz` from [Zenodo](https://zenodo.org/record/1324070) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/1324070/files/wt_H3K4me3_read1.fastq.gz
>    https://zenodo.org/record/1324070/files/wt_H3K4me3_read2.fastq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    As default, Galaxy takes the link as name, so rename them.
>
> 3. Rename the files to `reads_1` and `reads_2`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Create a paired collection named `Paired Reads`
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
{: .hands_on}

We just imported in Galaxy FASTQ files corresponding to paired-end data as we could get directly from a sequencing facility.

During sequencing, errors are introduced, such as incorrect nucleotides being called. Sequencing errors might bias the analysis and can lead to a misinterpretation of the data. The first step for any type of sequencing data is always to check their quality.

There is a dedicated tutorial for [quality control]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}) of sequencing data. We will not repeat the steps there. You should follow the [tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}) and apply it to your data before going further.

# Map reads on a reference genome

Read mapping is the process to align the reads on a reference genomes. A mapper takes as input a reference genome and a set of reads. Its aim is to align each read in the set of reads on the reference genome, allowing mismatches, indels and clipping of some short fragments on the two ends of the reads:

![Explanation of mapping](../../images/mapping/mapping.png "Illustration of the mapping process. The input consists of a set of reads and a reference genome. In the middle, it gives the results of mapping: the locations of the reads on the reference genome. The first read is aligned at position 100 and the alignment has two mismatches. The second read is aligned at position 114. It is a local alignment with clippings on the left and right. The third read is aligned at position 123. It consists of a 2-base insertion and a 1-base deletion.")

We need a reference genome to map the reads on.

{% include topics/sequence-analysis/tutorials/mapping/ref_genome_explanation.md answer_3="This data comes from the ChIP-seq of mice, so we will use mm10 (*Mus musculus*)."%}

Currently, there are over 60 different mappers, and their number is growing. In this tutorial, we will use [Bowtie2](http://bowtie-bio.sourceforge.net/bowtie2/), a fast and memory-efficient open-source tool particularly good at aligning sequencing reads of about 50 up to 1,000s of bases to relatively long genomes.

> <hands-on-title>Mapping with Bowtie2</hands-on-title>
> 1. {% tool [Bowtie2](toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.4+galaxy0) %} with the following parameters
>    - *"Is this single or paired library"*: `Paired-end`
>       - {% icon param-collection %} *"FASTQ Paired Dataset"*: `Paired Reads`
>       - *"Do you want to set paired-end options?"*: `No`
>
>           You should have a look at the parameters there, specially the mate orientation if you know it. They can improve the quality of the paired-end mapping.
>
>     - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>       - *"Select reference genome"*: `Mouse (Mus musculus): mm10`
>     - *"Select analysis mode"*: `Default setting only`
>
>         You should have a look at the non default parameters and try to understand them. They can have an impact on the mapping and improving it.
>
>     - *"Save the bowtie2 mapping statistics to the history"*: `Yes`
>
> 2. Inspect the `mapping stats` file by clicking on the {% icon galaxy-eye %} (eye) icon
>
>
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What information is provided here?
> 2. How many reads have been mapped exactly 1 time?
> 3. How many reads have been mapped more than 1 time? How is it possible? What should we do with them?
> 4. How many pair of reads have not been mapped? What are the causes?
>
> > <solution-title></solution-title>
> > 1. The information given here is a quantity one. We can see how many sequences are aligned. It does not tell us something about the quality.
> > 2. ~90% reads have been aligned exactly 1 time
> > 3. ~7% reads have been aligned concordantly >1 times. These are called multi-mapped reads. It can happen because of repetitions in the reference genome (multiple copies of a gene for example), particularly when the reads are small. It is difficult to decide where these sequences come from and therefore most of the pipelines ignore them. Always check the statistics there to be sure of not discarding too much information in any downstream analyses.
> > 4. ~3% pair of reads have not been mapped because
> >     - both reads in the pair aligned but their positions do not concord with pair of reads (`aligned discordantly 1 time`)
> >     - reads of these pairs are multi-mapped (`aligned >1 times` in `pairs aligned 0 times concordantly or discordantly`)
> >     - one read of these pairs are mapped but not the paired read (`aligned exactly 1 time` in `pairs aligned 0 times concordantly or discordantly`)
> >     - the rest are not mapped at all
>  {: .solution }
{: .question}

Checking the mapping statistics is an important step to do before continuing any analyses. There are several potential sources for errors in mapping, including (but not limited to):

- **Polymerase Chain Reaction (PCR) artifacts**: Many high-throughput sequencing (HTS) methods involve one or multiple PCR steps. PCR errors will show as mismatches in the alignment, and especially errors in early PCR rounds will show up in multiple reads, falsely suggesting genetic variation in the sample. A related error would be PCR duplicates, where the same read pair occurs multiple times, skewing coverage calculations in the alignment.
- **Sequencing errors**: The sequencing machine can make an erroneous call either for physical reasons (e.g. oil on an Illumina slide) or due to properties of the sequenced DNA (e.g., homopolymers). As sequencing errors are often random, they can be filtered out as singleton reads during variant calling.
- **Mapping errors**: The mapping algorithm can map a read to the wrong location in the reference. This often happens around repeats or other low-complexity regions.

So if the mapping statistics are not good, you should investigate the cause of these errors before going further in your analyses.

After that, you should have a look at the reads and inspect the BAM file where the read mappings are stored.

# Inspection of a BAM file

{% include topics/sequence-analysis/tutorials/mapping/bam_explanation.md mapper="Bowtie2" %}

The BAM file includes a lot of information about each read, particularly the quality of mapping.

> <hands-on-title>Summary of mapping quality</hands-on-title>
> 1. {% tool [Samtools Stats](toolshed.g2.bx.psu.edu/repos/devteam/samtools_stats/samtools_stats/2.0.8) %} with the following parameters
>    - {% icon param-file %} *"BAM file"*: `aligned reads` (output of **Bowtie2** {% icon tool %})
>    - *"Use reference sequence"*: `Use a built-in genome`
>      - *"Using genome"*: `Mouse (Mus musculus): mm10 Full`
>
>    > <tip-title> No options for reference sequence? </tip-title>
>    > If you do not see a list of options for the reference sequence, double-check that you selected a reference genome during the previous step (Bowtie2).
>    >
>    > You can also set the **"Database/build"** attribute on the input file manually (see below). Make sure to set it to *Mouse (Mus musculus): mm10 Full*
>    >
>    > {% snippet faqs/galaxy/datasets_change_dbkey.md %}
>    >
>    {: .tip}
>
> 2. Inspect the {% icon param-file %} `Stats` file
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What is the proportion of mismatches in the mapped reads when aligned to the reference genome?
> 2. What does the error rate represent?
> 3. What is the average quality? How is it represented?
> 4. What is the insert size average?
> 5. How many reads have a mapping quality score below 20?
>
> > <solution-title></solution-title>
> > 1. There are ~21,900 mismatches for ~4,753,900 bases mapped which on average produces ~0.005 mismatches per mapped bases.
> > 2. The error rate is the proportion of mismatches per mapped bases, so the ratio computed right before.
> > 3. The average quality is the mean quality score of the mapping. It is a Phred score like the one used in the FASTQ file for each nucleotide. But here the score is not per nucleotide, but per read and it represents the probability of mapping quality.
> > 4. The insert size is the distance between the two reads in the pairs.
> > 5. To get the info:
> >      1. {% tool [Filter BAM](toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.5.2+galaxy2) %} with a filter to keep only the reads with a mapping quality >= 20
> >      2. {% tool [Samtools Stats](toolshed.g2.bx.psu.edu/repos/devteam/samtools_stats/samtools_stats/2.0.5) %} on the output of **Filter**
> >
> >    Before filtering: 95,412 reads and after filtering: 89,664 reads.
>  {: .solution }
{: .question}

# Visualization using a Genome Browser

## IGV

The Integrative Genomics Viewer (IGV) is a high-performance visualization tool for interactive exploration of large, integrated genomic datasets. It supports a wide variety of data types, including array-based and next-generation sequence data, and genomic annotations. In the following, we will use it to visualize the mapped reads.

{% include topics/sequence-analysis/tutorials/mapping/igv.md tool="Bowtie2" region_to_zoom="chr2:98,666,236-98,667,473" %}

## JBrowse

{% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy0) %} is an alternative, web-based genome browser. Whereas IGV is a piece of software you must download and run, JBrowse instances are websites hosted online that provide an interface to browse genomics data. We'll use it to visualise the mapped reads.

{% include topics/sequence-analysis/tutorials/mapping/jbrowse.md tool="Bowtie2" region_to_zoom="chr2:98,666,236-98,667,473" %}

# Conclusion

After quality control, mapping is an important step of most analyses of sequencing data (RNA-Seq, ChIP-Seq, etc) to determine where in the genome our reads originated from and use this information for downstream analyses.

tools_json:
["Bowtie2", "toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.5.4+galaxy0", "Samtools Stats", "toolshed.g2.bx.psu.edu/repos/devteam/samtools_stats/samtools_stats/2.0.8", "Filter BAM", "toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.5.2+galaxy2", "toolshed.g2.bx.psu.edu/repos/devteam/samtools_stats/samtools_stats/2.0.5", "JBrowse", "toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy0"]


topic: sequence-analysis
tutorial: Manage_AB1_Sanger
title: Clean and manage Sanger sequences from raw files to aligned consensus
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: []
content_hash: 4f8cabe0c76dc172ab2e25d118003b67
last_modified: 2026-05-15T22:28:14.600223
zenodo_link: https://zenodo.org/records/7104640
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/sequence-analysis/tutorials/Manage_AB1_Sanger/tutorial.html

content:
The objective of this tutorial is to learn how to clean and manage AB1 data files freshly obtained from Sanger sequencing. 
This kind of sequencing is targeting a specific sequence with short single DNA strands called primers. These primers are delimiting ends of the targeted marker.
Usually, one gets two .ab1 files for each sample, representing the sense (forward) and the antisense (reverse) strands of DNA.

Here, we'll be using raw data from {% cite Garavaglia2022 %}.
In this article, two DNA markers are investiguated CHD8 (Chromodomain-helicase-DNA-binding protein 8) and AOPEP (Aminopeptidase O Putative).
We'll focus on CHD8 sequences but you can try to apply the same steps on the AOPEP sequences to practice after the tutorial !

In the first section of the tutorial, we'll be preparing primer's data by:
- selecting the right primer sequences with the identifier;
- removing eventual gaps included in the sequences;
- and compute the reverse-complement sequence for the antisense primer only.

In the second section of the tutorial, we'll be preparing the Sanger sequences data by:
- extracting ab1 files of the interest sequence (CHD8) and separating sense and antisense sequences in two distinct data collections;
- converting ab1 files to FASTQ to permit its use in the following tools;
- trimming low quality ends of the sequences;
- compute the reverse-complement for the antisense sequence only;
- align sense and antisense sequences;
- obtain a consensus sequence (which results the correspondance between nucleotides of the sense and the antisense sequences) for each three samples.

In the third section of the tutorial, primers and all consensus sequences are finally merged into a single file to be aligned and verified.

> <details-title>What is the reverse-complement sequence and why computing it for the antisense sequence?</details-title>
>
> Consider a double-strand DNA molecule with the following sequences:
> ![Double-strand DNA](../../images/Sanger_tuto/Double-strand_DNA.png "Double-strand DNA")
> 
> When sequencing, each strand of DNA are read separately in the 5'-3' orientation. Hence, in the sequence files each strand are provided as:
> ![Single-strand DNA sequences in output file](../../images/Sanger_tuto/two_single-strand_sequences.png "Single-strand DNA sequences in output file")
>
> To get the antisense sequence in its original orientation, the reverse sequence is computed:
> ![Reversed antisense sequence](../../images/Sanger_tuto/Reversed-antisense-sequence.png "Reversed antisense sequence")
>
> To align sense and antisense sequence, the complement sequence of the reversed antisense sequence is computed:
> ![Reverse-complement antisense sequence](../../images/Sanger_tuto/Reversed-antisense-sequence.png "Reversed antisense sequence")
>
> The two sequences can be aligned now:
> ![Aligned sense and antisense sequences](../../images/Sanger_tuto/Aligned_sense_antisense.png "Aligned sense and antisense sequences")
>
{: .details}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Get data

Authors of {% cite Garavaglia2022 %} have shared openly their raw AB1 files on Zenodo.

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) :
>
>    ```
>    https://zenodo.org/records/7104640/files/AOPEP_and_CHD8_sequences_20220907.zip
>    ```
>    
>    Change **Type (set all):** from "Auto-detect" to `zip` and click **Start**
>    
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 4. Create primer FASTA file, copy:
>    ```
>    >Forward_CHD8
>    GAGGTGAAAGAATCATAAATTGG
>    >Reverse_CHD8
>    CCCTGTGTACAAATAGCTTTTGT
>    >Forward_AOPEP
>    TCATGGTTCCAGGCAGAGTTATT
>    >Reverse_AOPEP
>    TGCTGTGACAAGCCAACCAATGG
>    ```
>    - Open the Galaxy Upload Manager ({% icon galaxy-upload %} on the top-right of the tool panel)
>    - Select **Paste/Fetch Data**
>    - Paste into the text field
>    - Change **Type (set all):** from "Auto-detect" to `fasta`
>    - Change the name from "New File" to "Primer file"
>    - Click **Start**
>
>      
>    Note these primer sequences were invented for the purpose of the tutorial, it is not the sequences used in the publication.
>
{: .hands_on}

# Prepare primer data

## Separate and format primers files

Primers must be separated in distinct files because sense (forward) and antisense (reverse) primers won't be subjected to the same formating.

> <hands-on-title> Create separate files for each primer </hands-on-title>
>
> 1. {% tool [Filter FASTA](toolshed.g2.bx.psu.edu/repos/galaxyp/filter_by_fasta_ids/filter_by_fasta_ids/2.3) %} with the following parameters:
>    - {% icon param-file %} *"FASTA sequences"*: `Primer file`
>    - *"Criteria for filtering on the headers"*: `Regular expression on the headers`
>        - *"Regular expression pattern the header should match"*: `Reverse_CHD8`
>    - Add tags "#Primer" and "#Reverse"
>      
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
>    {% snippet faqs/galaxy/tools_rerun.md %}
>
> 2. {% tool [Filter FASTA](toolshed.g2.bx.psu.edu/repos/galaxyp/filter_by_fasta_ids/filter_by_fasta_ids/2.3) %} with the following parameters:
>    - {% icon param-file %} *"FASTA sequences"*: `Primer file`
>    - *"Criteria for filtering on the headers"*: `Regular expression on the headers`
>        - *"Regular expression pattern the header should match"*: `Forward_CHD8`
>    - Add tags "#Primer" and "#Forward"
>      
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
> 3. Remove eventual gaps from primers {% tool [Degap.seqs](toolshed.g2.bx.psu.edu/repos/iuc/mothur_degap_seqs/mothur_degap_seqs/1.39.5.0) %} with the following parameters:
>    
>    {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>    
>    - {% icon param-files %} *"fasta - Dataset"*: `Two Filter FASTA outputs` (outputs of **Filter FASTA** {% icon tool %})
>
{: .hands_on}

In this previous hands-on, the step of removing eventual gaps (`-` in the FASTA files) is a precaution, there are no gaps in our primers file. However, it is important to remove gaps at this point in case you are using different data, otherwise some steps of the tutorial could fail (e.g. alignment).

This following hands-on is to be applied only on the sequence of the antisense (reverse) primer. 

> <hands-on-title> Compute Reverse-Complement of the antisense (reverse) primer </hands-on-title>
>
> 1. {% tool [Reverse-Complement](toolshed.g2.bx.psu.edu/repos/devteam/fastx_reverse_complement/cshl_fastx_reverse_complement/1.0.2+galaxy0) %} the sequence antisense (reverse) primer with the following parameters:
>    - {% icon param-file %} *"Input file in FASTA or FASTQ format"*: `Degap.seqs #Reverse FASTA output` (output of **Degap.seqs** {% icon tool %})
> 
> See in the introduction for explanations on the Reverse-Complement.
> 
{: .hands_on}

# Prepare sequence data

## Unzip data files

> <hands-on-title> Unzip </hands-on-title>
>
> 1. {% tool [Unzip](toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"input_file"*: `AOPEP_and_CHD8_sequences_20220907.zip?download=1`
>    - *"Extract single file"*: `All files`
>
> > <question-title></question-title>
> > How many files is there in the ZIP archive ?
> >
> > > <solution-title></solution-title>
> > > 12 (if you have a different number of files something likely went wrong)
> > >
> > {: .solution}
> >
> {: .question}
>
{: .hands_on}

From now on, we'll be working a lot on data collections:

{% snippet faqs/galaxy/tools_select_collection.md %}

## Filter collection to separate sense and antisense sequence files

As for primers, sense and antisense sequences will be subjected to slightly different procedures so they must be separated in distinct data collections.

> <hands-on-title> Filter </hands-on-title>
>
> 1. {% tool [Extract element identifiers](toolshed.g2.bx.psu.edu/repos/iuc/collection_element_identifiers/collection_element_identifiers/0.0.2) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: `output collection` (output of **Unzip** {% icon tool %})
>
> 2. {% tool [Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `output` (output of **Extract element identifiers** {% icon tool %})
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `^[A-Za-z0-9_-]+F$`
>            - *"Replacement"*: ``
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `^[A-Za-z0-9_-]+AOPEP[A-Za-z0-9_-]+$`
>            - *"Replacement"*: ``
>    - Tag output with "#Reverse"
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
>    {% snippet faqs/galaxy/tools_rerun.md %}
>
> 4. {% tool [Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `output` (output of **Extract element identifiers** {% icon tool %})
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `^[A-Za-z0-9_-]+R$`
>            - *"Replacement"*: ``
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `^[A-Za-z0-9_-]+AOPEP[A-Za-z0-9_-]+$`
>            - *"Replacement"*: ``
>    - Tag output with "#Forward"
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
> 5. {% tool [Filter collection](__FILTER_FROM_FILE__) %} with the following parameters:
>    - {% icon param-collection %} *"Input Collection*: `output collection` (output of **Unzip** {% icon tool %})
>    - *"How should the elements to remove be determined?"*: `Remove if identifiers are ABSENT from file`
>        - {% icon param-files %} *"Filter out identifiers absent from"*: `#Forward files list` & `#Reverse files list` (output of **Regex Find And Replace** {% icon tool %})
>    - Tag `(filtered)` outputs with "#Forward" and "#Reverse"
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>    
>    > <comment-title> What's happening in this section? </comment-title>
>    >
>    > First step: Extracting the list of file names in the data collection
>    > Second step: Removing file names containing a "F" and "AOPEP" -> creating a list of antisense (reverse) sequence files of the marker CHD8
>    > Third step: Removing file names containing a "R" and "AOPEP" -> creating a list of sense (forward) sequence files of the marker CHD8
>    > Fourth step: Select files in the collection -> creating two distinct collections with sense (forward) sequence files on one hand and antisense (reverse) sequence file on the other hand
>    >
>    > For the second and third step, we used regular expressions (Regex):
>    > 
>    > {% snippet  faqs/galaxy/analysis_regular_expressions.md %}
>    > 
>    > With `[A-Za-z0-9_-]` meaning any character between A to Z, a to z, 0 to 9 or _ or -, the following `+` meaning that any of these characters are found once or more. 
>    > 
>    {: .comment}
>
{: .hands_on}

## Convert AB1 sequence files to FASTQ and trim low-quality ends

In Sanger sequencing, ends tend to be of low trust levels (each nucleotide has a quality score reflecting this trust level), it is important to delete these sections of the sequences to ensure wrong nucleotides aren't introduced in the sequences. 

> <hands-on-title> AB1 to FASTQ files and trim low quality ends </hands-on-title>
> 
> Do these steps twice !! We have Froward and antisense (reverse) sequence data collections, do these steps starting with each "(filtered)" data collections, this could help:
> 
> {% snippet faqs/galaxy/tools_rerun.md %}
>
> 1. {% tool [ab1 to FASTQ converter](toolshed.g2.bx.psu.edu/repos/ecology/ab1_fastq_converter/ab1_fastq_converter/1.20.0) %} with the following parameters:
>    - {% icon param-collection %} *"Input ab1 file"*: `(filtered) output collection` (output of **Filter collection** {% icon tool %})
>    - *"Do you want trim ends according to quality scores ?"*: `No, use full sequences.`
> 
> In this tool, it is possible to trim low-quality ends along with the conversion of the file but parametrization is less precise.
> 
> 2. {% tool [seqtk_trimfq](toolshed.g2.bx.psu.edu/repos/iuc/seqtk/seqtk_trimfq/1.3.1) %} with the following parameters:
>    - {% icon param-collection %} *"Input FASTA/Q file"*: `output collection` (output of **ab1 to FASTQ converter** {% icon tool %})
>    - *"Mode for trimming FASTQ File"*: `Quality`
>        - *"Maximally trim down to INT bp"*: `0`
>
{: .hands_on}

## Compute reverse complement sequence for antisense (reverse) sequences only 

See in the introduction for explanations on the Reverse-Complement.

> <hands-on-title> Reverse complement </hands-on-title>
>
> 1. {% tool [FASTQ Groomer](toolshed.g2.bx.psu.edu/repos/devteam/fastq_groomer/fastq_groomer/1.1.5) %} with the following parameters:
>    - {% icon param-collection %} *"File to groom"*: `#Reverse output collection` (output of **seqtk_trimfq** {% icon tool %})
>    - *"Advanced Options"*: `Show Advanced Options`
>        - *"Summarize input data"*: `Do not Summarize Input (faster)`
>
>    > <comment-title> What is this step? </comment-title>
>    >
>    > It is a necessary step to get the right input format for the following step **Reverse-Complement** {% icon tool %}
>    {: .comment}
>
> 2. {% tool [Reverse-Complement](toolshed.g2.bx.psu.edu/repos/devteam/fastx_reverse_complement/cshl_fastx_reverse_complement/1.0.2+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Input file in FASTA or FASTQ format"*: `#Reverse output collection` (output of **FASTQ Groomer** {% icon tool %})
>
>
{: .hands_on}

## Merge corresponding sense and antisense sequences single files

> <hands-on-title> Sort collections </hands-on-title>
> 
> Do this step twice !! One has to make sure sense (forward) and antisense (reverse) sequences collections are in the same order to get the right sense and the right antisense sequence to be merged together
>
> {% snippet faqs/galaxy/tools_rerun.md %}
>
> 1. {% tool [Sort collection](__SORTLIST__) %} with the following parameters:
>    - {% icon param-collection %} *"Input Collection"*: `Collection` (output of **seqtk_trimfq** {% icon tool %} & output of **Reverse-Complement** {% icon tool %})
>    - *"Sort type"*: `alphabetical`
>
{: .hands_on}

> <hands-on-title> Merge sense (forward) and antisense (reverse) sequence files </hands-on-title>
>
> 1. {% tool [seqtk_mergepe](toolshed.g2.bx.psu.edu/repos/iuc/seqtk/seqtk_mergepe/1.3.1) %} with the following parameters:
>    - {% icon param-collection %} *"Input FASTA/Q file #1"*: `output` (output of **Sort collection** {% icon tool %})
>    - {% icon param-collection %} *"Input FASTA/Q file #2"*: `output` (output of **Sort collection** {% icon tool %})
>
> Check there is two sequences in each three files of the newly-created collection.
>
{: .hands_on}

## Convert FASTQ files to FASTA

> <hands-on-title> FASTQ to FASTA </hands-on-title>
>
> 1. {% tool [FASTQ Groomer](toolshed.g2.bx.psu.edu/repos/devteam/fastq_groomer/fastq_groomer/1.1.5) %} with the following parameters:
>    - {% icon param-collection %} *"File to groom"*: `default` (output of **seqtk_mergepe** {% icon tool %})
>    - *"Advanced Options"*: `Show Advanced Options`
>        - *"Summarize input data"*: `Do not Summarize Input (faster)`
>
>    > <comment-title> What is this step? </comment-title>
>    >
>    > It is a necessary step to get the right input format for the following step **FASTQ to FASTA** {% icon tool %}
>    > 
>    {: .comment}
>
> 1. {% tool [FASTQ to FASTA](toolshed.g2.bx.psu.edu/repos/devteam/fastq_to_fasta/cshl_fastq_to_fasta/1.0.2+galaxy2) %} with the following parameters:
>    - {% icon param-collection %} *"FASTQ file to convert"*: `output collection` (output of **FASTQ Groomer** {% icon tool %})
>    - *"Discard sequences with unknown (N) bases"*: `no`
>    - *"Rename sequence names in output file (reduces file size)"*: `no`
>    - *"Compress output FASTA"*: `No`
>
>    > <comment-title> information </comment-title>
>    >
>    > If this step doesn't work, one can try tools **FASTQ to tabular** {% icon tool %} and **tabular to FASTA** {% icon tool %} instead
>    {: .comment}
>
{: .hands_on}

## Align sequences and retrieve consensus for each sequence

> <hands-on-title> Align and consensus </hands-on-title>
>
> 1. {% tool [Align sequences](toolshed.g2.bx.psu.edu/repos/iuc/qiime_align_seqs/qiime_align_seqs/1.9.1.0) %} with the following parameters:
>    - {% icon param-collection %} *"Input fasta file"*: `output collection` (output of **FASTQ-to-FASTA** {% icon tool %})
>    - *"Method for aligning sequences"*: `clustalw`
>    - *"Minimum percent sequence identity to closest blast hit to include sequence in alignment"*: `0.1`
>
> 2. {% tool [Consensus sequence from aligned FASTA](toolshed.g2.bx.psu.edu/repos/ecology/aligned_to_consensus/aligned_to_consensus/1.0.0) %} with the following parameters:
>    - {% icon param-collection %} *"Input fasta file with at least two sequences"*: `aligned_sequences` (output of **Align sequences** {% icon tool %})
>    - Add tag "#Consensus"
>
> 3. {% tool [Merge.files](toolshed.g2.bx.psu.edu/repos/iuc/mothur_merge_files/mothur_merge_files/1.39.5.0) %} with the following parameters:
>    - *"Merge"*: `fasta files`
>        - {% icon param-collection %} *"inputs - fasta"*: `output collection` (output of **Consensus sequence from aligned FASTA** {% icon tool %})
>
{: .hands_on}

# Manage primers and sequences 

## Merge and align consensus sequences file and primer files

> <hands-on-title> Merge and format consensus sequences + primers file </hands-on-title>
>
> 1. {% tool [Merge.files](toolshed.g2.bx.psu.edu/repos/iuc/mothur_merge_files/mothur_merge_files/1.39.5.0) %} with the following parameters:
>    - *"Merge"*: `fasta files`
>        - {% icon param-files %} *"inputs - fasta"*: `consensus sequences` (output of **Merge.files** {% icon tool %}), `Reverse primer` (output of **Reverse-Complement** {% icon tool %}), `Forward primer` (output of **Degap.seqs** {% icon tool %})
> 
>    {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
> 
>    - Remove tags "#Forward" and "#Reverse"
>
> 2. {% tool [Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `output` (output of **Merge.files** {% icon tool %})
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `([A-Z-])>`
>            - *"Replacement"*: `\1\n>`
>
>    > <comment-title> What's going on in this second step? </comment-title>
>    >
>    > Sometimes, **Merge.files** {% icon tool %} doesn't keep linefeed between the files, this step permits to correct it and get a FASTA file that is formatted properly.
>    > 
>    > For the second step, we used regular expressions (Regex):
>    > 
>    > {% snippet  faqs/galaxy/analysis_regular_expressions.md %}
>    > 
>    > With `[A-Z-]` meaning any character between A to Z or -, `\1` repeat the character chain between brackets in the *"Find Regex"* section, `\n` meaning a line-feed. 
>    > 
>    {: .comment}
>
{: .hands_on}

When you have the consensus sequences, you can check if any ambiguous nucleotide is to be found in the sequences. If you find such nucleotides, it means different nucleotides were found in the sense and antisense sequence at the same position, some checks are needed.

> <details-title>Ambiguous nucleotide correspondance</details-title>
>
> - Y = C or T
> - R = A or G
> - W = A or T
> - S = G or C
> - K = T or G
> - M = C or A
>
{: .details}

> <hands-on-title> Look for ambiguous nucleotides </hands-on-title>
>
> 1. Click on output of **Regex Find and Replace** {% icon tool %} in the history to expand it
>
> 2. Click on {% icon galaxy-barchart %} Visualize
>
> 3. Select **Multiple Sequence Alignment**
> 
> 4. Set color scheme to `Clustal`, ambiguous nucleotides are highlighted in dark blue
>    
> 5. There are two nucleotide positions to check, Y at 121 in sequence `consensus_B05_CHD8-III6brother-18` and W at 286 in sequence `consensus_05_CHD8-III6mother-18`
>
> 6. You need to go back to your FASTQ sequences to understand the origin of the ambiguity
>
> 7. {% tool [Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `#Consensus #Primer output` (output of **Regex Find and Replace** {% icon tool %})
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `^[ACTG]+([ACTG]{20}Y)[ACTG]+$`
>            - *"Replacement"*: `\1`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `^[ACTG]+([ACTG]{20}W)[ACTG]+$`
>            - *"Replacement"*: `\1`
> 
>    > <comment-title> What's going on in this step? </comment-title>
>    >
>    > We want to retrieve the 20 nucleotides before the ambiguities.
>    > 
>    > We use regular expressions (Regex):
>    > 
>    > {% snippet  faqs/galaxy/analysis_regular_expressions.md %}
>    > 
>    > With `[ACTG]` meaning any character of the four unambiguous nucleotides followed by `+` meaning "at least once in the character chain" or by {20} meaning "20 times".
>    > 
>    {: .comment}
>
>      In the output of this tool we get:
>        - the 20 nucleotides before the Y at position 121 in sequence `consensus_B05_CHD8-III6brother-18`: `CAGGCACGATGTCATCGAAT`
>        - and the 20 nuleotides before the W at position 286 in sequence `consensus_05_CHD8-III6mother-18`: `AGTCCTCTTAGTTTATAGAT`
>
> 8. {% tool [FASTQ masker](toolshed.g2.bx.psu.edu/repos/devteam/fastq_masker_by_quality/fastq_masker_by_quality/1.1.5) %} with the following parameters:
>    - {% icon param-collection %} *"File to mask"*: `#Forward #Reverse collection` (output of **FASTQ groomer** {% icon tool %})
>    - *"Mask input with"*: `Lowercase`
>    - *"Quality score"*: `10`
>      
>   This tool displays low-quality bases in lowercase to permit better detection of potential errors.
> 
> 9. Open {% icon galaxy-eye %} `B05_CHD8-III6brother-18` output of **FASTQ masker** {% icon tool %} and ctrl+f : `CAGGCACGATGTCATCGAAT`.
>    In the sense sequence (ID ending with 18F), this fragment is followed by a `c` in low-quality, whereas in the antisense sequence it is followed by a `T` in decent quality.
>    Additionally, when looking into the {% icon galaxy-eye %} `#Consensus #Primer` output of **Regex Find and Replace** {% icon tool %}, we can see the two other consensus sequences (`consensus_05_CHD8-III6mother-18` and `consensus_07_CHD8-III6-18`) have a `T` at this same position.
>    It seems more likely that the nucleotide at position 121 in sequence `consensus_B05_CHD8-III6brother-18` is a `T`.
>     
> 10. Open {% icon galaxy-eye %} `05_CHD8-III6mother-18` outputs of **FASTQ masker** {% icon tool %} and ctrl+f : `AGTCCTCTTAGTTTATAGAT`.
>     In the antisense sequence (ID ending with 18R), this fragment is followed by a `t` in low-quality, whereas in the sense sequence it is followed by a `A` in decent quality.
>     Additionally, when looking into the {% icon galaxy-eye %} `#Consensus #Primer` output of **Regex Find and Replace** {% icon tool %}, we can see the two other consensus sequences (`consensus_B05_CHD8-III6brother-18` and `consensus_07_CHD8-III6-18`) have a `A` at this same position.
>     It seems more likely that the nucleotide at position 286 in sequence `consensus_05_CHD8-III6mother-18` is a `A`.
>
> 11. You can now correct them by clicking on output of **Regex Find and Replace** {% icon tool %} in the history to expand it
>
> 12. Click on {% icon galaxy-barchart %} Visualize
>
> 13. Select **Editor** and:
>     - replace manually the `Y` with `T` in `consensus_B05_CHD8-III6brother-18`
>     - replace manually the `W` with `A` in `consensus_05_CHD8-III6mother-18`
>     and click on **export** 
> 
{: .hands_on}

Now, one can align its sequences with primers. Ultimately, it is common to cut sequences between primers to get the right fragment for each sequence.

> <hands-on-title> Align sequences and primers </hands-on-title>
>
> 1. {% tool [Align sequences](toolshed.g2.bx.psu.edu/repos/iuc/qiime_align_seqs/qiime_align_seqs/1.9.1.0) %} with the following parameters:
>    - {% icon param-file %} *"Input fasta file"*: `out_file1` **Regex Find And Replace (modified)**
>    - *"Method for aligning sequences"*: `mafft`
>    - *"Minimum percent sequence identity to closest blast hit to include sequence in alignment"*: `0.1`
>
{: .hands_on}

## Check your sequences belongs to the right taxonomic group by computing a BLAST on the NCBI database

> <hands-on-title> NVBI Blast </hands-on-title>
>
> 1. {% tool [NCBI BLAST+ blastn](toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastn_wrapper/2.10.1+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Nucleotide query sequence(s)"*: `out_file1` (output of **Regex Find And Replace** {% icon tool %})
>    - *"Subject database/sequences"*: `Locally installed BLAST database`
>        - *"Nucleotide BLAST database"*: select most recent `nt_` database
>    - *"Output format"*: `Tabular (select which columns)`
>        - *"Standard columns"*: `qseqid`, `pident`, `mismatch` and `gapopen`
>        - *"Extended columns"*: `gaps` and `salltitles`
>        - *"Other identifier columns"*: `saccver`
>    - *"Advanced Options"*: `Show Advanced Options`
>        - *"Maximum hits to consider/show"*: `10`
>        - *"Restrict search of database to a given set of ID's"*: `No restriction, search the entire database`
>
> > <question-title></question-title>
> > The sequences we cleaned belong to what species?
> >
> > > <solution-title></solution-title>
> > > *Homo sapiens*
> > >
> > {: .solution}
> >
> {: .question}
>
{: .hands_on}

It is a good practice to proceed to such checks, its permits to make sure the sequencing went as planned and your samples haven't been contaminated. 

# Conclusion

We successfully cleaned AB1 sequence files !

# AOPEP Sanger files

The history following the same steps but for AOPEP marker files is available: [Clean AOPEP sequences](https://usegalaxy.fr/u/coline_royaux/h/tuto-sanger-w-aopep)

tools_json:
["Filter FASTA", "toolshed.g2.bx.psu.edu/repos/galaxyp/filter_by_fasta_ids/filter_by_fasta_ids/2.3", "Degap.seqs", "toolshed.g2.bx.psu.edu/repos/iuc/mothur_degap_seqs/mothur_degap_seqs/1.39.5.0", "Reverse-Complement", "toolshed.g2.bx.psu.edu/repos/devteam/fastx_reverse_complement/cshl_fastx_reverse_complement/1.0.2+galaxy0", "Unzip", "toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy0", "Extract element identifiers", "toolshed.g2.bx.psu.edu/repos/iuc/collection_element_identifiers/collection_element_identifiers/0.0.2", "Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.3", "Filter collection", "__FILTER_FROM_FILE__", "ab1 to FASTQ converter", "toolshed.g2.bx.psu.edu/repos/ecology/ab1_fastq_converter/ab1_fastq_converter/1.20.0", "seqtk_trimfq", "toolshed.g2.bx.psu.edu/repos/iuc/seqtk/seqtk_trimfq/1.3.1", "FASTQ Groomer", "toolshed.g2.bx.psu.edu/repos/devteam/fastq_groomer/fastq_groomer/1.1.5", "Sort collection", "__SORTLIST__", "seqtk_mergepe", "toolshed.g2.bx.psu.edu/repos/iuc/seqtk/seqtk_mergepe/1.3.1", "FASTQ to FASTA", "toolshed.g2.bx.psu.edu/repos/devteam/fastq_to_fasta/cshl_fastq_to_fasta/1.0.2+galaxy2", "Align sequences", "toolshed.g2.bx.psu.edu/repos/iuc/qiime_align_seqs/qiime_align_seqs/1.9.1.0", "Consensus sequence from aligned FASTA", "toolshed.g2.bx.psu.edu/repos/ecology/aligned_to_consensus/aligned_to_consensus/1.0.0", "Merge.files", "toolshed.g2.bx.psu.edu/repos/iuc/mothur_merge_files/mothur_merge_files/1.39.5.0", "FASTQ masker", "toolshed.g2.bx.psu.edu/repos/devteam/fastq_masker_by_quality/fastq_masker_by_quality/1.1.5", "NCBI BLAST+ blastn", "toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastn_wrapper/2.10.1+galaxy2"]


topic: sequence-analysis
tutorial: tapscan-streptophyte-algae
title: Identification and Evolutionary Analysis of Transcription-Associated Proteins
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: ["plants"]
content_hash: 18fd6b86f714bd6a9799835b0d73f0f2
last_modified: 2026-05-27T09:01:49.509496
zenodo_link: https://zenodo.org/records/15056031/
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/sequence-analysis/tutorials/tapscan-streptophyte-algae/tutorial.html

content:
The regulated expression of genes is essential for defining morphology, functional capacity, and developmental fate of both solitary living cells as well as cells inhabiting the social environment of a multicellular organism. In this regard, the regulation of transcription, that is, the synthesis of messenger RNA from a genomic DNA template, plays a crucial role. It contributes to the control of temporal and spatial RNA and protein levels in a cell and therefore has an essential function in all living organisms. Transcription‐associated proteins (TAPs) are essential players in gene regulatory networks (GRNs) as they involved in transcriptional regulation. TAPs are broadly classified into transcription factors (TFs) and transcriptional regulators (TRs). TFs bind sequence‐specifically to regulatory elements, resulting in enhancing or repressing of transcription ({% cite Richardt2007 %}; {% cite Wilhelmsson2017 %}). TRs, on the other hand, are involved in protein–protein interactions, may serve as regulators at the transcriptional core complex, as co‐activators and co‐repressors, chromatin modification or methylation. Additionally, there are proteins referred to as putative TAPs (PTs) that are thought to be involved in the regulation of transcription, but their exact function is undefined ({% cite Richardt2007 %}).

TAPScan v4 ({% cite Petroll2024 %}) is a comprehensive tool for annotating TAPs with a special focus on species belonging to the Archaeplastida. In general, the detection of TAPs is based on the detection of highly conserved protein domains.

In this tutorial, we will illustrate the identification of TAPs in streptophyte algae and land plants using [TAPScan Classify](https://github.com/Rensing-Lab/TAPscan-classify), followed by construction of the phylogenetic tree.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get data

In this tutorial, we will use representative protein sequences obtained from the [Genome Zoo database](https://github.com/Rensing-Lab/Genome-Zoo). The selected sequences represent different lineages:

- **Streptophyte Algae:**
  - *Chara braunii* (CHABR)
  - *Penium margaritaceum* (PENMA)

- **Bryophytes & Land Plants**
  - *Marchantia polymorpha* (MARPO)
  - *Physcomitrium patens* (PHYPAV6)
  - *Oryza sativa (spp. japonica)* (ORYSAJA)
  - *Selaginella moellendorffii* (SELMO)
  - *Arabidopsis thaliana* (ARATH)


> <hands-on-title> Prepare your Analysis History </hands-on-title>
>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history to something descriptive.
>    - for example "TAP analysis in Streptophyte algae and Land plants"
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
>
{: .hands_on}

Now, we need to import the data

> <hands-on-title>Import datasets</hands-on-title>
>
> 1. Import the following from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library:
>
>    ```
>    https://zenodo.org/records/15056031/files/ARATH.fa
>    https://zenodo.org/records/15056031/files/CHABR.fa
>    https://zenodo.org/records/15056031/files/MARPO.fa
>    https://zenodo.org/records/15056031/files/PENMA.fa
>    https://zenodo.org/records/15056031/files/SELMO.fa
>    https://zenodo.org/records/15056031/files/PHYPAV6.fa
>    https://zenodo.org/records/15056031/files/ORYSAJA.fa
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Create a dataset collection with all the input files
>    - name your collection `input sequences`
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
{: .hands_on}

If you were successful, all the input files should now be available as dataset collection in your history.

# Identification of TAPs

In order to detect TAPs from the given proteome(s), each sequence out of a species protein set is first scanned for protein domains (stored as profile Hidden Markov Models) using hmmsearch. The domains list consists of 154 profile HMMs and functions as the domain reference during the hmmsearch in background.

Afterwards, [TAPScan Classify](https://github.com/Rensing-Lab/TAPscan-classify) applies specialized rules to assign the protein sequences, scanned for 137 different TAP families, ensuring accurate family assignment using GA-thresholds and coverage values.

## Perform TAPScan Classify

Now that our dataset collection is ready, we can run [TAPScan Classify](https://github.com/Rensing-Lab/TAPscan-classify) to detect TAPs in all species.

> <hands-on-title> Detect TAPs </hands-on-title>
>
> 1. {% tool [TAPScan Classify](toolshed.g2.bx.psu.edu/repos/bgruening/tapscan/tapscan_classify/4.76+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Proteins in FASTA format"*: `input sequences` (the dataset collection you just made)
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
>    > <comment-title> on parameter </comment-title>
>    >
>    > If you would like to have Output table from the HMMer Search, Please Check on the box like below:
>    > - *"Output the HMMer domain hits table?"*: `Yes`
>    {: .comment}
>
{: .hands_on}

TAPScan Classify provides the user with three different output files. Each output file is tab-separated.

- **Output 1: "Detected TAPs"** - contains the detected domains and finally assigned TAP family for each gene ID. If domains are assigned to a sequence but not all rules are fulfilled, the sequence is assigned to *“0_no_family_found”*.
- **Output 2: "Family Counts"** - is a summary of the number of members for each TAP family.
- **Output 3: "Detected TAPs Extra"** - is similar to output 1 but contains additional information.

Below is an example of the "Detected TAPs Extra" output. It has 5 columns, the Sequence IDs, followed by the classification into a TAP family, subfamily, the number of classifications, and domains.

![example output](./images/tapscan-extra-output.png)

> <question-title></question-title>
>
> 1. What is the output format of TAPScan Classify?
> 2. How many sequences belong to the “0_no_family_found” in *Arabidopsis thaliana* (ARATH) ?
> 3. What extra information does Output 3 include compared to Output 1?
>
> > <solution-title>Hints</solution-title>
> >
> > 1. This is written on the history outputs in Galaxy
> > 2. Output 2, "Counts", contains this information
> > 3. Open one of the files in output 1 (Detected TAPs) and output 3 (Detected TAPs Extra), and compare the column headers
> >
> {: .solution}
>
>
> > <solution-title></solution-title>
> >
> > 1. Tabular
> > 2. 675, You can find this number by clicking on the eye button for Galaxy dataset output 2 for ARATH.
> > 3. Output 3 contains information about subfamilies with an additional column that provides more details beyond what is included in Output 1.
> >
> {: .solution}
>
{: .question}

Now that we have the output from TAPScan Classify, we can filter the results by selecting a TAP family so that we can run phylogenetic analysis on that. To demonstrate, we chose the Aux/IAA TAP family. A brief overview of this TAP family: Aux/IAA proteins function as transcriptional repressors (TRs) in the auxin signaling pathway. They inhibit the expression of auxin-responsive genes by dimerizing with auxin response factor (ARF) transcriptional activators. This repression prevents ARFs from activating their target genes, thereby controlling auxin-regulated processes such as cell division, elongation, and differentiation ({% cite Tiwari2004 %})

In order to filter the TAPScan output, we need to first filter either Output 1 or Output 3 by using the Filter tool. This will allow us to select only the entries corresponding to Aux/IAA TAP family.

> <hands-on-title> Filter the TAPScan output Based on the Column  </hands-on-title>
>
> 1. {% tool [Filter - data on any column using simple expressions](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `Detected TAPs` (Output 1 of **TAPScan Classify** {% icon tool %})
>    - *"With following condition"*: `c2=='Aux/IAA'`
>    - *"Number of header lines to skip"*: `1`
>
>    > <comment-title> What's happening in this section? </comment-title>
>    >
>    > This step filters the TAPScan results to retain only sequences classified as belonging to the our desired (Aux/IAA) TAP family.
>    {: .comment}
>    > <question-title></question-title>
>    >
>    > How many sequences belong to the 'Aux/IAA' TAP family in *Arabidopsis thaliana* (ARATH), excluding the header line?
>    >
>    > > <solution-title></solution-title>
>    > > 29, You can find this number by expanding the Galaxy dataset output for ARATH.
>    > >
>    > > ![screenshot of expanded arath output](./images/arath-filtered-expanded.png)
>    > >
>    > > you can see "30 lines, 4 columns", but one line is the header line, so 29 sequences in total
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}


Next, we would like to create FASTA files containing only those sequences belonging to this TAP family. In order to do that, we first need
to create a list of Sequnce IDs. This information is already in the first column of our output, so we will cut the first column to retain only the sequence IDs. Then, we will remove the header line to ensure a clean list of sequence identifiers for further analysis.


> <hands-on-title> Cut and Remove Header from Sequence IDs </hands-on-title>
>
> 1. {% tool [Cut - columns from a table](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - {% icon param-collection %} *"From"*: output of **Filter** {% icon tool %}
>
>
> 2. {% tool [Remove beginning - of a file](Remove beginning1) %} with the following parameters:
>
>    - *"Remove first\*"*: `1`
>    - {% icon param-collection %} *"from"*: output of **Cut** {% icon tool %}
>
> 3. **Examine** {% icon galaxy-eye %} the output
>    - You should have a list of only sequence IDs
>      ![screenshot of output file for ARASH](./images/sequence-ids.png)
>
> > <question-title></question-title>
> >
> > 1. How many sequences belonged to our TAP family for *Marchantia polymorpha* (MARPO)?
> > 2. What is the average number of sequences in the Aux/IAA TAP family for the Streptophyte Algae? [ Chara braunii (CHABR) and Penium margaritaceum (PENMA)]?
> > 3. And for Bryophytes & Land Plants?
> > 4. What does this suggest?
> >
> > > <solution-title></solution-title>
> > > 1. MARPO output has 3 lines, so that means 3 of the gene sequence in Marchantia polymorpha belonged to the Aux/IAA TAP family.
> > > 2. CHABR has 2 sequences, PENMA has 3, so the average is **2.5**
> > > 3. For the Bryohpytes & Land plants we have ARATH(29) + MARPO(3) + SELMO (10) + PHYPAV6 (5) + ORYSAJA (33)  = 80, divided by 5 is average of **16 genes**
> > > 4. While our dataset is quite small, we observe a lower number of Aux/IAA proteins in streptophyte algae compared to bryophytes and land plants.
> > >    This suggests that the Aux/IAA protein family expanded as plants transitioned from aquatic to terrestrial environments, where more complex signaling mechanisms were required for growth regulation.
> > {: .solution}
> {: .question}
>
{: .hands_on}


Now that we have our list of sequence IDs belonging to the Aux/IAA TAP family, we will now extract the corresponding FASTA sequences from our input dataset. We will use these FASTA sequences to create our evolutionary tree.


## Extract the sequences for TAP families

> <hands-on-title> Extract the FASTA sequences </hands-on-title>
>
> 1. {% tool [Filter FASTA - on the headers and/or the sequences](toolshed.g2.bx.psu.edu/repos/galaxyp/filter_by_fasta_ids/filter_by_fasta_ids/2.3) %} with the following parameters:
>    - {% icon param-collection %} *"FASTA sequences"*: `input sequences` (our original input dataset collection)
>    - *"Criteria for filtering on the headers"*: `List of IDs`
>        - {% icon param-file %} *"List of IDs to extract sequences for"*: output of **Remove beginning** {% icon tool %}
>        - *"Match IDs by"*: `Default: ID is expected at the beginning: >ID `
>
>
>    > <question-title></question-title>
>    >
>    > 1. What if my IDs don't match?
>    >
>    > > <solution-title></solution-title>
>    > > 1. If your FASTA file includes additional annotations in the headers, you may need to preprocess your ID list or modify the matching criteria.
>    > {: .solution}
>    >
>    {: .question}
>
> 2. **Examine** {% icon galaxy-eye %} the output files
>    - does everything look as expected?
>    - remember, we are expecting 29 sequences for Arabidopsis thaliana (ARATH) and 3 for Marchantia polymorpha (MARPO)
>    - it is always a good idea to check the outputs of an analysis step before continuing.
{: .hands_on}


# Evolutionary Analysis

We have now identified all sequences belonging to the Aux/IAA TAP family, across multiple species. Let's perform an evolutionary analysis on this data.


## Multiple Sequence Alignment (MSA)

First, we will perform a multiple sequence alignment in order to determine the similarity between all sequences in our TAP family.


> <hands-on-title> MSA of our protein sequences </hands-on-title>
>
> 1. {% tool [MAFFT](toolshed.g2.bx.psu.edu/repos/rnateam/mafft/rbc_mafft/7.526+galaxy1) %} with the following parameters:
>    - *"For multiple inputs generate"*: `a single MSA of all sequences from all inputs`
>      - In *"Input batch"*:
>        - {% icon param-repeat %} *"Insert Input batch"*
>          - {% icon param-collection %} *"Sequences to align"*: the collection of FASTA files from our TAP families (output of **Filter FASTA** {% icon tool %})
>    - *"Type of Sequences"*: Amino Acids
>    - *"Support unusual characters?"*: `Yes`
>    - *"MAFFT flavour"*: `Auto`
>    - *"Reorder output"*: `Yes`
>
>    > <question-title></question-title>
>    >
>    > 1. How many sequences do we have in total?
>    >
>    > > <solution-title></solution-title>
>    > > 1. 85 sequences. You can find this number by expanding the Galaxy dataset output from MAFFT.
>    > >    ![screenshot of MAFFT output expanded](./images/mafft-expanded.png)
>    > {: .solution}
>    >
>    {: .question}
>
> 2. **Examine** {% icon galaxy-eye %} the output file generated by MAFFT
>    -  you will see something similar to:
>       ```
>       >3702.ARATH_AT2G46990.1
>       -----PAVEDAEYVA---------------------------------------------
>       ---------------------------------------------------AVEEEE---
>       ------------------------------------------------------------
>       ---------------ENECNS--VGSFYVKVNMEGVPIGRKIDLMSLNGYRDLIRTLDFM
>       FNA-SILW-AE---------------------EE--------------------------
>       DMC---NEKSHVLTYADKEGDW--------------------------------------
>       ----MMVGDVPWEMFL-------------STVRRLKISRANYHY----------------
>       ------------------------------------------------------------
>       ---------------------------------------------------------
>       >3702.ARATH_AT3G62100.1
>       ----------------------MGRG-------------------------RSSSSSSIE
>       S-----------------------------------------------------------
>       -----DGVGAAEEMM---------------------------------------------
>       ---------------------------------------------------IMEEEE---
>       ------------------------------------------------------------
>       ---------------QNECNS--VGSFYVKVNMEGVPIGRKIDLLSLNGYHDLITTLDYM
>       FNA-SILW-AE---------------------EE--------------------------
>       DMC---SEKSHVLTYADKEGDW--------------------------------------
>       ----MMVGDVPWEMFL-------------SSVRRLKISRAYHY-----------------
>       ```
>
>       > <question-title> What's in an alignment? </question-title>
>       >
>       > 1. What do you see in this file?
>       > 2. What do the dashes signify?
>       >
>       > > <solution-title></solution-title>
>       > >
>       > > 1. This file contains a single alignment of all sequences. This format is similar to a FASTA file, but contains additional characters (e.g. dashes,`-`)
>       > > 2. Dashes indicate a gap in the alignment at that position for the sequence in question
>       > {: .solution}
>       >
>       {: .question}
>
{: .hands_on}

A nicer way to look at this output, is in the MSA viewer that is built into Galaxy.

> <hands-on-title> Visualise our Multiple Sequence Alignment </hands-on-title>
>
> 1. Click on the MAFFT output to **expand** it again in your history
> 2. Click on the {% icon galaxy-visualise %} **Visualize** button
> 3. In the main window, pick the "Multiple Sequence Alignment" option
>    ![button to click to select the MSA visualisation option]({% link topics/sequence-analysis/tutorials/tapscan-streptophyte-algae/images/visualize-msa-galaxy.png %})
> 4. You will see a visual representation of you MSA
>    ![screenshot of the multiple sequence alignment]({% link topics/sequence-analysis/tutorials/tapscan-streptophyte-algae/images/visualize-msa-screenshot.png %})
>
>    > <question-title> Alignment visualisation </question-title>
>    >
>    > 1. What do you see?
>    > 2. How long is the total alignment?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. You see every sequence in your dataset aligned together. On the left you see the sequence IDs. You can scroll down here to see more. On the top you see the position in the alignment. You can scroll left and right here to view different parts of the alignment.
>    > >
>    > >    Where amino acids align, you will see a colored square and the letter indicating the amino acid. Where there is a gap, you will see a dash (`-`). At the top, you will also see a histogram indicating how many sequences align at that position.
>    > >
>    > >    The bottom half of the screen shows a zoomed-out version of the entire alignment.
>    > >
>    > > 2. If you scroll all the way right in the aligment viewer, you will see a total length of around 1856 amino acids. This may not be the exact
>    > >    number, but gives you a pretty good idea.
>    > >
>    > >    ![screenshot of alignment when scrolled all the way to the right](./images/visualize-msa-galaxy-length.png)
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}


## Alignment trimming


Next, we would like to clean up this alignment by e.g. removing spurious sequences or poorly aligned regions from our multiple sequence alignment. {% icon tool %} **TrimAl** is a tool that can do this


> <hands-on-title> Clean up our alignment </hands-on-title>
>
> 1. {% tool [trimAl](toolshed.g2.bx.psu.edu/repos/iuc/trimal/trimal/1.5.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Alignment file"*: output of **MAFFT** {% icon tool %}
>    - *"Select trimming mode from the list"*: `custom mode`
>      - *"Gap threshold"*: 0.5
>      - *"Similarity Threshold"* 0.001
>
>    > <question-title></question-title>
>    > 1. How many sequences were removed in this trimming step?
>    >
>    > > <solution-title></solution-title>
>    > > 1. None. We still have all 85 sequences, meaning none of them were removed based on the gap threshold (0.5) or similarity threshold (0.001). However,
>    > >    TrimAl may have removed poorly aligned regions where gaps or similarity values did not meet the specified thresholds.
>    > {: .solution}
>    >
>    {: .question}
>
>
> 2. Examine {% icon galaxy-eye %} the **HTML report output from Trimal**
>    ![screenshot of TrimAl output]({% link topics/sequence-analysis/tutorials/tapscan-streptophyte-algae/images/trimal-screenshot.png %})
>
>    > <question-title></question-title>
>    > 1. Did trimAl remove any poorly aligned regions from our alignment?
>    > 2. How long was the total alignment before and after trimming?
>    >
>    > > <solution-title></solution-title>
>    > > 1. At the very top of the report, we can see the number of deleted and kept sequences and residues.
>    > >    As we already determined, all of the 85 sequences were kept, but 928 residues (columns in the alignment) were removed
>    > >
>    > >    ![screenshot of the table showing summary of trimal trimming](./images/trimal-screenshot-summary.png)
>    > > 2. We kept 929 residues and removed 928, so the total length before trimming was 929+928=1857.
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}



## Tree Generation

We can now analyse this multiple sequence alignment and determine the evolutionary tree.

> <hands-on-title> Tree Generationi with Quicktree </hands-on-title>
>
> 1. {% tool [Quicktree](toolshed.g2.bx.psu.edu/repos/iuc/quicktree/quicktree/2.5+galaxy0) %} with the following parameters:
>    - *"Provide an alignment file or a distance matrix?"*: `Alignment File`
>        - {% icon param-file %} *"Alignment file"*: `Trimmed alignment` (output of **trimAl** {% icon tool %})
>    - *"Calcuate bootstrap values with n iterations"*: `100`
>
> 2. **Examine** {% icon galaxy-eye %} the output from Quicktree
>    - You will see something like:
>      ```
>      (
>      (
>      (
>      4530.ORYSAJA_LOC_Os09g35870.1:0.24435,
>      (
>      (
>      4530.ORYSAJA_LOC_Os07g08460.1:0.19987,
>      4530.ORYSAJA_LOC_Os03g58350.1:0.19093)
>      94:0.03760,
>      (
>      4530.ORYSAJA_LOC_Os12g40900.1:0.15994,
>      4530.ORYSAJA_LOC_Os03g43410.1:0.18580)
>      97:0.04447)
>      67:0.02837)
>      19:0.00640,
>      ```
>    - This is *Newick* format, but not very informative to look at like this
>    - Newick trees are best viewed with special visualization tools, lets try one now.
>
> 3. Expand the output of Quicktree, and click on the {% icon galaxy-visualize %} visualise button
>    - Click on "Phylogenetic Tree Visualization"
>    - You should see a tree like
>      ![tree visualized with built-in tree viewer](./images/tree-viz-galaxy.png)
>
{: .hands_on}


> <details-title> Tree Generation Algorithms</details-title>
>
> There are different algorithms to create trees, which will produce different outputs.
> Quicktree uses the Neighbour-joining algorithm.
>
> Which algorithm is best for you, depends on your data and experiment.
>
> Another tool that gives you a bit more control over your tree generation is {% tool [IQ-TREE](toolshed.g2.bx.psu.edu/repos/iuc/iqtree/iqtree/2.3.6+galaxy0) %}
>
> If you would like to do that now, you can run IQ-tree as follows, which will generate both a Maximum-likelihood (ML) tree, as a
> Neighbour-joining (NJ) tree. These trees can be visualized with ETE-toolkit in the same way as you did for Quicktree
>
> 1. {% tool [IQ-TREE](toolshed.g2.bx.psu.edu/repos/iuc/iqtree/iqtree/2.3.6+galaxy0) %} with the following parameters:
>    - In *"General options"*:
>        - {% icon param-file %} *"Specify input alignment file in PHYLIP, FASTA, NEXUS, CLUSTAL or MSF format."*: `trimmed_output` (output of **trimAl** {% icon tool %})
>        - *"Specify sequence type"*: `AA`
>    - In *"Modelling Parameters"*:
>        - In *"Automatic model selection"*:
>            - *"Do you want to use a custom model"*: `Yes, I want to use a custom model`
>            - *"Model"*: `MFP`
>    - In *"Tree Parameters"*:
>        - In *"Single branch tests"*:
>            - *"Specify number of replicates (>=1000) to perform SH-like approximate likelihood ratio test"*: `1000`
>    - In *"Bootstrap Parameters"*:
>        - In *"Ultrafast bootstrap parameters"*:
>            - *"Specify number of bootstrap replicates"*: `1000`
>
{: .details}


## Tree Visualisation

We already had a look at the tree using Galaxy's built-in tree viewer, but sometimes it is nicer to look at with other tools.
For example, the [Phylogenetic Tree Viewer from ETE Toolkit](https://etetoolkit.org/treeview/). You could download your tree
to your machine, and upload it to ETE toolkit, but you can also do this directly within Galaxy!

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [ETE tree viewer](toolshed.g2.bx.psu.edu/repos/iuc/ete_treeviewer/ete_treeviewer/3.1.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Newick Tree to visualise"*: output of **Quicktree** {% icon tool %}
>    - *"Add alignment information to image?"*: `yes`
>        - {% icon param-file %} *"Multiple Alignment FASTA file"*: `Trimmed alignment` (output form **TrimAl** {% icon tool %} )
>    - *"Resolve Taxonomic IDs?"*: Yes
>    - *"Format of the output image."*: `PNG`
>
> 2. **Examine** {% icon galaxy-eye %} the output image
>    - You should see a tree image like this:
>
>      ![tree visualisation by ETE toolkit](./images/tree-viz-ete.png)
>
>    - As you can see, in addition to the tree itself, this also contains a visualisation of the alignments, and the taxonomy.
>
> > <question-title></question-title>
> > What do we see in this tree?
> >
> > > <solution-title></solution-title>
> > > We see that genes from the same group, such as land plants or bryophytes, cluster together, indicating their close evolutionary relationships and the potential for similar roles in auxin signaling.
> > > The branching patterns suggest that certain Aux/IAA genes from land plants and bryophytes are more closely related to each other, reflecting their shared functions in regulating growth and development under land conditions.
> > {: .solution}
> {: .question}
>
{: .hands_on}

> <details-title>Where did the taxonomy information come from?</details-title>
> Note that ETE toolkit will display this taxonomy based on your FASTA headers.
> If your sequence names start with the NCBI TaxID followed by a period, ETE
> TreeViewer will display them.
>
> For example, here are the headers in our file:
>
> ```
> >88036.SELMO_e_gw1.105.41.1
> -----------MNELLRLMPRDASIIGAIFVKLELLQL-V
> >88036.SELMO_fgenesh2_pg.C_scaffold_136000057
> GPS-KERRRKQMNELLRLMPRDASIIGAIFVKLELLQLSI
> ```
> where `88036` is the NCBI TaxID for Selaginella moellendortffii
>
{: .details}


# Conclusion

In this tutorial, utilizing the protein sequences of streptophyte algae, bryophytes, and land plants, we detected the TAP families and explored the distribution and evolutionary history of Aux/IAA proteins. Aux/IAA proteins play a critical role in the auxin signaling pathway, regulating plant growth and development, especially in response to environmental changes. Using [TAPScan Classify](https://github.com/Rensing-Lab/TAPscan-classify), we identified the presence of Aux/IAA proteins in these species and results showed lower number of Aux/IAA proteins in streptophyte algae compared to bryophytes and land plants. This suggests that the Aux/IAA protein family expanded as plants transitioned from aquatic to terrestrial environments, where more complex signaling mechanisms were required for growth regulation.

The phylogenetic tree analysis supported this finding, as genes from the same group, such as land plants or bryophytes, clustered together, indicating their close evolutionary relationships and the potential for similar roles in auxin signaling. The branching patterns suggest that certain Aux/IAA genes from land plants and bryophytes are more closely related to each other, reflecting their shared functions in regulating growth and development under land conditions.

tools_json:
["TAPScan Classify", "toolshed.g2.bx.psu.edu/repos/bgruening/tapscan/tapscan_classify/4.76+galaxy0", "Filter - data on any column using simple expressions", "Filter1", "Cut - columns from a table", "Cut1", "Remove beginning - of a file", "Remove beginning1", "Filter FASTA - on the headers and/or the sequences", "toolshed.g2.bx.psu.edu/repos/galaxyp/filter_by_fasta_ids/filter_by_fasta_ids/2.3", "MAFFT", "toolshed.g2.bx.psu.edu/repos/rnateam/mafft/rbc_mafft/7.526+galaxy1", "trimAl", "toolshed.g2.bx.psu.edu/repos/iuc/trimal/trimal/1.5.0+galaxy1", "Quicktree", "toolshed.g2.bx.psu.edu/repos/iuc/quicktree/quicktree/2.5+galaxy0", "IQ-TREE", "toolshed.g2.bx.psu.edu/repos/iuc/iqtree/iqtree/2.3.6+galaxy0", "ETE tree viewer", "toolshed.g2.bx.psu.edu/repos/iuc/ete_treeviewer/ete_treeviewer/3.1.3+galaxy0"]


topic: sequence-analysis
tutorial: quality-contamination-control
title: Quality and contamination control in bacterial isolate using Illumina MiSeq
difficulty: introductory
time_estimation: 2H
requirements_json: []
tags_json: ["illumina", "bacteria", "microgalaxy"]
content_hash: 900d00ea5dbe608247bdeabd48cb2b9f
last_modified: 2026-05-27T09:01:49.504938
zenodo_link: https://zenodo.org/record/10669812
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/sequence-analysis/tutorials/quality-contamination-control/tutorial.html

content:
Sequencing (determining of DNA/RNA nucleotide sequence) is used all over the world for all kinds of analysis. The product of these sequencers are reads, which are sequences of detected nucleotides. Depending on the technique these have specific lengths (30-500bp) or using Oxford Nanopore Technologies sequencing have much longer variable lengths.

{% snippet faqs/galaxy/sequencing_illumina_miseq.md %}

Contemporary sequencing technologies are capable of generating an enormous volume of sequence reads in a single run. Nonetheless, each technology has its imperfections, leading to various types and frequencies of errors, such as miscalled nucleotides. These inaccuracies stem from the inherent technical constraints of each sequencing platform. When sequencing bacterial isolates, it is crucial to verify the quality of the data but also to check the expected species or strains are found in the data or if there is any contamination.

To illustrate the process, we take raw data of a bacterial genome (KUN1163 sample) from sequencing data produced in "Complete Genome Sequences of Eight Methicillin-Resistant *Staphylococcus aureus* Strains Isolated from Patients in Japan" ({% cite Hikichi_2019 %}).

> Methicillin-resistant *Staphylococcus aureus* (MRSA) is a major pathogen
> causing nosocomial infections, and the clinical manifestations of MRSA
> range from asymptomatic colonization of the nasal mucosa to soft tissue
> infection to fulminant invasive disease. Here, we report the complete
> genome sequences of eight MRSA strains isolated from patients in Japan.
{: .quote cite="{% cite_url Hikichi_2019 %}"}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Galaxy and data preparation

Any analysis should get its own Galaxy history. So let's start by creating a new one and get the data (forward and reverse quality-controlled sequences) into it.

> <hands-on-title>Prepare Galaxy and data</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

Now, we need to import the data: 2 FASTQ files containing the reads from the sequencer.

> <hands-on-title>Import datasets</hands-on-title>
> 1. {% tool [Import](upload1) %} the files from [Zenodo]({{ page.zenodo_link }}) or from Galaxy shared data libraries:
>
>    ```
>    {{ page.zenodo_link }}/files/DRR187559_1.fastqsanger.bz2
>    {{ page.zenodo_link }}/files/DRR187559_2.fastqsanger.bz2
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
> 2. Rename the datasets to remove `.fastqsanger.bz2` and keep only the sequence run ID (`DRR187559_1` and `DRR187559_2`)
>
>    {% snippet faqs/galaxy/datasets_rename.md name="DRR187559_1" %}
>
> 3. Create a paired collection named `Paired Reads`
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
> 4. Tag the collection `#unfiltered`
>
>    {% snippet faqs/galaxy/collections_add_tag.md %}
>
> 5. **View** {% icon galaxy-eye %} the renamed files in the collection
>
{: .hands_on}

The datasets are both FASTQ files.

> <question-title></question-title>
>
> 1. What are the 4 main features of each read in a FASTQ file.
> 2. What does the `_1` and `_2` mean in your filenames?
>
> > <solution-title></solution-title>
> > 1. The following:
> >
> >    -   A `@` followed by a name and sometimes information of the read
> >    -   A nucleotide sequence
> >    -   A `+` (optional followed by the name)
> >    -   The quality score per base of nucleotide sequence (Each symbol
> >        represents a quality score, which will be explained later)
> >
> > 2. Forward and reverse reads, by convention, are labelled `_1` and `_2`, but they might also be `_f`/`_r` or `_r1`/`_r2`.
> {: .solution}
{: .question}

{% include _includes/cyoa-choices.html option1="Step-by-step" option2="Workflow" default="Step-by-step" text="Do you want to go step-by-step or using a workflow?" disambiguation="workflow"%}

<div class="Workflow" markdown="1">

In this section we will run a Galaxy workflow that performs the following tasks with the following tools:
1. Assess the reads quality before preprocessing it using [__Falco__](https://falco.readthedocs.io/en/latest/).
2. Trimming and filtering reads by length and quality using **Fastp** ({% cite Chen2018 %}).
3. Find witch microorgasnims are present using [__Kraken2__](https://ccb.jhu.edu/software/kraken2/) ({% cite Wood2014 %}).
4. Extract the species level with **Bracken** (Bayesian Reestimation of Abundance after Classification with Kraken) ({% cite Lu.2017 %}).
5. Detect minority organisms or contamination using **Recentrifuge** ({% cite marti2019recentrifuge %}).

We will run all these steps using a single workflow, then discuss each step and the results in more detail.

> <hands-on-title>Pre-Processing</hands-on-title>
>
> 1. **Import the workflow** into Galaxy
>    - Copy the URL (e.g. via right-click) of [this workflow]({{ site.baseurl }}{{ page.dir }}workflows/Quality-and-contamination-control-in-bacterial-isolate-using-Illumina-MiSeq-Data.ga) or download it to your computer.
>    - Import the workflow into Galaxy
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. Run **Workflow : Quality and contamination control in bacterial isolate using Illumina MiSeq Data** {% icon workflow %} using the following parameters
>    - *"Input Paired Reads"*: `Paired Reads`, which is the paired end reads collection.
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}

The workflow will take some time. Once completed, results will be available in your history. While waiting, read the next sections for details on each workflow step and the corresponding outputs.
</div>


# Read quality control and improvement

During sequencing, errors are introduced, such as incorrect nucleotides being called. These are due to the technical limitations of each sequencing platform. Sequencing errors might bias the analysis and can lead to a misinterpretation of the data. Adapters may also be present if the reads are longer than the fragments sequenced and trimming these may improve the number of reads mapped. **Sequence quality control is therefore an essential first step in any analysis.**

Assessing the quality by hand would be too much work. That's why tools like
[NanoPlot](https://github.com/wdecoster/NanoPlot) or
[Falco](https://falco.readthedocs.io/en/latest/) are made, as they  generate a summary and plots of the data statistics. NanoPlot is
mainly used for long-read data, like ONT and PACBIO and Falco for short read,
like Illumina and Sanger. Falco is an efficiency-optimized rewrite of [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/). You can read more in our dedicated [Quality Control Tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).

Before doing any analysis, the first questions we should ask about the input
reads include:

- What is the coverage of my genome?
- How good are my reads?
- Do I need to ask/perform for a new sequencing run?
- Is it suitable for the analysis I need to do?

## Quality control

<div class="Step-by-step" markdown="1">
> <hands-on-title>Quality Control</hands-on-title>
>
> 1. {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Raw read data from your current history"*: `Paired Reads`
>
> 2. Inspect the webpage outputs
>
{: .hands_on}
</div>

<div class="Workflow" markdown="1">

> <hands-on-title>Quality Control</hands-on-title>
> Inspect the webpage outputs of **Falco**
>
{: .hands_on}

</div>

**Falco** combines quality statistics from all separate reads and combines them in plots. An important plot is the Per base sequence quality.

DRR187559_1 | DRR187559_2
----------- | -----------
![Falco plot showing reads that stay in the green](../../../assembly/images/mrsa/falco-1.png) | ![Same as previous plot, but the beginning of the reads are slightly better quality](../../../assembly/images/mrsa/falco-2.png)

Here you have the reads sequence length on the x-axes against the quality score (Phred-score) on the y-axis. The colour of the beams indicates three quality levels:
- Green = good quality,
- Yellow = mediocre quality, and
- Red = bad quality.

For each position, a boxplot is drawn with:

- the median value, represented by the central intense coloured line
- the inter-quartile range (25-75%), represented by the green, yellow or red box
- the 10% and 90% values in the upper and lower whiskers

> <question-title></question-title>
>
> How does the median quality score change along the sequence?
>
> > <solution-title></solution-title>
> > The median quality score (intense coloured line) decreases at the sequences end. It is common for the median quality to drop towards the end of the sequences, as the sequencers are incorporating more incorrect nucleotides at the end. For Illumina data it is normal that the first few bases are of some lower quality and how longer the reads get the worse the quality becomes. This is often due to signal decay or phasing during the sequencing run.
> >
> {: .solution }
{: .question}

## Quality improvement

Depending on the analysis it could be possible that a certain quality or length
is needed. In this case we are going to trim the data using **fastp** ({% cite Chen2018 %}):

- Trim the start and end of the reads if those fall below a quality score of 20

  Different trimming tools have different algorithms for deciding when to cut but trimmomatic will cut based on the quality score of one base alone. Trimmomatic starts from each end, and as long as the base is below 20, it will be cut until it reaches one greater than 20. A sliding window trimming will be performed where if the average quality of 4 bases drops below 20, the read will be truncated there.

- Filter for reads to keep only reads with at least 30 bases: Anything shorter will complicate the assembly


<div class="Step-by-step" markdown="1">
> <hands-on-title>Quality improvement</hands-on-title>
>
> 1. {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/1.0.1+galaxy3) %} with the following parameters:
>    - *"Single-end or paired reads"*: `Paired Collection`
>        - *"Select paired collection(s)"*: `Paired Reads`
>    - In *"Filter Options"*:
>        - In *"Length filtering Options"*:
>            - *Length required*: `30`
>    - In *"Read Modification Options"*:
>        - In *"Per read cuitting by quality options"*:
>            - *Cut by quality in front (5')*: `Yes`
>            - *Cut by quality in tail (3')*: `Yes`
>            - *Cutting window size*: `4`
>            - *Cutting mean quality*: `20`
>    - In *"Output Options"*:
>        - *"Output JSON report"*: `Yes`
>
> 2. Edit the tags of the **fastp** FASTQ outputs to
>    1. Remove the `#unfiltered` tag
>    2. Add a new tag `#filtered`
{: .hands_on}
</div>

**fastp** generates also a report, similar to **Falco**, useful to compare the impact of the trimming and filtering.

> <question-title></question-title>
>
> Looking at **fastp** HTML report
>
> 1. How did the average read length change before and after filtering?
> 3. Did trimming improve the mean quality scores?
> 4. Did trimming affect the GC content?
> 5. Is this data ok to assemble? Do we need to re-sequence it?
>
> > <solution-title></solution-title>
> >
> > 1. Read lengths went down more significantly:
> >    - Before filtering: 190bp, 221bp
> >    - After filtering: 189bp, 219bp
> > 3. It increased the percentage of Q20 and Q30 bases (bases with quality score above 20 and 30 respectively)
> > 4. No, it did not. If it had, that would be unexpected.
> > 5. This data looks OK. The number of short reads in R1 is not optimal but assembly should partially work but not the entire, closed genome.
> >
> {: .solution}
{: .question}

# Identification of expected species and detection of contamination

When working with bacterial isolates, it is crucial to verify whether the expected species or strains are present in the data and to identify any potential contamination. Ensuring the presence of the intended species is essential for the accuracy and reliability of the research, as deviations could lead to erroneous conclusions. Additionally, detecting contamination is vital to maintain the integrity of the samples and to avoid misleading results that could compromise subsequent analyses and applications.

## Taxonomic profiling

To find out which microorganisms are present, we will compare the filtered reads of the sample to a reference database, i.e. sequences of known microorganisms stored in a database, using **Kraken2** ({% cite wood2019improved %}).

{% snippet topics/microbiome/faqs/kraken.md %}

For this tutorial, we will use the PlusPF database which contains the RefSeq Standard (archaea, bacteria, viral, plasmid, human, UniVec_Core), protozoa and fungi data.

<div class="Step-by-step" markdown="1">
> <hands-on-title> Assign taxonomic labels with Kraken2</hands-on-title>
>
> 1. {% tool [Kraken2](toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.1.3+galaxy2) %} with the following parameters:
>    - *"Single or paired reads"*: `Paired`
>        - *"Collection of paired reads"*: **fastp** `Paired-end output`
>    - *"Minimum Base Quality"*: `10`
>    - In *"Create Report"*:
>        - *"Print a report with aggregrate counts/clade to file"*: `Yes`
>    - *"Select a Kraken2 database"*: `PlusPF-16`
>
{: .hands_on}
</div>

**Kraken2** generates 2 outputs:

- **Classification**: tabular files with one line for each sequence classified by Kraken and 5 columns:

   1. `C`/`U`: a one letter indicating if the sequence classified or unclassified
   2. Sequence ID as in the input file
   3. NCBI taxonomy ID assigned to the sequence, or 0 if unclassified
   4. Length of sequence in bp (`read1|read2` for paired reads)
   5. A space-delimited list indicating the lowest common ancestor (LCA) mapping of each k-mer in the sequence

      For example, `562:13 561:4 A:31 0:1 562:3` would indicate that:
      1. The first 13 k-mers mapped to taxonomy ID #562
      2. The next 4 k-mers mapped to taxonomy ID #561
      3. The next 31 k-mers contained an ambiguous nucleotide
      4. The next k-mer was not in the database
      5. The last 3 k-mers mapped to taxonomy ID #562

      `|:|` indicates end of first read, start of second read for paired reads

   ```
   Column 1	Column 2	Column 3	Column 4	Column 5
   C	DRR187559.1	1280	164|85	0:1 1279:1 0:41 1279:10 0:5 1280:5 1279:1 0:1 1279:1 0:7 1279:5 0:2 1279:6 0:12 1279:5 0:19 1279:2 0:6 |:| 0:39 1279:2 0:6 1279:3 0:1
   C	DRR187559.2	1280	70|198	0:2 1279:5 0:29 |:| 0:52 1279:5 0:13 1279:3 0:23 1279:2 0:45 1280:1 0:9 1280:3 A:8
   C	DRR187559.3	1279	106|73	0:4 1279:3 0:36 1279:4 0:10 1279:5 0:3 1279:5 0:2 |:| 0:39
   C	DRR187559.4	1279	121|189	1279:6 0:17 1279:4 0:28 1279:2 0:30 |:| 0:7 1279:5 0:19 1279:5 0:20 1279:1 0:8 1279:1 0:1 1279:6 0:25 1279:2 0:44 A:11
   C	DRR187559.5	1279	68|150	1279:2 0:20 1279:3 0:9 |:| 0:10 1279:3 0:24 1279:2 0:9 1279:5 0:21 1279:5 0:20 1279:5 0:9 1279:1 0:2
   C	DRR187559.6	1280	137|246	0:2 1280:5 0:28 1279:1 0:28 1279:2 0:8 1279:2 0:23 1279:1 0:3 |:| 1279:1 0:2 1279:3 0:61 1279:2 0:14 1279:2 0:97 A:30
   ```

   > <question-title></question-title>
   >
   > 1. Is the first sequence in the file classified or unclassified?
   > 2. What is the taxonomy ID assigned to the first classified sequence?
   > 3. What is the corresponding taxon?
   >
   > > <solution-title></solution-title>
   > > 1. classified
   > > 2. 1280
   > > 3. 1280 corresponds to [Staphylococcus aureus](https://www.ncbi.nlm.nih.gov/datasets/taxonomy/tree/?taxon=1280) .
   > {: .solution}
   {: .question}

- **Report**: tabular files with one line per taxon and 6 columns or fields

   1. Percentage of fragments covered by the clade rooted at this taxon
   2. Number of fragments covered by the clade rooted at this taxon
   3. Number of fragments assigned directly to this taxon
   4. A rank code, indicating
      - (U)nclassified
      - (R)oot
      - (D)omain
      - (K)ingdom
      - (P)hylum
      - (C)lass
      - (O)rder
      - (F)amily
      - (G)enus, or
      - (S)pecies

      Taxa that are not at any of these 10 ranks have a rank code that is formed by using the rank code of the closest ancestor rank with a number indicating the distance from that rank. E.g., `G2` is a rank code indicating a taxon is between genus and species and the grandparent taxon is at the genus rank.

   5. NCBI taxonomic ID number
   6. Indented scientific name

   ```
   Column 1	Column 2	Column 3	Column 4	Column 5	Column 6
   0.24 	1065 	1065 	U 	0 	unclassified
   99.76 	450716 	14873 	R 	1 	root
   96.44 	435695 	2 	R1 	131567 	cellular organisms
   96.44 	435675 	3889 	D 	2 	Bacteria
   95.56 	431709 	78 	D1 	1783272 	Terrabacteria group
   95.53 	431578 	163 	P 	1239 	Firmicutes
   95.49 	431390 	4625 	C 	91061 	Bacilli
   94.38 	426383 	1436 	O 	1385 	Bacillales
   94.04 	424874 	2689 	F 	90964 	Staphylococcaceae
   93.44 	422124 	234829 	G 	1279 	Staphylococcus
   ```

   > <question-title></question-title>
   >
   > 1. How many taxa have been found?
   > 2. What are the percentage on unclassified?
   > 3. What are the domains found?
   >
   > > <solution-title></solution-title>
   > >
   > > 1. 627, as the number of lines
   > > 2. 0.24%
   > > 3. Only Bacteria
   > {: .solution}
   >
   {: .question}

## Species identification

In Kraken output, there are quite a lot of identified taxa with different levels. To obtain a more accurate and detailed understanding at the species level, we will use __Bracken__.

__Bracken__ refines the Kraken results by re-estimating the abundances of species in metagenomic samples, providing a more precise and reliable identification of species, which is crucial for downstream analysis and interpretation.

__Bracken__ (Bayesian Reestimation of Abundance after Classification with Kraken) is a "simple and worthwile addition to Kraken for better abundance estimates" ({% cite Ye.2019 %}). Instead of only using proportions of classified reads, it takes a probabilistic approach to generate final abundance profiles. It works by re-distributing reads in the taxonomic tree: "Reads assigned to nodes above the species level are distributed down to the species nodes, while reads assigned at the strain level are re-distributed upward to their parent species" ({% cite Lu.2017 %}).

<div class="Step-by-step" markdown="1">
> <hands-on-title>Extract species with Bracken</hands-on-title>
>
> 1. {% tool [Bracken](toolshed.g2.bx.psu.edu/repos/iuc/bracken/est_abundance/3.1+galaxy0) %} with the following parameters:
>     - {% icon param-collection %} *"Kraken report file"*: **Report** output of **Kraken**
>     - *"Select a kmer distribution"*: `PlusPF-16`, same as for Kraken
>
>        It is important to choose the same database that you also chose for Kraken2
>
>     - *"Level"*: `Species`
>     - *"Produce Kraken-Style Bracken report"*: `Yes`
>
{: .hands_on}
</div>

**Bracken** generates 2 outputs:

- **Kraken style report**: tabular files with one line per taxon and 6 columns or fields. Same configuration as the **Report** output of **Kraken**:

   1. Percentage of fragments covered by the clade rooted at this taxon
   2. Number of fragments covered by the clade rooted at this taxon
   3. Number of fragments assigned directly to this taxon
   4. A rank code, indicating
      - (U)nclassified
      - (R)oot
      - (D)omain
      - (K)ingdom
      - (P)hylum
      - (C)lass
      - (O)rder
      - (F)amily
      - (G)enus, or
      - (S)pecies
   5. NCBI taxonomic ID number
   6. Indented scientific name

   ```
   Column 1	Column 2	Column 3	Column 4	Column 5	Column 6
   100.00 	450408 	0 	R 	1 	root
   99.14 	446538 	0 	R1 	131567 	cellular organisms
   99.14 	446524 	0 	D 	2 	Bacteria
   99.14 	446524 	0 	D1 	1783272 	Terrabacteria group
   99.13 	446491 	0 	P 	1239 	Firmicutes
   99.13 	446491 	0 	C 	91061 	Bacilli
   99.06 	446152 	0 	O 	1385 	Bacillales
   99.06 	446152 	0 	F 	90964 	Staphylococcaceae
   99.04 	446101 	0 	G 	1279 	Staphylococcus
   95.62 	430661 	430661 	S 	1280 	Staphylococcus aureus
   ```

   > <question-title></question-title>
   >
   > 1. How many taxa have been found?
   > 2. What is the family found?
   >
   > > <solution-title></solution-title>
   > >
   > > 1. 119, as the number of lines
   > > 2. Staphylococcaceae, with 99.06%!
   > {: .solution}
   >
   {: .question}

- **Report**: tabular files with one line per taxon and 7 columns or fields

   1. Taxon name
   2. Taxonomy ID
   3. Level ID (S=Species, G=Genus, O=Order, F=Family, P=Phylum, K=Kingdom)
   4. Kraken assigned reads
   5. Added reads with abundance re-estimation
   6. Total reads after abundance re-estimation
   7. Fraction of total reads

   ```
   Column 1	Column 2	Column 3	Column 4	Column 5	Column 6	Column 7
   Staphylococcus aureus 	1280 	S 	182995 	247666 	430661 	0.95616
   Staphylococcus roterodami 	2699836 	S 	698 	48 	746 	0.00166
   Staphylococcus epidermidis 	1282 	S 	587 	13 	600 	0.00133
   Staphylococcus lugdunensis 	28035 	S 	511 	3 	514 	0.00114
   Staphylococcus schweitzeri 	1654388 	S 	409 	26 	435 	0.00097
   Staphylococcus simiae 	308354 	S 	398 	2 	400 	0.00089
   ```

> <question-title></question-title>
>
> 1. How many species have been found?
> 2. Which the species has been the most identified? And in which proportion?
> 3. What are the other species?
>
> > <solution-title></solution-title>
> > 1. 51 (52 lines including 1 line with header)
> > 2. *Staphylococcus aureus* with 95.6% of the reads.
> > 3. Most of the other species are from *Staphylococcus* genus, so same as *Staphylococcus aureus*. The other species in really low proportion.
> {: .solution}
{: .question}

As expected *Staphylococcus aureus* represents most of the reads in the data.

## Contamination detection

To explore **Kraken** report and specially to detect more reliably minority organisms or contamination, we will use **Recentrifuge** ({% cite marti2019recentrifuge %}).

**Recentrifuge** enhances analysis by reanalyzing metagenomic classifications with interactive charts that highlight confidence levels. It supports 48 taxonomic ranks of the [NCBI Taxonomy](https://www.ncbi.nlm.nih.gov/taxonomy), including strains, and generates plots for shared and exclusive taxa, facilitating robust comparative analysis.

**Recentrifuge** includes a novel contamination removal algorithm, useful when negative control samples are available, ensuring data integrity with control-subtracted plots. It also excels in detecting minority organisms in complex datasets, crucial for sensitive applications such as clinical and environmental studies.

<div class="Step-by-step" markdown="1">
> <hands-on-title> Identify contamination </hands-on-title>
>
> 1. {% tool [Recentrifuge](toolshed.g2.bx.psu.edu/repos/iuc/recentrifuge/recentrifuge/1.16.1+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Select taxonomy file tabular formated"*: **Classification** output of **Kraken2** {% icon tool %}
>    - *"Type of input file (Centrifuge, CLARK, Generic, Kraken, LMAT)"*: `Kraken`
>    - In *"Database type"*:
>        - *"Cached database whith taxa ID"*: `NCBI-2023-06-27`
>    - In *"Output options"*:
>        - *"Type of extra output to be generated (default on CSV)"*: `TSV`
>        - *"Remove the log file"*: `Yes`
>    - In *" Fine tuning of algorithm parameters"*:
>        - *"Strain level instead of species as the resolution limit for the robust contamination removal algorithm; use with caution, this is an experimental feature"*: `Yes`
>
{: .hands_on}
</div>

**Recentrifuge** generates 3 outputs:

- A **statistic table** with general statistics about assignations

   > <question-title></question-title>
   >
   > 1. How many sequences have been used?
   > 2. How many sequences have been classified?
   >
   > > <solution-title></solution-title>
   > > 1. 451,780
   > > 2. 450,715
   > {: .solution}
   {: .question}

- A **data table** with a report for each taxa

   > <question-title></question-title>
   >
   > 1. How many taxa have been kept?
   > 2. What is the lowest level in the data?
   >
   > > <solution-title></solution-title>
   > > 1. 187 (190 lines including 3 header lines)
   > > 2. The lowest level is strain.
   > {: .solution}
   {: .question}

- A **HTML report** with a Krona chart

   <iframe id="recentrifuge" src="recentrifuge_report.html" frameBorder="0" width="100%" height="900px"> ![Krona chart with multi-layered pie chart representing the community profile with in the center the higher taxonomy levels (i.e. domain) and on the exterior the more detailed ones (i.e. species)](./images/recentrifuge.png) </iframe>

   > <question-title></question-title>
   >
   > 1. What is the percentage of classified sequences?
   > 2. When clicking on *Staphylococcus aureus*, what can we say about the strains?
   > 3. Is there any contamination?
   >
   > > <solution-title></solution-title>
   > >
   > > 1. 99.8%
   > > 2. 99% of sequences assigned to *Staphylococcus aureus* are not assigned to any strains, probably because they are too similar to several strains. *Staphylococcus aureus* subsp. aureus JKD6159 is the strain with the most classified sequences, but only 0.3% of the sequences assigned to *Staphylococcus aureus*.
   > > 3. There is no sign of a possible contamination. Most sequences are classified to taxon on the *Staphylococcus aureus* taxonomy. Only 3% of the sequences are not classified to *Staphylococcus*.
   > >
   > {: .solution}
   >
   {: .question}

Once we have identified contamination, if any is present, the next step is to **remove the contaminated sequences** from the dataset to ensure the integrity of the remaining data. This can be done using bioinformatics tools designed to filter out unwanted sequences, such as **BBduk** ({% cite Bushnell2017 %}).

**BBduk** {% icon tool %} is a member of the [BBTools](https://sourceforge.net/projects/bbmap/) package, where 'Duk' stands for *Decontamination Using Kmers*. **BBduk** filters or trims reads for adapters and contaminants using k-mers, effectively removing unwanted sequences and improving the quality of the dataset.

Additionally, it's important to document and **report the contamination findings** to maintain transparency and guide any necessary adjustments in sample collection or processing protocols.

# Conclusion

In this tutorial, we inspected the quality of the bacterial isolate sequencing data and checked the expected species and potential contamination. Prepared short reads **can be used in downstream analysis**, like [Genome Assembly]({% link topics/assembly/tutorials/mrsa-illumina/tutorial.md %}). Even after the assembly, you can identify contamination using a tool like **CheckM** ({% cite Parks_2015 %}). **CheckM** {% icon tool %} can be used for screening the 'cleanness' of bacterial assemblies.

To learn more about data quality control, you can follow this tutorial: [Quality Control]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).

tools_json:
["Import", "upload1", "Falco", "toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0", "fastp", "toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/1.0.1+galaxy3", "Kraken2", "toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.1.3+galaxy2", "Bracken", "toolshed.g2.bx.psu.edu/repos/iuc/bracken/est_abundance/3.1+galaxy0", "Recentrifuge", "toolshed.g2.bx.psu.edu/repos/iuc/recentrifuge/recentrifuge/1.16.1+galaxy0"]


topic: sequence-analysis
tutorial: ncbi-fcs
title: Screening assembled genomes for contamination using NCBI FCS
difficulty: intermediate
time_estimation: 90M
tools_json: []
requirements_json: []
tags_json: []
content_hash: 25740905c3765abc21f1bab4f7b9e2b0
last_modified: 2026-05-15T22:28:14.617601
zenodo_link: https://zenodo.org/records/10932013
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/sequence-analysis/tutorials/ncbi-fcs/tutorial.html

content:
The National Center for Biotechnology Information (NCBI) performs contamination screening of genome assemblies submitted to the archival GenBank. Advances in genome sequencing have accelerated the production of genome assemblies and their submission to public databases, necessitating high-performance screening tools. Since contaminants can lead to misleading conclusions about the biology of the organism in question (e.g. gene content, gene evolution, etc.), ideally contamination screening should be performed after the initial contig assembly and prior to downstream genome analyses. 

NCBI has released a publicly-available Foreign Contamination Screen (FCS) tool suite to detect contaminants from various sources and produce a cleaned sequence set. This tutorial provides a quick example of two current FCS tools: FCS-adaptor identifies synthetic sequences used in library preparation, and FCS-GX ({% cite Astashyn2024 %}) identifies sequences from foreign organisms assigned to discordant taxonomies compared to the user-declared source organism.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Retrieving the data

FCS operates on assembled genome sequences and is not intended for use on raw reads. The following tutorial uses an assembled genome from yeast (*Saccharomyces cerevisiae*) with contaminants artificially inserted into the genome. The first step is to retrieve the genome FASTA. 

## Upload the genome FASTA from Zenodo

The following steps provide instructions to upload the test dataset into your Galaxy instance.
> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet  faqs/galaxy/histories_create_new.md %}
>
> 1. Copy the datasets URLs into clipboard.
>    Click on the copy button in the right upper corner of the box below.
>    ```
>    https://zenodo.org/records/10932013/files/FCS_combo_test.fa
>    ```
>
> 1. Upload `fasta.gz` dataset into Galaxy
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

## Confirm dataset upload

Your data should look like this:

![Galaxy History with FCS.combo.test.fa showing in green, successfully uploaded.](../../images/ncbi-fcs/fcs-galaxy-1.jpg "Galaxy History with Uploaded Data.")

# Importing Workflows

Next we will import a Galaxy workflow - a chain of tools to perform a set of operations on a user-supplied input. Specificially in this workflow, we will:

1. Screen the genome for foreign organism sequences using **FCS-GX** `screen` mode.
1. Produce a cleaned set of contigs using **FCS-GX** `clean` mode.
1. Screen the genome for synthetic sequences using **FCS-adaptor** and remove identified contaminants. 
1. Produce a final set of cleaned contigs using a second iteration of **FCS-GX** `clean` mode.

![The workflow pictured has five steps, input data is provided to FCS-GX screen, and both of these are inputs to FCS-GX clean. This output is fed to FCS-adapter screen, and both of these outputs are fed to FCS-adapter clean](../../images/ncbi-fcs/fcs-galaxy-2.jpg "NCBI Foreign Contamination Screen Galaxy Workflow.")

> <hands-on-title> Importing Galaxy Workflows </hands-on-title>
>
> 1. Ensure you are logged in to Galaxy
>
> 1. Import the workflow into Galaxy
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/sequence-analysis/tutorials/ncbi-fcs/workflows/NCBI-Foreign-Contamination-Screen.ga" title="NCBI Foreign Contamination Screen Workflow" %}
>
>    > <warning-title>Log in to Galaxy</warning-title>
>    > If the workflow failed to import, it is usually because you are not logged in.
>    {: .warning}
>
{: .hands_on}

Once the workflow is loaded, your workflow page should look like this:

![Galaxy Workflows page with NCBI Foreign Contamination Screen workflow loaded and ready to run.](../../images/ncbi-fcs/fcs-galaxy-3.jpg "Galaxy Workflows page with NCBI Foreign Contamination Screen.")

# Contamination screening

## Running the NCBI FCS Workflow

Next you will run the Galaxy workflow. Here you will configure the parameters for the FCS run according to the target organism.

> <hands-on-title> Running the NBCI FCS Workflow </hands-on-title>
> 1. Collect inputs
>    1. Assembled genome in `fasta/fasta.gz` format
>    1. Taxonomic information for source organism
>
> 1. Launch the Workflow
>    1. Click the Workflow menu located in the top menu bar
>    1. Click the {% icon workflow-run %} button located in the NCBI Foreign Contamination Screen Workflow box
>    1. Click "Expand to full workflow form." 
>    1. In the {% icon param-file %} 1: Input genome menu: 
>       - {% icon param-file %} *"Input file (Fasta file)"*: `1:FCS.combo.test.fa`
>    1. In the **2: FCS-GX screen** menu:
>       - Select the appropriate taxonomic division for **Taxonomy entry**: {% icon galaxy-wf-edit %} **div** (`fung:budding yeasts` in this example). 
>       - Set the **Advanced Options**: {% icon galaxy-wf-edit %} **Database location** to `/cvmfs/data.galaxyproject.org/byhand/ncbi_fcs_gx/all`
>    1. In the **4: FCS-adaptor screen** menu:
>       - Select the appropriate taxonomy for {% icon galaxy-wf-edit %} **Choose the taxonomy** (`Eukaryotes` in this example).
>    1. Run the workflow
>    1. Take a coffee break! 
{: .hands_on}

## Interpreting FCS Output

Running NCBI FCS on Galaxy is dependent on loading a large reference database on an adequate host. Currently the database is not persistent, meaning run times may vary. Most runs *should* complete in around one hour.

After the workflow is completed, you will be able to visualize tables of identified contaminants and can access FASTA files of clean sequences separated from contaminants.

> <hands-on-title> Reviewing FCS Contamination Reports </hands-on-title>
> 1. Look at the FCS-GX contamination report
>
>    1. In your Galaxy history, click the {% icon galaxy-eye %} icon for 3: **NCBI FCS GX on data 1: Action report**
>    1. Confirm that the taxonomic division you specified for the workflow run appears in the metadata in the first row of the file. It should appear as the "asserted-div" and in the list of "inferred-primary-divs".
>
>       > <warning-title> My organism's division is not in inferred-primary-divs </warning-title>
>       > If your target organism division is not in the set of inferred-primary-divs, it means that the assembly is likely heavily contaminated. FCS-GX will still call contaminants with respect to the taxonomy you define as the target organism (i.e. the primary division), but you should review the sequencing and assembly data for errors. 
>       {: .warning}
>
>    1. Review the contaminated sequence list, the suggested contamination cleanup actions, and the taxonomic divisions assigned to contaminants
>
>    > <question-title></question-title>
>    >
>    > 1. What are the major contaminants in this genome assembly?
>    > 2. When was this contamination likely introduced in the genome assembly process?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. *Homo sapiens* (human) and *Pseudomonas aeruginosa*. Note that for various reasons the actual contaminating genus/species isn't reported as the top_tax_name.
>    > > 2. Contamination was likely introduced at the sample/library preparation stage. *Pseudomonas aeruginosa* can be found in multiple human tissues, including skin flora.
>    > {: .solution}
>    {: .question}
>
> 1. Look at the FCS-GX cleaned sequences
>
>    1. Click the box in your workflow history - 1: **FCS.combo.test.fa**. Note the total number of sequences.
>    1. Click the box in your workflow history - 4: **NCBI FCS GX on data 3 and data 1: Fasta for EXCLUDE entries**. Note the total number of sequences.
>    1. Click the box in your workflow history - 5: **NCBI FCS GX on data 3 and data 1: Cleaned Fasta**. Note the total number of sequences as well as Applied actions.
>    1. Confirm that Cleaned Fasta sequences (Step 5) = Total sequences (Step 1) - Contaminant sequences (Step 4)
>
> 1. Look at the FCS-adaptor contamination report
>
>    1. In your Galaxy history, click the {% icon galaxy-eye %} icon for 6: **NCBI FCS Adaptor on data 5: Adaptor report**
>    1. Review the contaminated sequence list, the suggested contamination cleanup actions, and the identity of adaptor/vector contaminants. Adaptor/vector sequences can be found on the [NCBI UniVec FTP](https://ftp.ncbi.nlm.nih.gov/pub/UniVec/UniVec) page.
>  
>    > <question-title></question-title>
>    >
>    > 1. What are the adaptor/vector contaminants in this genome assembly?
>    > 2. When was this contamination likely introduced in the genome assembly process?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Illumina PCR Primer sequence.
>    > > 2. Contamination was likely introduced during contig assembly or polishing. Often the initial contig assembly is generated using long reads alone, so the introduction of contamination via polishing with untrimmed reads is the more likely explanation.
>    > {: .solution}
>    {: .question}
>
> 1. Look at the FCS-adaptor cleaned sequences
>
>    1. Click the box in your workflow history - 5: **NCBI FCS GX on data 3 and data 1: Cleaned Fasta**. Note the total number of sequences.
>    1. Click the box in your workflow history - 8: **NCBI FCS GX on data 6 and data 5: Fasta for EXCLUDE entries**. Note the total number of sequences as well as Applied actions.
>    1. Click the box in your workflow history - 9: **NCBI FCS GX on data 6 and data 5: Cleaned Fasta**. Note the total number of sequences.
>
> > <warning-title> Two different cleaned FASTA files from FCS-adaptor step </warning-title>
> > FCS-adaptor only removes sequences that are completely/mostly adaptor (assigned `ACTION_EXCLUDE`) and adaptors near contig ends (assigned `ACTION_TRIM`). FCS-adaptor **does not remove** small internal trims in large sequences. In this example, the FASTA file in workflow history - 7: **NCBI FCS Adaptor on data 5: Cleaned Fasta** is the same as the uncleaned input FASTA. FCS-GX `clean` is required to handle these internal spans. The FASTA file in workflow history - 9: **NCBI FCS GX on data 6 and data 5: Cleaned Fasta** has the contigs seq_1 through seq_16 split into two separate contigs at the `ACTION_TRIM` sites. If you want to hardmask these regions instead, you must download the tabular adaptor report, convert `ACTION_TRIM` values to `FIX`, and run the FCS-GX `clean` tool separately.
> {: .warning}
{: .hands_on}

# Conclusion

In this example, we removed 200 contaminant sequences from non-target organisms using FCS-GX and removed 16 internal adaptor sequences using FCS-adaptor, producing a cleaned yeast genome. Following contamination detection, we can perform separate validation checks to support the removal of these sequences from our assembly. One example is to perform BLAST searches or NUCmer alignments against reference genomes of the same or closely related species.


topic: sequence-analysis
tutorial: viral_primer_design
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: ["virology"]
content_hash: 8a47672b20190a953d7abf53c1d8cb21
last_modified: 2026-05-27T09:01:49.510180
zenodo_link: https://zenodo.org/records/14845698
workflows_json: []
title:
Primer and primer scheme design for pan-specific detection and sequencing of viral pathogens across genotypes

url:
https://training.galaxyproject.org/training-material/topics/sequence-analysis/tutorials/viral_primer_design/tutorial.html

content:
Modern molecular, DNA and RNA-based methods have seen ever increasing use, over the last decades, in the detection and analysis of viral pathogens. Most of these methods rely on PCR for amplification of viral DNA/RNA and are, thus, dependent on primers that recognize viral targets reliably and selectively. Designing such primers can be challenging if the diversity of viral genotypes that need to be recognized by the primers is high.

Quantitative real-time PCR (qPCR) is a very well established tool for detection of viral pathogens and for estimating viral load in samples. In its most widely used form in molecular diagnosis, qPCR requires two primers for amplification of a stretch of viral genetic material and an additional oligonucleotide probe that recognizes the amplified sequence and is itself used for fluorescence-based detection. Hence, in this case, three oligonucleotide sequences need to be designed that must recognize all viral genotypes of interest.

More recently, starting with the Zika virus epidemic from 2015 to 2016 and boosted by the 2020-2023 SARS-CoV-2 pandemic, tiled-amplicon sequencing has emerged as a powerful method for reconstructing full viral genomes directly from samples obtained from patients ({% cite Quick2017 %}). This method relies on multiplex PCRs to obtain amplification products (amplicons) that together cover (nearly) the complete genome of the viral pathogen. Overlap between amplicons (tiling) is required to achieve gap-free genome reconstruction and overlapping amplicons are generated in separate multiplex PCR reactions (pools) to avoid interaction of products during PCR. In this case each single amplicon is generated from two primers that, ideally, all need to be able to recognize all relevant viral genotypes and should, as far as possible not interact with other primers present in the same amplification pool. To illustrate the complexity involved, consider the example of SARS-CoV-2: [version 5.3.2](https://github.com/artic-network/primer-schemes/tree/master/nCoV-2019/V5.3.2) of the [ARTIC network](https://artic.network/)'s SARS-CoV-2 primer scheme defines 192 primers that generate 48 amplicons in each of two amplification pools and all these primers should bind efficiently to the genome sequences of all currently (first half of 2024) circulating omicron lineages (see the [v5.3.2 primer scheme announcement](https://community.artic.network/t/sars-cov-2-version-5-3-2-scheme-release/462) for the full details).

While SARS-CoV-2 is an extremely well-studied example, many other viral pathogens display a far greater sequence diversity than that of existing SARS-CoV-2 lineages making it even harder to come up with pan-specific primer sets for reliable amplification of arbitrary samples of unknown genotype. The three serotypes of poliovirus (PV-1, PV-2 and PV-3), for example, share only ~70% pairwise sequence identity with one another (compared to still ~ 99% for current SARS-CoV-2 lineages).

In this tutorial we investigate a robust bioinformatics approach that lets you design pan-specific PCR primers or entire primer schemes for viral pathogens with high sequence diversity. The design process starts from known genome sequences of viral isolates, which are supposed to represent the existing sequence diversity of the viral pathogen across its genotypes. From these sequences, we build a multiple sequence alignment, which then serves as input to the primer design tool **varVAMP**. This tool will try to identify possible primer sites that are conserved enough to work across genotypes while considering user-specified constraints like amplicon sizes for tiled-amplicon sequencing or desired qPCR probe characteristics.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data and Galaxy preparation

## Prepare a new Galaxy history
Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Prepare the Galaxy history</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md name="viral-primer-scheme-design tutorial" %}
>
{: .hands_on}

## Get viral reference sequences

> <hands-on-title>Get the Polio 1 sequence alignment</hands-on-title>
>
>    1. Import the Polio 1 sequence alignment from
>
>       ```
>       https://raw.githubusercontent.com/jonas-fuchs/ViralPrimerSchemes/79db0cc128079d770f2282b68a6e28142fd77473/input_alignments/polio1.aln
>       ```
>       and make sure the dataset format is set to `fasta`.
>
>       {% snippet faqs/galaxy/datasets_import_via_link.md format="fasta" %}
>    2. Rename the dataset
>
>       {% snippet faqs/galaxy/datasets_rename.md name="Polio 1 alignment" %}
>
{: .hands_on}

## Transform aligned sequences into unaligned FASTA format

For the primer design, we need a multiple sequence alignment (MSA), which we can generate with **MAFFT** ({% cite Katoh2013 %}). The data we just downloaded consists of *already* aligned sequences, but we want to follow here a complete primer design workflow.
That's why, as our very first step, we will undo the alignment to obtain just a simple collection of viral sequences, then use MAFFT to recreate the alignment. To undo the existing alignment we will use one tool from the **EMBOSS** tool suite, a powerful and versatile collection of little tools to manipulate and analyze sequence data.

> <hands-on-title>Degap the aligned Polio1 genome sequences</hands-on-title>
>
>    1. {% tool [degapseq](toolshed.g2.bx.psu.edu/repos/devteam/emboss_5/EMBOSS: degapseq20/5.0.0) %} with the following parameters:
>       - {% icon param-file %} *"On query"*: `Polio 1 alignment` (Input Polio 1 dataset)
>       - *"Output sequence file format"*: `FASTA (m)`
>
>    2. Rename the dataset
>
>       {% snippet faqs/galaxy/datasets_rename.md name="Polio 1 unaligned" %}
>
{: .hands_on}

# Designing a pan-specific primer/probe combination for qPCR

## Multiple alignment of reference sequences

Now lets (re)create the multiple sequence alignment. varVAMP, the tool that will then design primers for us, needs the MSA to identify regions that are conserved enough between all the different viral genomes to offer candidate pan-specific primer binding sites.
To generate an MSA, we align the previously unaligned genome datasets with **MAFFT**. MAFFT is a very popular program for multiple sequence alignment that gives the user the choice between many different alignment algorithms. For our purposes, we can, essentially, just use the default settings of the tool.

> <hands-on-title>Generate multiple sequence alignment</hands-on-title>
>
> 1. {% tool [MAFFT](toolshed.g2.bx.psu.edu/repos/rnateam/mafft/rbc_mafft/7.526+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Sequences to align"*: `Polio 1 unaligned` (the output of the **EMBOSS degapseq** tool)
>    - *"Type of sequences"*: `auto-detect` or `Nucleic acids`
>    - *"MAFFT flavour"*: `FFT-NS-2 (fast, progressive method)`
>    - *"Output format"*: `FASTA`
>
> 2. Rename the dataset
>
>    {% snippet faqs/galaxy/datasets_rename.md name="Polio 1 MSA" %}
>
{: .hands_on}

That was easy to set up and the tool run should not take very long with the `FFT-NS-2` alignment algorithm. MAFFT is a very powerful tool though and it may be worth considering not so simple usage scenarios. If you are mostly interested in using varVAMP, you can safely skip the following detail boxes, but if you want to learn more about using MAFFT, read on.

> <details-title> MAFFT - Multiple input datasets </details-title>
>
> With our Polio 1 data we had all the sequences that we wanted to align in a single multi-sequence FASTA dataset and could simply pass this one dataset to MAFFT.
> In many situations you will be downloading the starting sequences one by one and have them organized as a collection of single-sequence datasets. We can split out all our Polio 1 sequences into a collection to mimick this situation, then tweak the tool interface of MAFFT slightly to align them from that collection.
>
> > <hands-on-title>Create a MSA from a collection of input sequences</hands-on-title>
> > 1. {% tool [Split file to dataset collection](toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/split_file_to_collection/0.5.2) %} with the following parameters:
> >    - *"Select the file type to split"*: `FASTA`
> >    - {% icon param-file %} *"FASTA file to split"*: `Polio 1 unaligned` (the output of the **EMBOSS degapseq** tool)
> >
> >    This will create a separate dataset for each of our Polio 1 sequences and put them all into a collection.
> >
> > 2. {% tool [MAFFT](toolshed.g2.bx.psu.edu/repos/rnateam/mafft/rbc_mafft/7.526+galaxy1) %} with the following parameters:
> >    - *"For multiple inputs generate"*: `a single MSA of all sequences from all inputs`
> >      - In *"Input batch"*:
> >        - {% icon param-repeat %} *"1: Input batch"
> >          - {% icon param-collection %} *"Sequences to align"*: the collection of individual Polio 1 sequences (the output of the **Split file** tool run)
> >    - *"Type of sequences"*: `auto-detect` or `Nucleic acids`
> >    - *"MAFFT flavour"*: `FFT-NS-2 (fast, progressive method)`
> >    - *"Output format"*: `FASTA`
> >
> {: .hands_on}
>
> The above steps should result in the same MSA that you created from the multi-sequence FASTA dataset.
> The tool also supports more complex input scenarios like collections of *multi-sequence* datasets where you want to create a *collection* of MSAs, one per input collection element, for which you'd use the *"For multiple inputs generate"*: `one or several MSAs depending on input structure`, and others.
>
{: .details}

> <details-title> MSA: Multiple sequence alignment with MAFFT - Methods </details-title>
>
>   MAFFT is a program that offers different methods to align sequences.
>   These methods differ in terms of their speed and accuracy.
>
>   1. The progressive methods, including the default setting `FFT-NS-2`, can align large numbers of sequences in relatively short time. These methods are the fastest but also have somewhat reduced accuracy (especially the `FFT-NS-1` method, which only does a single round of guide tree and alignment construction). The default `FFT-NS-2` method (with two rounds of guide tree and alignment construction) is often a good compromise between speed and accuracy.
>
>   2. The iterative refinement methods, `FFT-NS-i` and `NW-NS-i`, first generate a progressive alignment like the fast methods above, but then improve its accuracy over several refinement rounds.
>
>   3. The slowest methods are those that also evaluate the consistency of the multiple sequence alignment with pairwise alignments of the sequences during the iterative refinement phase. The three implementations of this approach (`L-INS-i`, `E-INS-i` and `G-INS-i`) differ in the algorithm that they use to generate the pairwise sequence alignments (local alignment with affine gap costs, local alignment with generalized affine gap costs, or global alignment, respectively). These methods are the most accurate ones for difficult alignments with many gaps. However, they are also very slow, which means that, in practice, their use is restricted to alignments of small numbers of sequences (a few hundred maximum).
>
>   You can find more information about the individual methods in the [MAFFT manual](https://mafft.cbrc.jp/alignment/software/algorithms/algorithms.html).
>
{: .details}

Our polio 1 sequence alignment is ready for further use, but let us take a closer look at it before we proceed with the primer design. To explore multiple sequence alignments, Galaxy offers a convenient interactive visualization.

> <hands-on-title>Visualize the multiple sequence alignment</hands-on-title>
>
>    1. Click on the MAFFT output to expand this dataset
>    2. Click on {% icon galaxy-barchart %} **Visualize**
>    3. Maximize the browser window (the visualization we are about to launch may not rescale properly if you do this later!)
>    4. Select **Multiple Sequence Alignment** as the visualization
>    5. Wait for the alignment to finish loading
>    6. You can now navigate through the alignment with its horizontal and vertical scroll-bars, or by dragging the alignment itself around. Vertical navigation can also be done with the keyboard and you can go to any base position in the alignment directly via **Extras** -> **Jump to a column**.
>
{: .hands_on}

Take your time to familiarize yourself with the navigation options, then use them to compare your alignment to the view in the figure below.

![Galaxy-provided view into a small region of the aligned Polio 1 genomes. Blue buttons for configuring the visualization occupy the top row of the image. Below them are yellowish histogram bars the meaning of which will be explained in a questions/solutions box below. The lower part of the image has the view into the actual alignment with sequences color-coded by nucleotide and sequence IDs to the left of the actual sequences.](../../images/mafft_varvamp_training/aligned_Polio1_sequence128-142.png "Visualization of the aligned Polio 1 genomes. The focused view is of sequences 128-142 in the region 1346-1394 of the alignment.")

Finally, before we proceed with the actual  primer design, here are some questions regarding the multiple sequence alignment and its generation.

> <question-title></question-title>
>
> 1. How many sequences were aligned and over what length?
> 2. In the alignment visualization, what are those vertical bars in the top row above the base positions?
> 3. Does the alignment give reason to believe that pan-specific primers for detection of all polio 1 sequences can be found?
>
> > <solution-title></solution-title>
> >
> > 1. In total 241 sequences were aligned.
> >    You can expand the MSA dataset to get this number directly from Galaxy
> >    or you can scroll down to the last sequence in the MSA visualization.
> >
> >    The alignment has a total length of 7498 bases.
> >    You can scroll horizontally through the visualization, then read off the rightmost position to get this value.
> >
> >    If you want to know the length of the individual sequences instead of the alignment length (which includes gaps), you can, for example, run:
> >
> >    >  <hands-on-title>Get sequence lengths from a FASTA input</hands-on-title>
> >    >  1. {% tool [Compute sequence length](toolshed.g2.bx.psu.edu/repos/devteam/fasta_compute_length/fasta_compute_length/1.0.4) %}
> >    >     - *"Compute length for these sequences"*: `Polio 1 unaligned` (the output of the **EMBOSS degapseq** tool)
> >    >
> >    >     Note that you need to select the *unaligned* sequences as input to the tool in order to avoid having gaps counted towards the sequence lengths.
> >    >
> >    {: .hands_on}
> >
> >    The output of this extra step will be a tabular dataset with the names of sequences in the first and their lengths in the second column.
> >    This simple format could then be processed further with various other tools (like **Datamash** {% icon tool %}) to obtain some descriptive statistics of the lengths.
> > 2. The height of the bars indicates the sequence conservation at that site, with taller bars representing higher conservation.
> > 3. The alignment shows generally good conservation among the different polio 1 virus sequences and it looks like it should be possible to find at least *some* conserved primer sites.
> >
> {: .solution}
>
{: .question}

## One-step primer and probe design for qPCR

*Variable VirusAMPlicons* (*varVAMP*) is a tool to design primers for highly diverse viruses. The input required is an alignment of your viral (full-genome) sequences.

As a first step, we want to use the tool to design qPCR primer/probe schemes based on our polio 1 sequence alignment.

> <details-title>varVAMP modes</details-title>
>
> For many virus genera, it is challenging to design pan-specific primers. varVAMP addresses this issue by introducing ambiguous characters into primers and minimizing mismatches at the 3' end. While the primers may not work for all sequences in your input alignment, they should recognize the vast majority.
>
>   varVAMP offers three different modes:
>
>   1. **SINGLE:** varVAMP searches for the best primers and reports non-overlapping amplicons suitable for PCR-based screening approaches.
>   2. **TILED:** Using a graph-based approach, varVAMP designs overlapping (tiled) amplicons that together cover the entire (as much as possible) viral genome. These amplicons are ideal for Oxford Nanopore or Illumina-based full-genome sequencing.
>   3. **QPCR:** varVAMP searches for small amplicons with an optimized internal probe (TaqMan). It minimizes temperature differences between the primers and checks for secondary structures in the amplicons.
>
{: .details}

> <hands-on-title>Generate qPCR primer/probe set</hands-on-title>
>
> 1. {% tool [varVAMP](toolshed.g2.bx.psu.edu/repos/iuc/varvamp/varvamp/1.2.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Multiple alignment of viral sequences"*: `Polio 1 MSA` (output of **MAFFT** {% icon tool %})
>    - *"What kind of primers would you like to design? (varvamp mode)"*: `qPCR primers (qpcr)`
>        - *"How to set the main parameters, threshold for consensus nucleotides and max ambiguous nts per primer?"*: `Specify values for both`
>        - *"Threshold for consensus nucleotides"*: `0.93`
>        - *"Maximum number of ambiguous nucleotides per primer to be tolerated"*: `2`
>        - *"Maximum number of ambiguous nucleotides per qPCR probe to be tolerated"*: `1`
>
> 2. Move outputs to a dataset collection
>
>    Oh, that's a *lot* of outputs!
>
>    Since, in this tutorial, we are going to run varVAMP several times with different settings it would be easy to get lost in all the output datasets if we are not organizing things a bit.
>    So for the purpose of this tutorial, whenever a run of varVAMP has completed, we will move all of that run's output datasets into a *collection*.
>    Let's do so for the varVAMP outputs just generated:
>    1. Wait for all the varVAMP outputs to turn green
>    2. Move the outputs to a collection
>
>       {% snippet  faqs/galaxy/collections_build_list.md datasets_description="all outputs of the varVAMP run just finished" %}
{: .hands_on}

As you can see varVAMP offers a wide range of outputs. For example, it is possible to obtain the locations of the designed primers and amplicons in BED file format or an overview plot of all possible amplicons as PDF. These outputs provide detailed information about the regions of interest and other potential primers. The varVAMP *"Analysis Log"* file contains information about the tool's settings and procedures and is always included with the outputs. Take some time to review the different types of outputs and get familiar with the results.

> <comment-title>Output control</comment-title>
>
> Check your output files by comparing them with the example files from the [varVAMP-qPCR-output GitHub page for Polio 1 virus](https://github.com/jonas-fuchs/ViralPrimerSchemes/tree/main/varvamp_qpcr/polio1). There you can check if you created the same primers. The primer locations can be found in the BED file "primers.bed". These primers were designed using varVAMP version 0.7.
{: .comment}

Here a few additional questions for you to check your results and assess your understanding

> <question-title></question-title>
>
> 1. How long are the three different amplicons?
> 2. Why are there primers for three different amplicons and not just one?
> 3. Given the penalty scores of the different amplicons, which of them should be used for qPCR?
> 4. There are so many configuration options for varVAMP. How can you quickly check which exact settings were in use for a given run of varVAMP?
>
> > <solution-title></solution-title>
> >
> > 1. The first amplicon has a length of 93 bases, the second 142 bases and the third is 77 bases long, as you can see in the file *"qPCR amplicon details"*.
> > 2. varVAMP shows all amplicons it could find (well, up to 50 if you are not changing the *"Top n qPCR amplicons to test"* parameter) that fulfill the specified conditions like threshold or amplicon length. In our case three amplicons fulfill all conditions.
> > 3. The penalty score of each amplicon is simply the sum of the 3 penalty scores of its oligonucleotides. Lower values are better than higher ones for both the individual scores as well as for their sum. Regarding just the overall penalty score, scheme 0, starting at position 1743 (1-based position), has the best value with 6.7 and would be the prefered scheme to use. However, it is also useful to take a closer look at the file *"primer details"* to compare the number of ambiguous bases, individual penalty scores and melting temperatures per oligonucleotide, especially for the probe.
> >
> >    > <details-title>1-based vs 0-based reporting of positions?</details-title>
> >    >
> >    > varVAMP uses different formats to report nucleotide positions in its different outputs, which can be a bit confusing if you are not familiar with interval file formats.
> >    > The BED format specification requires 0-based positions (i.e. the first base in a sequence has position 0) and regions are specified with non-inclusive ends. In other words, a sequence region covering everything from the first base through the fifth would be represented as 0 (Start) - 5 (non-inclusive End) in BED format, while you would probably have written it as 1 - 5 in your regular notes.
> >    > As you can see from the above, you can
> >    > - calculate the length of a region in BED format simply by subtracting Start from End
> >    > - transform BED format to 1-based, end-inclusive form by simply adding 1 to the Start position
> >    >
> >    > varVAMP sticks to the format specification for all its BED outputs, but uses standard 1-based, end-inclusive format in all tabular reports.
> >    > That is why qPCR scheme 0 in our case is reported with 1742 (Start) - 1835 (End) in the *"Amplicon locations"* BED output, for example, but with 1743 (start) - 1835 (stop) in the tabular *"qPCR amplicon details"* output.
> >    {: .details}
> >
> > 4. The *"Analysis Log"* output lists all settings of all parameters that varVAMP used during that run.
> {: .solution}
>
{: .question}

## Advanced approach considering potential off-target binding sites of primers
Imagine that we want to design our qPCR so that it reliably detects Polio 1 virus, but *not* non-polio enteroviruses.

With varVAMP, it is possible to use a BLAST database as an off-target background sequence collection. Using the tool *NCBI BLAST+ makeblastdb* ({% cite Cock2015 %}), we can create a BLAST database containing the genome sequences of other enteroviruses. varVAMP can then match primers it identifies against this off-target database, and take predicted off-target binding sites into account when ranking amplicons. For this, varVAMP sorts potential primers first by presence of off-target effects, then by their penalty scores, which makes sure that amplicons containing off-target effect primers are always reported *after* any other amplicons.

Lets do this step-by-step.
First, you need to upload some enterovirus genome data into Galaxy.

> <hands-on-title>Get the genome sequences of non-polio enteroviruses</hands-on-title>
> 1. Import the sequences from [Zenodo](https://zenodo.org/records/14845698)
>
>    ```
>    https://zenodo.org/records/14845698/files/entero_genomes.fasta
>    ```
>    as a dataset of type `fasta`.
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="fasta" %}
>
> 2. Rename the dataset (optional)
>
>    {% snippet faqs/galaxy/datasets_rename.md name="Enterovirus genome sequences" %}
>
{: .hands_on}

Next, build the BLAST database from the uploaded sequences and use it with varVAMP.

> <hands-on-title>Build BLAST enterovirus database and (re-)design primers using it</hands-on-title>
>
> 1. {% tool [NCBI BLAST+ makeblastdb](toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_makeblastdb/2.14.1+galaxy2) %} with the following parameters:
>    - *"Molecule type of input"*: `nucleotide`
>        - {% icon param-repeat %} **Select Input**
>            - *"Input is a"*: `Dataset in history`
>            - *"FASTA input"*: `Enterovirus genome sequences`
>    - *"Title for BLAST database"*: `Enterovirus genome db`
>
> 2. Rename the dataset
>
>    {% snippet faqs/galaxy/datasets_rename.md name="Enterovirus genome db" %}
> 3. {% tool [varVAMP](toolshed.g2.bx.psu.edu/repos/iuc/varvamp/varvamp/1.2.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Multiple alignment of viral sequences"*: `Polio 1 MSA`
>    - *"What kind of primers would you like to design? (varvamp mode)"*: `qPCR primers (qpcr)`
>        - *"How to set the main parameters, threshold for consensus nucleotides and max ambiguous nts per primer?"*: `Specify values for both`
>           - *"Threshold for consensus nucleotides"*: `0.93`
>        - *"Avoid amplicons with off-target primer products?"*: `Yes`
>           - *"BLAST database"*: the output of **NCBI BLAST+ makeblastdb** `Enterovirus genome db`
>
> 4. Move outputs to a dataset collection
>
>    {% snippet  faqs/galaxy/collections_build_list.md datasets_description="all outputs of the varVAMP run just finished" %}
{: .hands_on}

A quick look at the *"Amplicon locations"* output shows that, again, varVAMP has found three suitable qPCR schemes. Now compare them to the ones from the previous run to see if anything changed.

> <tip-title>Comparing multiple datasets side-by-side</tip-title>
>    Tip: Galaxy's {% icon galaxy-scratchbook %} Window Manager, which you can enable (and disable again) from the menu bar can be very helpful for comparing multiple datasets.
>
{: .tip}

You can also inspect the *"qPCR amplicon details"* output to get more information about the suggested amplicons.

> <question-title></question-title>
>
> 1. Is there a difference between the primers/amplicons suggested in the first varVAMP run (without a database of enteroviral background sequences) and the new run?
> 2. Does the output contain amplicons with predicted off-target effects?
> 3. Why didn't varVAMP design primers without off-target binding sites?
> 4. What settings can be modified to generate more possible primers?
>
> > <solution-title></solution-title>
> >
> > 1. No, the primers generated without consideration of off-target sites are identical to those generated with such consideration.
> > 2. Yes, all amplicons are predicted to have off-target effects (see the *"qPCR amplicon details"* output).
> > 3. Before varVAMP can account for off-target sites, it must consider settings such as the threshold and the maximum number of ambiguous nucleotides. These settings resulted in only three amplicons, all of which show off-target effects.
> > 4. Our threshold for consensus sequences is set to 0.93, which is quite high. Lowering this threshold would likely generate more potential primer schemes. As an alternative we could increase the maximum number of ambiguous nucleotides allowed per primer.
> {: .solution}
>
{: .question}

Now let's experiment with a lower consensus nucleotide threshold to try and generate additional primer schemes and see if some of these maybe come without off-target amplification products. Lowering the threshold may lead to primer schemes with less power to amplify all posssible Polio 1 (*i.e.*, on-target) sequences, but values above 0.7 are typically still acceptable, depending on the specific use case. Since the default threshold value is 0.8, let's proceed with this setting.

> <hands-on-title>qPCR primer design with BLAST db and lower threshold</hands-on-title>
>
> 1. {% tool [varVAMP](toolshed.g2.bx.psu.edu/repos/iuc/varvamp/varvamp/1.2.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Multiple alignment of viral sequences"*: `Polio 1 MSA`
>    - *"What kind of primers would you like to design? (varvamp mode)"*: `qPCR primers (qpcr)`
>        - *"How to set the main parameters, threshold for consensus nucleotides and max ambiguous nts per primer?"*: `Specify values for both`
>           - *"Threshold for consensus nucleotides"*: `0.8`
>        - *"Avoid amplicons with off-target primer products?"*: `Yes`
>           - *"BLAST database"*: the output of **NCBI BLAST+ makeblastdb** `Enterovirus genome db`
>
> 2. Creating a dataset collection
>
>    {% snippet  faqs/galaxy/collections_build_list.md name="varVAMP Polio1 qpcr threshold 0.8 + BLAST" %}
{: .hands_on}

The newly designed primer schemes can be checked again with the files *"qPCR amplicon details"* and *"Primer details"*. We have prepared some questions for you to assess your understanding:

> <question-title></question-title>
>
> 1. What is the difference with a lower threshold? Did you get a primer scheme with no predicted off-target amplification?
> 2. Why doesn't the amplicon collection include the amplicons that we generated with the higher threshold?
> 3. Consider other use cases for avoiding off-targets using a BLAST database.
> 4. Can we find a better qPCR scheme if we allow varVAMP to estimate the consensus threshold?
>
> > <solution-title></solution-title>
> > 1. With the lower threshold, we generated more potential primer schemes, including four with no predicted off-target effects.
> > 2. The lower threshold leads to more potential amplicons that can achieve better scores than those with a higher threshold (but will potentially recognize a lower fraction of the genome sequences in the multiple-sequence alignment). The primers generated with the higher threshold are not considered in the new varVAMP run with the lower threshold because all of them had predicted off-target binding and relatively high scores.
> > 3. You may, for example, want to prepare primers that amplify specific viral genome sequences in the presence of host DNA, in which case you could build an off-target BLAST database from host genome sequences.
> > 4. We can run the following to let varVAMP estimate the optimal consensus threshold for us:
> >    > <hands-on-title>qPCR primer design with BLAST db and auto-estimated threshold</hands-on-title>
> >    >
> >    > 1. {% tool [varVAMP](toolshed.g2.bx.psu.edu/repos/iuc/varvamp/varvamp/1.2.2+galaxy0) %} with the following parameters:
> >    >    - {% icon param-file %} *"Multiple alignment of viral sequences"*: `Polio 1 MSA`
> >    >    - *"What kind of primers would you like to design? (varvamp mode)"*: `qPCR primers (qpcr)`
> >    >    - *"How to set the main parameters, threshold for consensus nucleotides and max ambiguous nts per primer?"*: `Specify max ambiguous nts, estimate suitable threshold`
> >    >    - *"Avoid amplicons with off-target primer products?"*: `Yes`
> >    >    - *"BLAST database"*: the output of **NCBI BLAST+ makeblastdb** `Enterovirus genome db`
> >    {: .hands-on}
> >
> >    When the tool run is finished you can inspect its *"Analysis Log"* output to learn that the estimated threshold is 0.92 so just 0.01 lower than our initial choice.
> >    If you then inspect the *"qPCR amplicon details"* output, you should find that this small difference enabled varVAMP to find 5 primer schemes instead of three, one of which has no predicted off-target effects.
> >
> >    Since the auto-estimated threshold of 0.92 is considerably higher than our second attempt of 0.8, the new qPCR scheme will likely recognize more of our polio 1 sequences and, thus, will likely be preferable over the ones found with the lower threshold.
> {: .solution}
>
{: .question}

# Designing a complete pan-specific primer scheme for tiled-amplicon sequencing

To explore additional varVAMP functionality, we will now design primers for another use case using the TILED flavour. This flavour is designed to cover the entire genome of our input data. Therefore, the resulting primers are suitable for Oxford Nanopore or Illumina-based full-genome sequencing.

> <comment-title> Online repositories of established primer schemes </comment-title>
>
> For many genomes, existing primer schemes are already available online, so you don't have to generate them by yourself. For example, you can explore various schemes from the [Quick lab](https://github.com/quick-lab/primerschemes/tree/main/primerschemes), [pha4ge](https://github.com/pha4ge/primer-schemes/tree/main/schemes), and [ARTIC-network](https://github.com/artic-network/primer-schemes) primer scheme repositories.
{: .comment}


To focus on designing the primer scheme, we will skip the multiple sequence alignment step this time and begin with the pre-aligned Polio 1-3 sequences available as part of the [ViralPrimerScheme](https://github.com/jonas-fuchs/ViralPrimerSchemes) repository.

> <hands-on-title>Get Polio 1-3 sequence alignment</hands-on-title>
>
>    1. Get the Polio 1-3 sequence alignment from
>
>       ```
>       https://raw.githubusercontent.com/jonas-fuchs/ViralPrimerSchemes/9b54b17246abdff1b39c6cdd362fff67eb2945c3/input_alignments/polio1-3.aln
>       ```
>       and upload it to your history via the link above and make sure the dataset format is set to `fasta`.
>
>       {% snippet faqs/galaxy/datasets_import_via_link.md format="fasta" %}
>    2. Rename the dataset
>
>       {% snippet faqs/galaxy/datasets_rename.md name="Polio 1-3 alignment" %}
>
{: .hands_on}

In this varVAMP run, we will use the TILED flavour. Similar to the qPCR primer design, you can determine the number of ambiguous nucleotides and set a threshold for consensus nucleotides. However, this time we will only specify the maximum number of ambiguous nucleotides and allow the threshold to be automatically estimated.

> <hands-on-title>Primer design with the TILED-flavour for whole genome amplification</hands-on-title>
>
> 1. {% tool [varVAMP](toolshed.g2.bx.psu.edu/repos/iuc/varvamp/varvamp/1.2.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Multiple alignment of viral sequences"*: the `Polio 1-3 alignment` just uploaded
>    - *"What kind of primers would you like to design? (varvamp mode)"*: `Tiled primer scheme for whole-genome sequencing (tiled)`
>        - *"How to set the main parameters, threshold for consensus nucleotides and max ambiguous nts per primer?"*: `Specify max ambiguous nts, estimate suitable threshold`
>        - *"Maximum number of ambiguous nucleotides per primer to be tolerated (default: 2)"*: `2`
>
> 2. Creating a dataset collection
>
>    {% snippet  faqs/galaxy/collections_build_list.md name="varVAMP Polio1-3 tiled" %}
>
>       > <comment-title>Dataset collection</comment-title>
>       >
>       >If you create a dataset collection at this point, you cannot include the "per-pool primer sequences", as this is already a dataset collection containing two files with the primer sequences for each pool.
>       {: .comment}
{: .hands_on}

Once again, we have prepared some questions for you to verify your results and deepen your understanding of the output:

> <question-title></question-title>
>
> 1. How many primers are used to cover the whole Polio genome?
> 2. What is encoded in the column named "score" in the *"Amplicon locations"* output.
> 3. How long are the amplicons for the tiled primer scheme and for qPCR and why do they differ in size?
> 4. What's the difference of the functionality of the qPCR and TILED modes?
> 5. If you want to tailor the generated tiling scheme to your experimental needs, which are the main options you would want to tweak?
>
> > <solution-title></solution-title>
> >
> > 1. varVAMP (at version 1.2.2) suggests 7 primer pairs to amplify nearly the whole Polio virus genome.
> >    The *"Amplicon design overview plot"* offers a nice view of the tiled scheme.
> > 2. The score indicates whether the primer pair for an amplicon belongs to pool 1 or 2.
> > 3. For the tiled primer scheme, the amplicons are between 1100 and 1500 bases long, while for qPCR, they covered around 100 bases.
> >    The difference in length exists because the goal of the tiled primer scheme is to cover the entire genome without using too many primers that might interact.
> >    In contrast, qPCR focuses on detecting the presence of matching DNA in a sample and the amplicons are kept short to enhance the amplification process.
> > 4. The qPCR mode generates small amplicons with an optimized internal probe, while the TILED mode produces overlapping amplicons designed for whole-genome sequencing.
> > 5. You could increase the length of the amplicons, their minimum required overlap, the consensus threshold, or reduce the maximum number of ambiguous nucleotides per primer.
> >
> {: .solution}
>
{: .question}

# Conclusion

The combination of the tools MAFFT and varVAMP offers a way to design primers and entire primer schemes that have a high chance of working across a set of diverse viral genotypes. You may still have to play with varVAMP's settings a bit to obtain optimal primers, or have to modify a resulting primer scheme a little by hand, and you might have to perform some wetlab-based optimizations (such as adjusting primer concentrations for certain amplicons), but this in-silico tool chain simplifies the task substantially.

tools_json:
["degapseq", "toolshed.g2.bx.psu.edu/repos/devteam/emboss_5/EMBOSS: degapseq20/5.0.0", "MAFFT", "toolshed.g2.bx.psu.edu/repos/rnateam/mafft/rbc_mafft/7.526+galaxy1", "Split file to dataset collection", "toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/split_file_to_collection/0.5.2", "Compute sequence length", "toolshed.g2.bx.psu.edu/repos/devteam/fasta_compute_length/fasta_compute_length/1.0.4", "varVAMP", "toolshed.g2.bx.psu.edu/repos/iuc/varvamp/varvamp/1.2.2+galaxy0", "NCBI BLAST+ makeblastdb", "toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_makeblastdb/2.14.1+galaxy2"]


topic: sequence-analysis
tutorial: ncbi-blast-against-the-madland
title: NCBI BLAST+ against the MAdLand
difficulty: intermediate
time_estimation: 15m
requirements_json: []
tags_json: ["plants"]
content_hash: 7f9c972d62c387a2f0d2d5436f23c848
last_modified: 2026-05-15T22:28:14.617096
zenodo_link: https://zenodo.org/records/7524427
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/sequence-analysis/tutorials/ncbi-blast-against-the-madland/tutorial.html

content:
MAdLandDB is a protein database comprising of a comprehensive collection of fully sequenced plant and algal genomes, with a particular emphasis on non-seed plants and streptophyte algae. Additionally, for comparative analysis, the database also includes genomes from various other organisms such as fungi, animals, the SAR group, bacteria, and archaea. The database is actively developed and maintained by the [Rensing lab](http://plantco.de) and released in the [MAdLand](https://madland.science/) setting. It employs a system of species abbreviation using a 5 letter code, which is constructed using the first three letters of the genus and the first two letters of the species name, for example, CHABR for *Chara braunii*. Furthermore, the database provides gene identification through the addition of gene ID’s and supplementary information such as the encoding source of the gene, whether it is plastome encoded (pt) or transcriptome-based (tr) in cases when a genome is not yet available. The key advantage of this database is its non-redundant nature, and the fact that all sequences are predominantly from genome projects, thereby increasing their reliability.



> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import the file `query.faa` from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    https://zenodo.org/records/7524427/files/query.faa
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>
{: .hands_on}

We just imported a FASTA file into Galaxy. Now, the next would be to perfrom the BLAST analysis against MAdLandDB.

## Perform NCBI Blast+ on Galaxy

Since MAdLandDB is the collection of protein sequences, You can perform {% tool [BLASTp](toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastp_wrapper/2.10.1+galaxy2) %} and {% tool [BLASTx](toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastx_wrapper/2.10.1+galaxy2) %} tools.

> <hands-on-title> Similarity search against MAdLand Database </hands-on-title>
>
> 1. {% tool [BLASTp](toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastp_wrapper/2.10.1+galaxy2) %} OR {% tool [BLASTx](toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastx_wrapper/2.10.1+galaxy2) %} with the following parameters:
>    - _"Protein query sequence(s)"_: `Amino acid input sequence` (In case of BLASTp) *OR*
>    - _"Translated nucleotide query sequence(s)"_: `Translated nucleotide input sequence` (In case of BLASTx)
>    - _"Subject database/sequences"_: `Locally installed BLAST database`
>    - _"Protein BLAST database"_: `MadLandDB (Genome zoo) plant and algal genomes with a focus on non-seed plants and streptophyte algae (22 Dec 2022)`
>    - _"Set expectation value cutoff"_: `0.001`
>    - _"Output format"_:
>    - In _"Output Options"_: `Tabular (extended 25 columns)`
> <img src="../../images/ncbi-blast-against-the-madland/blast-example.png" alt="blast against madland" width="80%">
>
{: .hands_on}

## Blast output

{% icon tool %} The BLAST output will be in tabular format (you can select the desired output format from the drop down menu) and include the following fields :

| Column | NCBI name | Description |
|-------|------------|-------------|
| 1 | qseqid | Query Seq-id (ID of your sequence) |
| 2 | sseqid | Subject Seq-id (ID of the database hit) |
| 3 | pident | Percentage of identical matches |
| 4 | length | Alignment length |
| 5 | mismatch | Number of mismatches |
| 6 | gapopen | Number of gap openings |
| 7 | qstart | Start of alignment in query |
| 8 | qend | End of alignment in query |
| 9 | sstart | Start of alignment in subject (database hit) |
| 10 | send | End of alignment in subject (database hit) |
| 11 | evalue | Expectation value (E-value) |
| 12 | bitscore | Bit score |

The fields are separated by tabs, and each row represents a single hit. For more details for BLAST analysis and output, we recommand you to follow the [Similarity-searches-blast]({% link topics/genome-annotation/tutorials/genome-annotation/tutorial.md %}#similarity-searches-blast) tutorial.


> <details-title>Further Reading about BLAST Tools in Galaxy</details-title>
>
> See {% cite Cock2015 %} and {% cite Cock2013 %}
>
{: .details}

## More Similarity Search Tools on Galaxy

* **Diamond**: {% tool [Diamond](toolshed.g2.bx.psu.edu/repos/bgruening/diamond/bg_diamond/2.0.15+galaxy0) %} is a high-throughput program for alignment of  large-scale data sets. It aligns sequences to the reference database using a compressed version of the reference sequences called a "database diamond" which is faster to read and can save computational time (~20,000 times the speed of Blastx, with high sensitivity).

> <details-title>Diamond in depth</details-title>
>
> See {% cite Buchfink2014 %} for more discussion.
{: .details}

tools_json:
["BLASTp", "toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastp_wrapper/2.10.1+galaxy2", "BLASTx", "toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastx_wrapper/2.10.1+galaxy2", "Diamond", "toolshed.g2.bx.psu.edu/repos/bgruening/diamond/bg_diamond/2.0.15+galaxy0"]


topic: sequence-analysis
tutorial: quality-control
title: Quality Control
difficulty: introductory
time_estimation: 1H30M
requirements_json: []
tags_json: ["deutsch", "espa\u00f1ol", "italiano"]
content_hash: 20dfbc9dbe3a12014544fbdf3f171a70
last_modified: 2026-05-27T09:01:49.506405
zenodo_link: https://zenodo.org/records/61771
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/sequence-analysis/tutorials/quality-control/tutorial.html

content:
During sequencing, the nucleotide bases in a DNA or RNA sample (library) are determined by the sequencer. For each fragment in the library, a sequence is generated, also called a **read**, which is simply a succession of nucleotides.

Modern sequencing technologies can generate a massive number of sequence reads in a single experiment. However, no sequencing technology is perfect, and each instrument will generate different types and amount of errors, such as incorrect nucleotides being called. These wrongly called bases are due to the technical limitations of each sequencing platform.

Therefore, it is necessary to understand, identify and exclude error-types that may impact the interpretation of downstream analysis.
Sequence quality control is therefore an essential first step in your analysis. Catching errors early saves time later on.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Inspect a raw sequence file

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import the file `female_oral2.fastq-4143.gz` from [Zenodo](https://zenodo.org/record/3977236) or from the data library (ask your instructor)
>    This is a microbiome sample from a snake {% cite StJacques2021 %}.
>
>    ```
>    https://zenodo.org/record/3977236/files/female_oral2.fastq-4143.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the imported dataset to `Reads`.
{: .hands_on}

We just imported a file into Galaxy. This file is similar to the data we could get directly from a sequencing facility: a [FASTQ file](https://en.wikipedia.org/wiki/FASTQ_format).

> <hands-on-title>Inspect the FASTQ file</hands-on-title>
>
> 1. Inspect the file by clicking on the {% icon galaxy-eye %} (eye) icon
>
{: .hands_on}

Although it looks complicated (and maybe it is), the FASTQ format is easy to understand with a little decoding.

Each read, representing a fragment of the library, is encoded by 4 lines:

Line  | Description
--- | ---
1 | Always begins with `@` followed by the information about the read
2 | The actual nucleic sequence
3 | Always begins with a `+` and contains sometimes the same info in line 1
4 | Has a string of characters which represent the quality scores associated with each base of the nucleic sequence; must have the same number of characters as line 2

So for example, the first sequence in our file is:

```
@M00970:337:000000000-BR5KF:1:1102:17745:1557 1:N:0:CGCAGAAC+ACAGAGTT
+
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGFGGGFGGGGGGAFFGGFGGGGGGGGFGGGGGGGGGGGGGGFGGG+38+35*311*6,,31=******441+++0+0++0+*1*2++2++0*+*2*02*/***1*+++0+0++38++00++++++++++0+0+2++*+*+*+*+*****+0**+0**+***+)*.***1**//*)***)/)*)))*)))*),)0(((-((((-.(4(,,))).,(())))))).)))))))-))-(
```

It means that the fragment named `@M00970` corresponds to the DNA sequence `GTGCCAGCCGCCGCGGTAGTCCGACGTGGCTGTCTCTTATACACATCTCCGAGCCCACGAGACCGAAGAACATCTCGTATGCCGTCTTCTGCTTGAAAAAAAAAAAAAAAAAAAACAAAAAAAAAAAAAGAAGCAAATGACGATTCAAGAAAGAAAAAAACACAGAATACTAACAATAAGTCATAAACATCATCAACATAAAAAAGGAAATACACTTACAACACATATCAATATCTAAAATAAATGATCAGCACACAACATGACGATTACCACACATGTGTACTACAAGTCAACTA` and this sequence has been sequenced with a quality `GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGFGGGFGGGGGGAFFGGFGGGGGGGGFGGGGGGGGGGGGGGFGGG+38+35*311*6,,31=******441+++0+0++0+*1*2++2++0*+*2*02*/***1*+++0+0++38++00++++++++++0+0+2++*+*+*+*+*****+0**+0**+***+)*.***1**//*)***)/)*)))*)))*),)0(((-((((-.(4(,,))).,(())))))).)))))))-))-(`.

{% snippet topics/sequence-analysis/faqs/quality_score.md %}

> <question-title></question-title>
>
> 1. Which ASCII character corresponds to the worst Phred score for Illumina 1.8+?
> 2. What is the Phred quality score of the 3rd nucleotide of the 1st sequence?
> 3. How to calculate the accuracy of the nucleotide base with the ASCII code `/`?
> 4. What is the accuracy of this 3rd nucleotide?
>
> > <solution-title></solution-title>
> > 1. The worst Phred score is the smallest one, so 0. For Illumina 1.8+, it corresponds to the `!` character.
> > 2. The 3rd nucleotide of the 1st sequence has a ASCII character `G`, which correspond to a score of 38.
> > 3. This can be calculated as follows:
> >    - ASCII code for `/` is 47
> >    - Quality score = 47-33=14
> >    - Formula to find the probability of error: \\(P = 10^{-Q/10}\\)
> >    - Probability of error = \\(10^{-14/10}\\) = 0.03981 = 3.981 %
> >    - Therefore Accuracy = 100 - 3.981 = 96.019%
> > 4. The corresponding nucleotide `G` has an accuracy of approximately 96%
> >
> {: .solution }
{: .question}

> <comment-title></comment-title>
> The current lllumina (1.8+) uses Sanger format (Phred+33). If you are working with older datasets you may encounter the older scoring schemes. **FastQC** {% icon tool %}, a tool we will use later in this tutorial, can be used to try to determine what type of quality encoding is used (through assessing the range of Phred values seen in the FASTQ).
{: .comment}

When looking at the file in Galaxy, it looks like most the nucleotides have a high score (`G` corresponding to a score 38). Is it true for all sequences? And along the full sequence length?


# Assess quality with FASTQE 🧬😎 - short reads only

To take a look at sequence quality along all sequences, we can use [FASTQE](https://fastqe.com/). It is an open-source tool that provides a simple and fun way to quality control raw sequence data and print them as emoji. You can use it to give a quick impression of whether your data has any problems of which you should be aware before doing any further analysis.

> <hands-on-title>Quality check</hands-on-title>
>
> 1. {% tool [FASTQE](toolshed.g2.bx.psu.edu/repos/iuc/fastqe/fastqe/0.3.1+galaxy0) %} with the following parameters
>    - {% icon param-files %} *"FastQ data"*: `Reads`
>    - {% icon param-select %} *"Score types to show"*: `Mean`
>
> 2. Inspect the generated HTML file
>
{: .hands_on}

Rather than looking at quality scores for each individual read, FASTQE looks at quality collectively across all reads within a sample and can calculate the mean for each nucleotide position along the length of the reads. Below shows the mean values for this dataset.

![FASTQE before](../../images/quality-control/fastqe-mean-before.png "FASTQE mean scores")

You can see the score for each [emoji in fastqe's documentation](https://github.com/fastqe/fastqe#scale). The emojis below, with Phred scores less than 20, are the ones we hope we don't see much.

Phred Quality Score | ASCII code | Emoji
--- | --- | ---
0 | ! | 🚫
1 | " | ❌
2 | # | 👺
3 | $ | 💔
4 | % | 🙅
5 | & | 👾
6 | ' | 👿
7 | ( | 💀
8 | ) | 👻
9 | * | 🙈
10 | + | 🙉
11 | , | 🙊
12 | - | 🐵
13 | . | 😿
14 | / | 😾
15 | 0 | 🙀
16 | 1 | 💣
17 | 2 | 🔥
18 | 3 | 😡
19 | 4 | 💩


> <question-title></question-title>
>
> What is the lowest mean score in this dataset?
>
> > <solution-title></solution-title>
> > The lowest score in this dataset is 😿 13.
> {: .solution }
{: .question}


# Assess quality with FastQC - short & long reads

An additional or alternative way we can check sequence quality is with [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/). It provides a modular set of analyses which you can use to check whether your data has any problems of which you should be aware before doing any further analysis.  We can use it, for example, to assess whether there are known adapters present in the data. We'll run it on the FASTQ file.

> <hands-on-title>Quality check</hands-on-title>
>
> 1. {% tool [FASTQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0) %} with the following parameters
>    - {% icon param-files %} *"Raw read data from your current history"*: `Reads`
>
> 2. Inspect the generated HTML file
>
{: .hands_on}

> <question-title></question-title>
>
> Which Phred encoding is used in the FASTQ file for these sequences?
>
> > <solution-title></solution-title>
> > The Phred scores are encoded using `Sanger / Illumina 1.9` (`Encoding` in the top table).
> {: .solution }
{: .question}

## Per base sequence quality

With FastQC we can use the per base sequence quality plot to check the base quality of the reads, similar to what we did with FASTQE.

![Per base sequence quality](../../images/quality-control/per_base_sequence_quality-before.png "Per base sequence quality")

On the x-axis are the base position in the read. In this example, the sample contains reads that are up to 296 bp long.

> <details-title>Non uniform x-axis</details-title>
>
> The x-axis is not always uniform. When you have long reads, some binning is applied to keep things compact. We can see that in our sample. It starts out with individual 1-10 bases. After that, bases are binned across a window a certain number of bases wide. Data binning means grouping and is a data pre-processing technique used to reduce the effects of minor observation errors. The number of base positions binned together depends on the length of the read. With reads >50bp, the latter part of the plot will report aggregate statistics for 5bp windows. Shorter reads will have smaller windows and longer reads larger windows. Binning can be removed when running FastQC by setting the paramter "Disable grouping of bases for reads >50bp" to Yes.
{: .details}

For each position, a boxplot is drawn with:

- the median value, represented by the central red line
- the inter-quartile range (25-75%), represented by the yellow box
- the 10% and 90% values in the upper and lower whiskers
- the mean quality, represented by the blue line

The y-axis shows the quality scores. The higher the score, the better the base call. The background of the graph divides the y-axis into very good quality scores (green), scores of reasonable quality (orange), and reads of poor quality (red).

It is normal with all Illumina sequencers for the median quality score to start out lower over the first 5-7 bases and to then rise. The quality of reads on most platforms will drop at the end of the read. This is often due to signal decay or phasing during the sequencing run. The recent developments in chemistry applied to sequencing has improved this somewhat, but reads are now longer than ever.


> <details-title>Signal decay and phasing</details-title>
>
> - Signal decay
>
>  The fluorescent signal intensity decays with each cycle of the sequencing process. Due to the degrading fluorophores, a proportion of the strands in the cluster are not being elongated. The proportion of the signal being emitted continues to decrease with each cycle, yielding to a decrease of quality scores at the 3' end of the read.
>
> - Phasing
>
>  The signal starts to blur with the increase of number of cycles because the cluster looses synchronicity. As the cycles progress, some strands get random failures of nucleotides to incorporate due to:
>
>  - Incomplete removal of the 3' terminators and fluorophores
>  - Incorporation of nucleotides without effective 3' terminators
>
>  This leads to a decrease in quality scores at the 3' end of the read.
{: .details}


> <details-title>Other sequence quality profiles</details-title>
>
> These are some per base sequence quality profiles that can indicate issues with the sequencing.
>
> - Overclustering
>
>   Sequencing facilities can overcluster the flow cells. It results in small distances between clusters and an overlap in the signals. Two clusters can be interpreted as a single cluster with mixed fluorescent signals being detected, decreasing signal purity. It generates lower quality scores across the entire read.
>
> - Instrumentation breakdown
>
>   Some issues can occasionally happen with the sequencing instruments during a run. Any sudden drop in quality or a large percentage of low quality reads across the read could indicate a problem at the facility. Some examples of such issues:
>
>    - Manifold burst
>
>      ![Manifold burst](../../images/quality-control/per_base_sequence_quality_manifold_burst.png)
>
>    - Cycles loss
>
>      ![Cycles loss](../../images/quality-control/per_base_sequence_quality_cycle_loss.png)
>
>    - Read 2 failure
>
>      ![Cycles loss](../../images/quality-control/per_base_sequence_quality_read2_failure.png)
>
>    With such data, the sequencing facility should be contacted for discussion. Often, a resequencing then is needed (and from our experience also offered by the company).
>
{: .details}

> <question-title></question-title>
>
> 1. How does the mean quality score change along the sequence?
> 2. Is this tendency seen in all sequences?
>
> > <solution-title></solution-title>
> > 1. The mean quality score (blue line) drops about midway though these sequences. It is common for the mean quality to drop towards the end of the sequences, as the sequencers are incorporating more incorrect nucleotides at the end. However, in this sample there is a very large drop in quality from the middle onwards.
> > 2. The box plots are getting wider from position ~100.  It means a lot of sequences have their score dropping from the middle of the sequence. After 100 nucleotides, more than 10% of the sequences have scores below 20.
> >
> {: .solution }
{: .question}

When the median quality is below a Phred score of ~20, we should consider trimming away bad quality bases from the sequence. We will explain that process in the Trim and filter section.

### Adapter Content

![Adapter Content](../../images/quality-control/adapter_content-before.png "Adapter Content")

The plot shows the cumulative percentage of reads with the different adapter sequences at each position. Once an adapter sequence is seen in a read it  is counted as being present right through to the end of the read so the percentage increases with the read length. FastQC can detect some adapters by default (e.g. Illumina, Nextera), for others we could provide a contaminants file as an input to the FastQC tool.

Ideally Illumina sequence data should not have any adapter sequence present. But with long reads, some of the library inserts are shorter than the read length resulting in read-through to the adapter at the 3' end of the read. This microbiome sample has relatively long reads and we can see Nextera dapater has been detected.

> <details-title>Other adapter content profiles</details-title>
>
> Adapter content may also be detected with RNA-Seq libraries where the distribution of library insert sizes is varied and likely to include some short inserts.
>
> ![Adapter Content](../../images/quality-control/adapter_content_rna_seq.png)
>
{: .details}

We can run an trimming tool such as Cutadapt to remove this adapter. We will explain that process in the filter and trim section.


> <tip-title>Take a shortcut</tip-title>
>
> The following sections go into detail about some of the other plots generated by FastQC. Note that some plots/modules may give warnings but be normal
> for the type of data you're working with, as discussed below and [in the FASTQC FAQ](https://rtsf.natsci.msu.edu/genomics/tech-notes/fastqc-tutorial-and-faq/).
> The other plots give us information to more deeply understand the quality of the data, and to see if changes could be made in the lab to get higher-quality data in the future.
> These sections are **optional**, and if you would like to skip these you can:
>   - Jump straight to the [next section](#trim-and-filter---short-reads) to learn about trimming paired-end data
{: .tip}

### Per tile sequence quality

This plot enables you to look at the quality scores from each tile across all of your bases to see if there was a loss in quality associated with only one part of the flowcell. The plot shows the deviation from the average quality for each flowcell tile. The hotter colours indicate that reads in the given tile have worse qualities for that position than reads in other tiles. With this sample, you can see that certain tiles show consistently poor quality, especially from ~100bp onwards. A good plot should be blue all over.

![Per tile sequence quality](../../images/quality-control/per_tile_sequence_quality-before.png "Per tile sequence quality")

This plot will only appear for Illumina library which retains its original sequence identifiers. Encoded in these is the flowcell tile from which each read came.

> <details-title>Other tile quality profiles</details-title>
>
> In some cases, the chemicals used during sequencing becoming a bit exhausted over the time and the last tiles got worst chemicals which makes the sequencing reactions a bit error-prone. The "Per tile sequence quality" graph will then have some horizontal lines like this:
>
> ![Per tile sequence quality with horizontal lines](../../images/quality-control/per_tile_sequence_quality_horizontal_lines.png)
>
{: .details}

## Per sequence quality scores

It plots the average quality score over the full length of all reads on the x-axis and gives the total number of reads with this score on the y-axis:

![Per sequence quality scores](../../images/quality-control/per_sequence_quality_scores-before.png "Per sequence quality scores")

The distribution of average read quality should be tight peak in the upper range of the plot. It can also report if a subset of the sequences have universally low quality values: it can happen because some sequences are poorly imaged (on the edge of the field of view etc), however these should represent only a small percentage of the total sequences.

## Per base sequence content

![Per base sequence content](../../images/quality-control/per_base_sequence_content-before.png "Per base sequence content for a DNA library")

"Per Base Sequence Content" plots the percentage of each of the four nucleotides (T, C, A, G) at each position across all reads in the input sequence file. As for the per base sequence quality, the x-axis is non-uniform.

In a random library we would expect that there would be little to no difference between the four bases. The proportion of each of the four bases should remain relatively constant over the length of the read with `%A=%T` and `%G=%C`, and the lines in this plot should run parallel with each other. This is amplicon data, where 16S DNA is PCR amplified and sequenced, so we'd expect this plot to have some bias and not show a random distribution.

> <details-title>Biases by library type</details-title>
>
> It's worth noting that some library types will always produce biased sequence composition, normally at the start of the read. Libraries produced by priming using random hexamers (including nearly all RNA-Seq libraries), and those which were fragmented using transposases, will contain an intrinsic bias in the positions at which reads start (the first 10-12 bases). This bias does not involve a specific sequence, but instead provides enrichment of a number of different K-mers at the 5' end of the reads. Whilst this is a true technical bias, it isn't something which can be corrected by trimming and in most cases doesn't seem to adversely affect the downstream analysis. It will, however, produce a warning or error in this module.
>
> ![Per base sequence content for RNA-seq data](../../images/quality-control/per_base_sequence_content_rnaseq.png)
>
> ChIP-seq data can also encounter read start sequence biases in this plot if fragmenting with transposases. With bisulphite converted data, e.g. HiC data, a separation of G from C and A from T is expected:
>
> ![Per base sequence content for Bisulphite data](../../images/quality-control/per_base_sequence_content_bisulphite.png)
>
> At the end, there is an overall shift in the sequence composition. If the shift correlates with a loss of sequencing quality, it can be suspected that miscalls are made with a more even sequence bias than bisulphite converted libraries.  Trimming the sequences fixed this problem, but if this hadn't been done it would have had a dramatic effect on the methylation calls which were made.
{: .details}

> <question-title></question-title>
>
> Why is there a warning for the per-base sequence content graphs?
>
> > <solution-title></solution-title>
> > In the beginning of sequences, the sequence content per base is not really good and the percentages are not equal, as expected for 16S amplicon data.
> >
> {: .solution }
{: .question}


## Per sequence GC content

![Per sequence GC content](../../images/quality-control/per_sequence_gc_content-before.png "Per sequence GC content")

This plot displays the number of reads vs. percentage of bases G and C per read. It is compared to a theoretical distribution assuming an uniform GC content for all reads, expected for whole genome shotgun sequencing, where the central peak corresponds to the overall GC content of the underlying genome. Since the GC content of the genome is not known, the modal GC content is calculated from the observed data and used to build a reference distribution.

An unusually-shaped distribution could indicate a contaminated library or some other kind of biased subset. A shifted normal distribution indicates some systematic bias, which is independent of base position. If there is a systematic bias which creates a shifted normal distribution then this won't be flagged as an error by the module since it doesn't know what your genome's GC content should be.

But there are also other situations in which an unusually-shaped distribution may occur. For example, with RNA sequencing there may be a greater or lesser distribution of mean GC content among transcripts causing the observed plot to be wider or narrower than an ideal normal distribution.

> <question-title></question-title>
>
> Why is there a fail for the per sequence GC content graphs?
>
> > <solution-title></solution-title>
> > There are multiple peaks. This can be indicative of unexpected contamination, such as adapter, rRNA or overrepresented sequences. Or it may be normal if it is amplicon data or you have highly abundant RNA-seq transcripts.
> {: .solution }
{: .question}

### Sequence length distribution

This plot shows the distribution of fragment sizes in the file which was analysed. In many cases this will produce a simple plot showing a peak only at one size, but for variable length FASTQ files this will show the relative amounts of each different size of sequence fragment. Our plot shows variable length as we trimmed the data. The biggest peak is at 296bp but there is a second large peak at ~100bp. So even though our sequences range up to 296bp in length, a lot of the good-quality sequences are shorter. This corresponds with the drop we saw in the sequence quality at ~100bp and the red stripes starting at this position in the per tile sequence quality plot.

![Sequence length distribution](../../images/quality-control/sequence_length_distribution-before.png "Sequence length distribution")

Some high-throughput sequencers generate sequence fragments of uniform length, but others can contain reads of widely varying lengths. Even within uniform length libraries some pipelines will trim sequences to remove poor quality base calls from the end or the first $$n$$ bases if they match the first $$n$$ bases of the adapter up to 90% (by default), with sometimes $$n = 1$$.

## Sequence Duplication Levels

The graph shows in blue the percentage of reads of a given sequence in the file which are present a given number of times in the file:

![Sequence Duplication Levels](../../images/quality-control/sequence_duplication_levels-before.png "Sequence Duplication Levels")

In a diverse library most sequences will occur only once in the final set. A low level of duplication may indicate a very high level of coverage of the target sequence, but a high level of duplication is more likely to indicate some kind of enrichment bias.

Two sources of duplicate reads can be found:
- PCR duplication in which library fragments have been over-represented due to biased PCR enrichment

  It is a concern because PCR duplicates misrepresent the true proportion of sequences in the input.

- Truly over-represented sequences such as very abundant transcripts in an RNA-Seq library or in amplicon data (like this sample)

  It is an expected case and not of concern because it does faithfully represent the input.

> <details-title>More details about duplication</details-title>
>
> FastQC counts the degree of duplication for every sequence in a library and creates a plot showing the relative number of sequences with different degrees of duplication. There are two lines on the plot:
> - Blue line: distribution of the duplication levels for the full sequence set
> - Red line: distribution for the de-duplicated sequences with the proportions of the deduplicated set which come from different duplication levels in the original data.
>
> For whole genome shotgun data it is expected that nearly 100% of your reads will be unique (appearing only 1 time in the sequence data). Most sequences should fall into the far left of the plot in both the red and blue lines. This indicates a highly diverse library that was not over sequenced. If the sequencing depth is extremely high (e.g. > 100x the size of the genome) some inevitable sequence duplication can appear: there are in theory only a finite number of completely unique sequence reads which can be obtained from any given input DNA sample.
>
> More specific enrichments of subsets, or the presence of low complexity contaminants will tend to produce spikes towards the right of the plot. These high duplication peaks will most often appear in the blue trace as they make up a high proportion of the original library, but usually disappear in the red trace as they make up an insignificant proportion of the deduplicated set. If peaks persist in the red trace then this suggests that there are a large number of different highly duplicated sequences which might indicate either a contaminant set or a very severe technical duplication.
>
> It is usually the case for RNA sequencing where there is some very highly abundant transcripts and some lowly abundant. It is expected that duplicate reads will be observed for high abundance transcripts:
>
> ![Sequence Duplication Levels for RNA-seq](../../images/quality-control/sequence_duplication_levels_rna_seq.png)
>
{: .details}

## Over-represented sequences

A normal high-throughput library will contain a diverse set of sequences, with no individual sequence making up a tiny fraction of the whole. Finding that a single sequence is very over-represented in the set either means that it is highly biologically significant, or indicates that the library is contaminated, or not as diverse as expected.

FastQC lists all of the sequence which make up more than 0.1% of the total. For each over-represented sequence FastQC will look for matches in a database of common contaminants and will report the best hit it finds. Hits must be at least 20bp in length and have no more than 1 mismatch. Finding a hit doesn't necessarily mean that this is the source of the contamination, but may point you in the right direction. It's also worth pointing out that many adapter sequences are very similar to each other so you may get a hit reported which isn't technically correct, but which has a very similar sequence to the actual match.

RNA sequencing data may have some transcripts that are so abundant that they register as over-represented sequence. With DNA sequencing data no single sequence should be present at a high enough frequency to be listed, but we can sometimes see a small percentage of adapter reads.

> <question-title></question-title>
>
> How could we find out what the overrepreseented sequences are?
>
> > <solution-title></solution-title>
> > We can BLAST overrepresented sequences to see what they are. In this case, if we take the top overrepresented sequence
> > ```
> > >overrep_seq1
> > GTGTCAGCCGCCGCGGTAGTCCGACGTGGCTGTCTCTTATACACATCTCC
> > ```
> > and use [blastn](https://blast.ncbi.nlm.nih.gov/Blast.cgi) against the default Nucleotide (nr/nt) database we don't get any hits. But if we use [VecScreen](https://www.ncbi.nlm.nih.gov/tools/vecscreen/) we see it is the Nextera adapter.
> > ![VecScreen](../../images/quality-control/vecscreen-nextera.png "Nextera adapter")
> {: .solution }
{: .question}


> <details-title>More details about other FastQC plots</details-title>
>
>
> #### Per base N content
>
> ![Per base N content](../../images/quality-control/per_base_n_content-before.png "Per base N content")
>
> If a sequencer is unable to make a base call with sufficient confidence, it will write an "N" instead of a conventional base call. This plot displays the percentage of base calls at each position or bin for which an N was called.
>
> It's not unusual to see a very high proportion of Ns appearing in a sequence, especially near the end of a sequence. But this curve should never rises noticeably above zero. If it does this indicates a problem occurred during the sequencing run. In the example below, an error caused the instrument to be unable to call a base for approximately 20% of the reads at position 29:
>
> ![Per base N content](../../images/quality-control/per_base_n_content_error.png)
>
>
> #### Kmer Content
>
> This plot not output by default. As stated in the tool form, if you want this module it needs to be enabled using a custom Submodule and limits file. With this module, FastQC does a generic analysis of all of the short nucleotide sequences of length k (kmer, with k = 7 by default) starting at each position along the read in the library to find those which do not have an even coverage through the length of your reads. Any given kmer should be evenly represented across the length of the read.
>
> FastQC will report the list of kmers which appear at specific positions with a greater frequency than expected. This can be due to different sources of bias in the library, including the presence of read-through adapter sequences building up on the end of the sequences. The presence of any overrepresented sequences in the library (such as adapter dimers) causes the kmer plot to be dominated by the kmer from these sequences. Any biased kmer due to other interesting biases may be then diluted and not easy to see.
>
> The following example is from a high-quality DNA-Seq library. The biased kmers nearby the start of the read likely are due to slight sequence dependent efficiency of DNA shearing or a result of random priming:
>
> ![Kmer Content](../../images/quality-control/kmer_content.png "Kmer content")
>
> This module can be very difficult to interpret. The adapter content plot and overrepesented sequences table are easier to interpret and may give you enough information without needing this plot. RNA-seq libraries may have highly represented kmers that are derived from highly expressed sequences. To learn more about this plot, please check the [FastQC Kmer Content documentation](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/3%20Analysis%20Modules/11%20Kmer%20Content.html).
>
{: .details}

We tried to explain here there different FastQC reports and some use cases. More about this and also some common next-generation sequencing problems can be found on [QCFAIL.com](https://sequencing.qcfail.com/)

> <details-title>Specific problem for alternate library types</details-title>
>
> #### Small/micro RNA
>
> In small RNA libraries, we typically have a relatively small set of unique, short sequences. Small RNA libraries are not randomly sheared before adding sequencing adapters to their ends: all the reads for specific classes of microRNAs will be identical. It will result in:
>
> - Extremely biased per base sequence content
> - Extremely narrow distribution of GC content
> - Very high sequence duplication levels
> - Abundance of overrepresented sequences
> - Read-through into adapters
>
> #### Amplicon
>
> Amplicon libraries are prepared by PCR amplification of a specific target. For example, the V4 hypervariable region of the bacterial 16S rRNA gene. All reads from this type of library are expected to be nearly identical. It will result in:
>
> - Extremely biased per base sequence content
> - Extremely narrow distribution of GC content
> - Very high sequence duplication levels
> - Abundance of overrepresented sequences
>
> #### Bisulfite or Methylation sequencing
>
> With Bisulfite or methylation sequencing, the majority of the cytosine (C) bases are converted to thymine (T). It will result in:
>
> - Biased per base sequence content
> - Biased per sequence GC content
>
> #### Adapter dimer contamination
>
> Any library type may contain a very small percentage of adapter dimer (i.e. no insert) fragments. They are more likely to be found in amplicon libraries constructed entirely by PCR (by formation of PCR primer-dimers) than in DNA-Seq or RNA-Seq libraries constructed by adapter ligation. If a sufficient fraction of the library is adapter dimer it will become noticeable in the FastQC report:
>
> - Drop in per base sequence quality after base 60
> - Possible bi-modal distribution of per sequence quality scores
> - Distinct pattern observed in per bases sequence content up to base 60
> - Spike in per sequence GC content
> - Overrepresented sequence matching adapter
> - Adapter content > 0% starting at base 1
>
{: .details}

> <comment-title>Bad quality sequences</comment-title>
> If the quality of the reads is not good, we should always first check what is wrong and think about it: it may come from the type of sequencing or what we sequenced (high quantity of overrepresented sequences in transcriptomics data, biased percentage of bases in HiC data).
>
> You can also ask the sequencing facility about it, especially if the quality is really bad: the quality treatments can not solve everything. If too many bad quality bases are cut away, the corresponding reads then will be filtered out and you lose them.
{: .comment}

# Trim and filter - short reads

The quality drops in the middle of these sequences. This could cause bias in downstream analyses with these potentially incorrectly called nucleotides. Sequences must be treated to reduce bias in downstream analysis. Trimming can help to increase the number of reads the aligner or assembler are able to succesfully use, reducing the number of reads that are unmapped or unassembled. In general, quality treatments include:

1. Trimming/cutting/masking sequences
    - from low quality score regions
    - beginning/end of sequence
    - removing adapters
2. Filtering of sequences
    - with low mean quality score
    - too short
    - with too many ambiguous (N) bases

To accomplish this task we will use [Cutadapt](https://cutadapt.readthedocs.io/en/stable/guide.html) {% cite marcel2011cutadapt %}, a tool that enhances sequence quality by automating adapter trimming as well as quality control.  We will:

- Trim low-quality bases from the ends. Quality trimming is done before any adapter trimming. We will set the quality threshold as 20, a commonly used threshold, see more [in GATK's Phred Score FAQ](https://gatk.broadinstitute.org/hc/en-us/articles/360035531872-Phred-scaled-quality-scores).
- Trim adapter with Cutadapt. For that we need to supply the sequence of the adapter. In this sample, Nextera is the adapter that was detected. We can find the sequence of the Nextera adapter on the [Illumina website here](https://support.illumina.com/bulletins/2016/12/what-sequences-do-i-use-for-adapter-trimming.html) `CTGTCTCTTATACACATCT`. We will trim that sequence from the 3' end of the reads.
- Filter out sequences with length < 20 after trimming

> <hands-on-title>Improvement of sequence quality</hands-on-title>
>
> 1. {% tool [Cutadapt](toolshed.g2.bx.psu.edu/repos/lparsons/cutadapt/cutadapt/4.9+galaxy1) %} with the following parameters
>    - *"Single-end or Paired-end reads?"*: `Single-end`
>       - {% icon param-file %} *"FASTQ/A file"*: `Reads` (Input dataset)
>
>          > <tip-title>Files not selectable?</tip-title>
>          > If your FASTQ file cannot be selected, you might check whether the format is FASTQ with Sanger-scaled quality values (`fastqsanger.gz`). You can edit the data type by clicking on the pencil symbol.
>          {: .tip}
>    - In *"Read 1 Adapters"*:
>       - *"1: 3' (End) Adapters"*:
>          - *"Source"*: `Enter custom sequence`
>          - *"Custom 3' adapter sequence"*: `CTGTCTCTTATACACATCT`
>    - In *"Other Read Trimming Options"*
>       - *"Quality cutoff(s) (R1)"*: `20`
>    - In *"Read Filtering Options"*
>       - *"Minimum length (R1)"*: `20`
>    - {% icon param-select %} *"Additional outputs to generate"*: `Report`
>
> 2. Inspect the generated txt file (`Report`)
>
>    > <question-title></question-title>
>    >
>    > 1. What % reads contain adapter?
>    > 2. What % reads have been trimmed because of bad quality?
>    > 3. What % reads have been removed because they were too short?
>    >
>    > > <solution-title></solution-title>
>    > > 1. 56.8% reads contain adapter (`Reads with adapters:`)
>    > > 2. 35.1% reads have been trimmed because of bad quality (`Quality-trimmed:`)
>    > > 3. 0 % reads were removed because they were too short
>    > {: .solution }
>    {: .question}
{: .hands_on}


> <details-title>Trimming with Cutadapt</details-title>
>
> One of the biggest advantage of Cutadapt compared to other trimming tools (e.g. TrimGalore!) is that it has a good [documentation](https://cutadapt.readthedocs.io) explaining how the tool works in detail.
>
> Cutadapt quality trimming algorithm consists of three simple steps:
>
> 1. Subtract the chosen threshold value from the quality value of each position
> 2. Compute a partial sum of these differences from the end of the sequence to each position
>    (as long as the partial sum is negative)
> 3. Cut at the minimum value of the partial sum
>
> In the following example, we assume that the 3’ end is to be quality-trimmed with a threshold of 10 and we have the following quality values
>
> ```
> 42 40 26 27 8 7 11 4 2 3
> ```
>
> 1. Subtract the threshold
>
>     ```
>     32 30 16 17 -2 -3 1 -6 -8 -7
>     ```
>
> 2. Add up the numbers, starting from the 3' end (partial sums) and stop early if the sum is greater than zero
>
>     ```
>     (70) (38) 8 -8 -25 -23 -20, -21 -15 -7
>     ```
>
>     The numbers in parentheses are not computed (because 8 is greater than zero), but shown here for completeness.
>
> 3. Choose the position of the minimum (`-25`) as the trimming position
>
> Therefore, the read is trimmed to the first four bases, which have quality values
>
> ```
> 42 40 26 27
> ```
>
> Note that therefore, positions with a quality value larger than the chosen threshold are also removed if they are embedded in regions with lower quality (the partial sum is decreasing if the quality values are smaller than the threshold). The advantage of this procedure is that it is robust against a small number of positions with a quality higher than the threshold.
>
>
> Alternatives to this procedure would be:
>
> * Cut after the first position with a quality smaller than the threshold
> * Sliding window approach
>
>     The sliding window approach checks that the average quality of each sequence window of specified length is larger than the threshold. Note that in contrast to cutadapt's approach, this approach has one more parameter and the robustness depends of the length of the window (in combination with the quality threshold). Both approaches are implemented in Trimmomatic.
{: .details}


We can examine our trimmed data with FASTQE and/or FastQC.

> <hands-on-title>Checking quality after trimming</hands-on-title>
>
> 1. {% tool [FASTQE](toolshed.g2.bx.psu.edu/repos/iuc/fastqe/fastqe/0.3.1+galaxy0) %}: Re-run **FASTQE** with the following parameters
>    - {% icon param-files %} *"FastQ data"*: `Cutadapt Read 1 Output`
>    - {% icon param-select %} *"Score types to show"*: `Mean`
>
> 2. Inspect the new FASTQE report
>
>    > <question-title></question-title>
>    >
>    > Compare the FASTQE output to the previous one before trimming above. Has sequence quality been improved?
>    >
>    > {% snippet faqs/galaxy/features_scratchbook.md %}
>    >
>    > > <solution-title></solution-title>
>    > > Yes, the quality score emojis look better (happier) now.
>    > >
>    > > ![FASTQE before](../../images/quality-control/fastqe-mean-before.png "Before trimming")
>    > >
>    > > ![FASTQE after](../../images/quality-control/fastqe-mean-after.png "After trimming")
>    > >
>    > {: .solution }
>    {: .question}
{: .hands_on}

With FASTQE we can see we improved the quality of the bases in the dataset.

We can also, or instead, check the quality-controlled data with FastQC.


> <hands-on-title>Checking quality after trimming</hands-on-title>
>
> 1. {% tool [FASTQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0) %} with the following parameters
>    - {% icon param-files %} *"Short read data from your current history"*: `Cutadapt Read 1 Output`
>
> 2. Inspect the generated HTML file
>
{: .hands_on}

> <question-title></question-title>
> 1. Does the per base sequence quality look better?
> 2. Is the adapter gone?
>
> > <solution-title></solution-title>
> > 1. Yes. The vast majority of the bases have a quality score above 20 now.
> > ![Per base sequence quality](../../images/quality-control/per_base_sequence_quality-after.png "Per base sequence quality")
> >
> > 2. Yes. No adapter is detected now.
> > ![Adapter Content](../../images/quality-control/adapter_content-after.png)
> >
> {: .solution }
{: .question}

With FastQC we can see we improved the quality of the bases in the dataset and removed the adapter.

> <details-title>Other FastQC plots after trimming</details-title>
>
> ![Per tile sequence quality](../../images/quality-control/per_tile_sequence_quality-after.png)
> We have some red stripes as we've trimmed those regions from the reads.
>
> ![Per sequence quality scores](../../images/quality-control/per_sequence_quality_scores-after.png)
> We now have one peak of high quality instead of one high and one lower quality that we had previously.
>
> ![Per base sequence content](../../images/quality-control/per_base_sequence_content-after.png)
> We don't have equal representation of the bases as before as this is amplicon data.
>
> ![Per sequence GC content](../../images/quality-control/per_sequence_gc_content-after.png)
> We now have a single main GC peak due to removing the adapter.
>
> ![Per base N content](../../images/quality-control/per_base_n_content-after.png)
> This is the same as before as we don't have any Ns in these reads.
>
> ![Sequence length distribution](../../images/quality-control/sequence_length_distribution-after.png)
> We now have multiple peaks and a range of lengths, instead of the single peak with had before trimming when all sequences were the same length.
>
> ![Sequence Duplication Levels](../../images/quality-control/sequence_duplication_levels-after.png)
> > <question-title></question-title>
> >
> > What does the top overrepresented sequence `GTGTCAGCCGCCGCGGTAGTCCGACGTGG` correspond to?
> >
> > > <solution-title></solution-title>
> > > If we take the top overrepresented sequence
> > > ```
> > > >overrep_seq1_after
> > > GTGTCAGCCGCCGCGGTAGTCCGACGTGG
> > > ```
> > > and use [blastn](https://blast.ncbi.nlm.nih.gov/Blast.cgi) against the default Nucleotide (nr/nt) database we see the top hits are to 16S rRNA genes. This makes sense as this is 16S amplicon data, where the 16S gene is PCR amplified.
> > {: .solution }
> {: .question}
{: .details}


# Processing multiple datasets

## Process paired-end data

With paired-end sequencing, the fragments are sequenced from both sides. This approach results in two reads per fragment, with the first read in forward orientation and the second read in reverse-complement orientation. With this technique, we have the advantage to get more information about each DNA fragment compared to reads sequenced by only single-end sequencing:

```
    ------>                       [single-end]

    ----------------------------- [fragment]

    ------>               <------ [paired-end]
```
The distance between both reads is known and therefore is additional information that can improve read mapping.

Paired-end sequencing generates 2 FASTQ files:
- One file with the sequences corresponding to **forward** orientation of all the fragments
- One file with the sequences corresponding to **reverse** orientation of all the fragments

Usually we recognize these two files which belong to one sample by the name which has the same identifier for the reads but a different extension, e.g. `sampleA_R1.fastq` for the forward reads and `sampleA_R2.fastq` for the reverse reads. It can also be `_f` or `_1` for the forward reads and `_r` or `_2` for the reverse reads.

The data we analyzed in the previous step was single-end data so we will import a paired-end RNA-seq dataset to use. We will run FastQC and aggregate the two reports with MultiQC {% cite ewels2016multiqc %}.

> <hands-on-title>Assessing the quality of paired-end reads</hands-on-title>
>
> 1. Import the paired-end reads `GSM461178_untreat_paired_subset_1.fastq` and `GSM461178_untreat_paired_subset_2.fastq` from [Zenodo](https://zenodo.org/record/61771) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/61771/files/GSM461178_untreat_paired_subset_1.fastq
>    https://zenodo.org/record/61771/files/GSM461178_untreat_paired_subset_2.fastq
>    ```
>
> 2. {% tool [FASTQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0) %} with both datasets:
>    - {% icon param-files %} *"Raw read data from your current history"*: both the uploaded datasets.
>
>    {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>
> 3. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.9+galaxy1) %} with the following parameters to aggregate the FastQC reports of both forward and reverse reads
>      - In *"Results"*
>        - *"Which tool was used generate logs?"*: `FastQC`
>        - In *"FastQC output"*
>           - *"Type of FastQC output?"*: `Raw data`
>           - {% icon param-files %} *"FastQC output"*: `Raw data` files (output of both **FastQC** {% icon tool %})
>
> 4. Inspect the webpage output from MultiQC.
>
{: .hands_on}


> <question-title></question-title>
>
> 1. What do you think about the quality of the sequences?
> 2. What should we do?
>
> > <solution-title></solution-title>
> >
> > 1. The quality of the sequences seems worse for the reverse reads than for the forward reads:
> >     - Per Sequence Quality Scores: distribution more on the left, i.e. a lower mean quality of the sequences
> >     - Per base sequence quality: less smooth curve and stronger decrease at the end with a mean value below 28
> >     - Per Base Sequence Content: stronger bias at the beginning and no clear distinction between C-G and A-T groups
> >
> >    The other indicators (adapters, duplication levels, etc) are similar.
> >
> > 2. We should trim the end of the sequences and filter them with **Cutadapt** {% icon tool %}
> >
> {: .solution}
{: .question}

With paired-end reads the average quality scores for forward reads will almost always be higher than for reverse reads.

After trimming, reverse reads will be shorter because of their quality and then will be eliminated during the filtering step. If one of the reverse reads is removed, its corresponding forward read should be removed too. Otherwise we will get different number of reads in both files and in different order, and order is important for the next steps. Therefore **it is important to treat the forward and reverse reads together for trimming and filtering**.

> <hands-on-title>Improving the quality of paired-end data</hands-on-title>
> 1. {% tool [Cutadapt](toolshed.g2.bx.psu.edu/repos/lparsons/cutadapt/cutadapt/4.9+galaxy1) %} with the following parameters
>    - *"Single-end or Paired-end reads?"*: `Paired-end`
>       - {% icon param-file %} *"FASTQ/A file #1"*: `GSM461178_untreat_paired_subset_1.fastq` (Input dataset)
>       - {% icon param-file %} *"FASTQ/A file #2"*: `GSM461178_untreat_paired_subset_2.fastq` (Input dataset)
>
>          The order is important here!
>
>       - In *Read 1 Adapters* or *Read 2 Adapters*
>
>         No adapters were found in these datasets. When you process your own data and you know which adapter sequences were used during library preparation, you should provide their sequences here.
>
>    - In *"Other Read Trimming Options"*
>       - *"Quality cutoff(s) (R1)"*: `20`
>    - In *"Read Filtering Options"*
>       - *"Minimum length (R1)"*: `20`
>    - {%icon param-select%} *"Additional outputs to generate"*: `Report`
>
> 2. Inspect the generated txt file (`Report`)
>
>    > <question-title></question-title>
>    >
>    > 1. How many basepairs has been removed from the reads because of bad quality?
>    > 2. How many sequence pairs have been removed because they were too short?
>    >
>    > > <solution-title></solution-title>
>    > > 1. 44,164 bp (`Quality-trimmed:`) for the forward reads and 138,638 bp for the reverse reads.
>    > > 2. 1,376 sequences have been removed because at least one read was shorter than the length cutoff (322 when only the forward reads were analyzed).
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

In addition to the report, Cutadapt generates 2 files:
- Read 1 with the trimmed and filtered forward reads

tools_json:
["FASTQE", "toolshed.g2.bx.psu.edu/repos/iuc/fastqe/fastqe/0.3.1+galaxy0", "FASTQC", "toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0", "Cutadapt", "toolshed.g2.bx.psu.edu/repos/lparsons/cutadapt/cutadapt/4.9+galaxy1", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.9+galaxy1", "Nanoplot", "toolshed.g2.bx.psu.edu/repos/iuc/nanoplot/nanoplot/1.41.0+galaxy0", "PycoQC", "toolshed.g2.bx.psu.edu/repos/iuc/pycoqc/pycoqc/2.5.2+galaxy0"]


topic: sequence-analysis
tutorial: human-reads-removal
title: Removal of human reads from SARS-CoV-2 sequencing data
difficulty: intermediate
time_estimation: 1h
requirements_json: []
tags_json: ["covid19"]
content_hash: 700bf822c021e072a1059d130a0de59d
last_modified: 2026-05-15T22:28:14.611536
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/sequence-analysis/tutorials/human-reads-removal/tutorial.html

content:
Patient samples for pathogen detection are usually "contaminated" with human host DNA.
Such contamination, if not removed from sequencing data, may pose an issue with certain types of
data analyses.
Another issue is that submitting the raw sequenced reads with the human sequence "contamination" included to a public database may simply violate national or institutional regulations for handling patient data.
For this second reason in particular, researchers will regularly have to remove traces of human sequencing data even from samples that, by their nature, should be highly enriched for just pathogen reads, as is the case, for example, for ampliconic viral samples, in which viral sequences have been PCR-amplified with virus-specific primers before sequencing.

This tutorial will guide you through a typical workflow for clearing human sequences from any kind (ampliconic or not) of viral sequenced sample, which retains non-human reads in a format ready to be submitted to public databases.

> <comment-title>Nature of the input data</comment-title>
> We will use sequencing data of bronchoalveolar lavage fluid (BALF) samples obtained from early COVID-19 patients in China as our input data.
> Since such samples are expected to be contaminated signficantly with human sequenced reads, the effect of the cleaning steps will be much more apparent than for ampliconic sequencing data from diagnostic patient swabs.
> Nevertheless the cleaning processs does not rely on any particular sample isolation or preprcocessing method and could be used unaltered on, for example, ARTIC amplified SARS-CoV-2 samples.
>
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get data

As always, it is best to give each analysis you are performing with Galaxy its own history, so let's start with preparing this:

> <hands-on-title>Prepare new history</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

## Getting the sequencing data

The sequenced reads datasets used in this tutorial (4 files representing 2 Illumina paired-end sequenced samples) have been deposited on [Zenodo](https://zenodo.org/record/3732359/) and can be uploaded to Galaxy via their URLs.
After that, we will arrange the uploaded data into a collection in our Galaxy history to facilitate its handling in the analysis.

> <hands-on-title>Data upload and rearrangement</hands-on-title>
>
> 1. Upload the data from Zenodo via URLs
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    The URLs for our example data are these:
>
>    ```
>    https://zenodo.org/record/3732359/files/SRR10903401_r1.fq.gz
>    https://zenodo.org/record/3732359/files/SRR10903401_r2.fq.gz
>    https://zenodo.org/record/3732359/files/SRR10903402_r1.fq.gz
>    https://zenodo.org/record/3732359/files/SRR10903402_r2.fq.gz
>    ```
>
> 2. Arrange the data into a paired dataset collection
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
>    For the example datasets this means:
>    - You need to tell Galaxy about the suffix for your forward and reverse reads, respectively:
>      - set the text of *unpaired forward* to: `_r1.fq.gz`
>      - set the text of *unpaired reverse* to: `_r2.fq.gz`
>      - click: `Auto-pair`
>
>      All datasets should now be moved to the *paired section* of the dialog, and the middle column there should show that only the sample accession numbers, *i.e.* `SRR10903401` and `SRR10903402`, will be used as the pair names.
>
>    - Make sure *Hide original elements* is checked to obtain a cleaned-up history after building the collection.
>    - Click *Create Collection*
>
{: .hands_on}

# Read trimming and mapping

In order to learn which of the reads of each of the input samples are of likely human origin, we need to map the reads to the human reference genome. To improve the quality of this step we will also trim low-quality bases from the ends of all reads before passing them to the read mapping software.

> <comment-title></comment-title>
>
> Do you want to learn more about the principles behind quality control (including trimming) and mapping? Follow our dedicated tutorials:
> - [Quality Control]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).
> - [Mapping]({% link topics/sequence-analysis/tutorials/mapping/tutorial.md %}).
{: .comment}

> <hands-on-title>Trim reads and map them to the human reference genome</hands-on-title>
>
> 1. {% tool [Trimmomatic](toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.38.0) %} with default settings and:
>    - *"Single-end or paired-end reads?"*: `Paired-end (as collection)`
>      - *"Select FASTQ dataset collection with R1/R2 pair"*: the collection of sequenced reads input datasets
> 2. {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2) %} with the following parameters:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>      - *"Using reference genome"*: `Human Dec. 2013 (GRCh38/hg38)(hg38)`
>
>        The exact name of this version of the human reference genome may vary between Galaxy servers. Start typing `hg38` in the select box to reveal available choices on your server.
>
>    - *"Single or Paired-end reads"*: `Paired Collection`
>      - *"Select a paired collection"*: the `paired` output of **Trimmomatic**
>
{: .hands_on}

# Obtaining the identifiers of non-human read pairs

The mapping step above has produced a collection of BAM datasets that store, for each sample, which of its reads map to the human genome (and if so to which position on the genome though we are not interested in this information), and which ones could not be mapped to it. It is these *unmapped* reads that we are interested in, but how do we get at them?

The logic of the following steps is:

1. Filter the BAM dataset for each sample for only those reads that could not be mapped to the human reference genome, and for which also the read mate in the pair could not be mapped to this genome.

   In other words, if only one read of a read pair can be mapped to the human genome we play it safe and discard the pair.

2. Emit the remaining reads in a format (we choose FASTA here) from which it is easy to extract the read identifiers

3. Extract the read identifiers

> <hands-on-title>Filter for read pairs not mapping to the human genome and extract their identifiers</hands-on-title>
>
> 1. {% tool [Samtools fastx](toolshed.g2.bx.psu.edu/repos/iuc/samtools_fastx/samtools_fastx/1.9+galaxy1) %} with the following parameter settings:
>    - {% icon param-collection %} *"BAM or SAM file to convert"*: the output of **Map with BWA-MEM**
>    - *"Output format"*: `FASTA`
>    - *"outputs"*: select `READ1`
>
>      With this setting we are choosing to retrieve only the forward reads of each pair as a single result dataset.
>      Since the forward and reverse reads of a pair share their read identifier, we do not need the reverse reads.
>
>    - *"Omit or append read numbers"*: `Do not append /1 and /2 to read names`
>
>      Adding this first or second read of the pair indicator to the read name would make the result dependent on which half of the reads we retrieve. We do not want this.
>
>    - *"Require that these flags are set"*: 
>      - `read is unmapped`
>      - `mate is unmapped`
>
>      This setting makes sure we retain only read pairs for which none of the two reads could be mapped to the human genome.
>
> 2. {% tool [Select lines that match an expression](Grep1) %} with the following parameters:
>    - {% icon param-collection %} *"Select lines from"*: the output of **Samtools fastx**
>    - *"that"*: `Matching`
>    - *"the pattern"*: `^>.+`
>
>    This will select only those lines from the FASTA inputs that start with a `>` character, *i.e.* will retain the identifier lines, but not the sequence lines.
>
> 3. {% tool [Replace Text in entire line](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2) %} with the following parameters:
>    - {% icon param-collection %} *"File to process"*: the output of **Select**
>    - {% icon param-repeat %} *"Replacement"*:
>      - *"Find Pattern"*: `^>(.+)`
>      - *"Replace with"*: `\1`
>
>      This find and replacement pattern will work on lines starting with a `>` character and replace each such line with the content following the `>` character (the `(.+)` part that gets referenced as `\1` in the replacement pattern).
>
{: .hands_on}

# Use the non-human read identifiers to extract the reads of interest from the original inputs

We are almost there! We already know the identifiers of all reads we would like to retain, we only need a way to filter the original input datasets for these identifiers.

> <comment-title>Why not use the unmapped reads directly?</comment-title>
>
> When you followed the previous steps you may have noticed that **Samtools fastx** offers *fastq* as an output format.
> Since we used this tool to report the unmapped reads, you may be wondering why we did not simply ask the tool to write the reads in the format we want and be done with it.
>
> The answer is: we could have done that (and some people follow this approach), but remember that we trimmed the reads before mapping them, which means the extracted unmapped reads would not be identical to the input reads.
>
> It is good practice to submit sequencing data to public databases in as raw a state as possible to leave it up to the people downloading that data to apply processing steps as they see fit.
>
{: .comment}

The **seqtk_subseq** tool offfers the identifier-based filtering functionality we are looking for.
As a slight complication, however, the tool is not prepared to handle paired read collections like the one we organized our input data into.
For this reason, we need to first unzip our collection into two simple list collections - one with the forward reads, the other one with the reverse reads of all samples.
Then, after processing both with **seqtk_subseq**, we zip the filtered collections back into one.

> <hands-on-title>Identifier-based extraction of non-human reads from the input data</hands-on-title>
>
> 1. {% tool [Unzip Collection](__UNZIP_COLLECTION__) %} with the following setting:
>    - *"Input Paired Dataset"*: the collection of original sequenced reads input datasets
>
> 2. {% tool [seqtk_subseq](toolshed.g2.bx.psu.edu/repos/iuc/seqtk/seqtk_subseq/1.3.1) %} with the following parameters:
>    - {% icon param-collection %} *"Input FASTA/Q file"*: the first (forward reads) output of **Unzip Collection**
>    - *"Select source of sequence choices"*: `FASTA/Q ID list`
>      - {% icon param-collection %} the collection of read identifiers; output of **Replace Text**
>
> 3. {% tool [seqtk_subseq](toolshed.g2.bx.psu.edu/repos/iuc/seqtk/seqtk_subseq/1.3.1) %} with the following parameters:
>    - {% icon param-collection %} *"Input FASTA/Q file"*: the second (reverse reads) output of **Unzip Collection**
>    - *"Select source of sequence choices"*: `FASTA/Q ID list`
>      - {% icon param-collection %} the collection of read identifiers; output of **Replace Text**
>
> 4. {% tool [Zip Collection](__ZIP_COLLECTION__) %} with the following settings:
>    - {% icon param-collection %} *"Input Dataset (Forward)"*: the collection of cleaned forward reads; output of the first run of **seqtk_subseq**
>    - {% icon param-collection %} *"Input Dataset (Reverse)"*: the collection of cleaned reverse reads; output of the second run of **seqtk_subseq**
>
{: .hands_on}

# Conclusion


Data cleaning is a standard procedure for clinical sequencing datasets and is a rather straightforward process in Galaxy.
You just learnt how to remove human reads from any number of paired-end sequenced samples in parallel using collections and just a handful of tools.
Cleaning of single-end data would use the same tools (with adjusted settings), but would work on data arranged in a simple list collection instead of in a paired collection.

tools_json:
["Trimmomatic", "toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.38.0", "Map with BWA-MEM", "toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2", "Samtools fastx", "toolshed.g2.bx.psu.edu/repos/iuc/samtools_fastx/samtools_fastx/1.9+galaxy1", "Select lines that match an expression", "Grep1", "Replace Text in entire line", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2", "Unzip Collection", "__UNZIP_COLLECTION__", "seqtk_subseq", "toolshed.g2.bx.psu.edu/repos/iuc/seqtk/seqtk_subseq/1.3.1", "Zip Collection", "__ZIP_COLLECTION__"]


topic: sequence-analysis
tutorial: sars-with-galaxy-on-anvil
title: SARS-CoV-2 Viral Sample Alignment and Variant Visualization
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: ["covid19", "gmod", "jbrowse1"]
content_hash: a04a51e741e608ef7a8102ca72c3b0ee
last_modified: 2026-05-15T22:28:14.779366
zenodo_link: https://doi.org/10.5281/zenodo.8115178
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/sequence-analysis/tutorials/sars-with-galaxy-on-anvil/tutorial.html

content:
There is a growing need for undergraduate students to learn cutting-edge concepts in genomics data science, including performing analysis on the cloud instead of a personal computer. 

This lesson aims to introduce a mutant detection bioinformatics pipeline based on a publicly available genetic sample of SARS-CoV-2. Students will be introduced to the sequencing revolution, variants, genetic alignments, and essentials of cloud computing prior to the lab activity. During the lesson, students will work hands-on with the point-and-click Galaxy interface on the AnVIL cloud computing resource to check data, perform an alignment, and visualize their results.

> <agenda-title></agenda-title>
> 
> In this tutorial we will deal with:
> 
> 1. TOC 
> {:toc}
>
{: .agenda}


# Background Concepts

## What is a Variant?

This lecture module introduces genetic variants. It provides several examples of genetic variants, background on the structure of DNA, and a review of the “Central Dogma” of molecular biology - the process of turning DNA into RNA into protein.

Learning Objectives
- Answer “what is a genetic variant?”
- Learn about the molecular structure of a variant

<a href="http://www.youtube.com/watch?feature=player_embedded&v=kELyrelihP8" target="https://youtu.be/kELyrelihP8"><img src="http://img.youtube.com/vi/kELyrelihP8/0.jpg" alt="Video thumbnail image for the What is a Variant background video." width="240" height="180" border="10" /></a>

## The Sequencing Revolution

This lecture module introduces the history of the sequencing revolution. It highlights the enormous proliferation of genomic data that has accompanied the rapidly growing technology. It also suggests opportunities for careers in genomics, as well as an in-depth look at how some sequencing technologies actually work.

Learning Objectives
- Learn the history the sequencing revolution
- Introduce the sequencing workforce
- Explore the evolution of sequencing technology

<a href="http://www.youtube.com/watch?feature=player_embedded&v=tXLTHqNT6Jg" target="https://youtu.be/tXLTHqNT6Jg"><img src="http://img.youtube.com/vi/tXLTHqNT6Jg/0.jpg" alt="Video thumbnail image for the Sequencing Revolution background video." width="240" height="180" border="10" /></a>

## Alignments

This lecture module introduces the structure of genomic data and how alignments work. It touches on the “shredded book” analogy, demonstrates how short chunks of data can be compared to find variation, and reviews data files needed for alignments, including reference genomes and read data. It also reviews some diverse applications of variant detection made possible via alignment tools.

Learning Objectives
- Learn about data as “reads” & shredded books
- Become familiar with reference genomes and alignments
- Explore the file structure of genomic data and quality scoring

<a href="http://www.youtube.com/watch?feature=player_embedded&v=MEZP_AzlLyg" target="https://youtu.be/MEZP_AzlLyg"><img src="http://img.youtube.com/vi/MEZP_AzlLyg/0.jpg" alt="Video thumbnail image for the Alignments background video." width="240" height="180" border="10" /></a>

## Cloud Computing

This lecture module introduces cloud computing and computing architecture. It reviews the utility of cloud computing for genomics and also highlights how all modules of this activity fit together.

Learning Objectives
- Learn about different types of computers
- Answer the question “What is cloud computing?”
- Learn about cloud computing for genomics
- Revisit the big picture, from variants to alignments

<a href="http://www.youtube.com/watch?feature=player_embedded&v=hreCMUQaA6s" target="https://youtu.be/hreCMUQaA6s"><img src="http://img.youtube.com/vi/hreCMUQaA6s/0.jpg"  alt="Video thumbnail image for the Cloud Computing background video." width="240" height="180" border="10" /></a>

# Overview

This overview video introduces the lab activity. It briefly reviews some of the essential background for the activity, highlights key areas to focus on for activity assessment questions, and provides a detailed walk-through of the steps - from starting AnVIL and Galaxy to browsing the genome and shutting down the cloud computing instance.

Learning Objectives:
- Review lecture content
- Review big steps in the analysis
- Cover the setup on AnVIL in detail

<a href="http://www.youtube.com/watch?feature=player_embedded&v=D-bkZRA2TQ4" target="https://www.youtube.com/watch?v=OFGa6x2bGHs"><img src="http://img.youtube.com/vi/D-bkZRA2TQ4/0.jpg" alt="Video thumbnail image for the Activity Overview background video." width="240" height="180" border="10" /></a>

# Set Up

In the next few steps, you will walk through how to get set up to use Galaxy on the AnVIL platform. AnVIL is centered around different “Workspaces”. Each Workspace functions almost like a mini code laboratory - it is a place where data can be examined, stored, and analyzed. The first thing we want to do is to copy or “clone” a Workspace to create a space for you to experiment.

> <comment-title></comment-title>
> Because AnVIL runs on a commerical cloud provider, you will need to have set up billing for yourself or through your institution to follow along with this exercise, or you will need to be added to a billing account created for a training event.
> 
> Learn more about options for creating billing accounts at this link: https://jhudatascience.org/AnVIL_Book_Getting_Started/overview-pis.html.
{: .comment}

> <tip-title>Screen view</tip-title>
>
> * At this point, it might make things easier to open up a new window in your browser and split your screen. That way, you can follow along with this guide on one side and execute the steps on the other.
{: .tip}

> <hands-on-title>Clone the Workspace </hands-on-title>
>
> 1. Use a web browser to go to the AnVIL website. In the browser type: `anvil.terra.bio`. Log into AnVIL.
> 2. Click "View Workspaces". 
>   - Select the “Public” tab. 
>   - In the top search bar type the activity workspace `SARS-CoV-2-Genome`. You can also go directly to the following link: https://anvil.terra.bio/#workspaces/gdscn-exercises/SARS-CoV-2-Genome.
>   ![Screenshot of the AnVIL platform workspaces page highlighting the Public tab and the SARS-CoV-2-Genome Workspace search result.](../../images/sars-with-galaxy-in-anvil/01-sars-workspace.png)
> 3. Clone the workspace by clicking the teardrop button and selecting “Clone”.
>    ![Screenshot showing the teardrop button. The button has been clicked revealing the "clone" option. The Clone option and the teardrop button are highlighted.](../../images/sars-with-galaxy-in-anvil/02-clone-button.png)
>   - In the new window, give your Workspace clone a name by adding an underscore (“_”) and your name.
>   - Next, select the Billing project provided by your instructor. 
>   - Leave the Description and Authorization Domain boxes as-is.
> ![Screenshot showing the "clone a workspace" popout. The Workspace name, Billing Project, and Clone Workspace button have been filled in and highlighted.](../../images/sars-with-galaxy-in-anvil/03-clone-settings.png)
>   - Click “CLONE WORKSPACE”.
> 
{: .hands_on}

## Starting Galaxy

Galaxy is a great tool for performing bioinformatics analysis without having to update software or worry too much about coding. In order to use Galaxy, we need to create a cloud environment. This is like quickly renting a few computers from Google as the engine to power our Galaxy analysis. 

> <warning-title>Internet Browser</warning-title>
> Google Chrome is the most recommended browser to use AnVIL Galaxy cloud environments to operate as expected. Safari and Firefox may be used as well, though there may be some functionality that is not supported in these or other browsers.
{: .warning}

> <hands-on-title>Clone the Workspace </hands-on-title>
>
> 1. In your new Workspace, click on the “ANALYSES” tab. Next, click on “START”. You should see a popup window on the right side of the screen. 
> ![Screenshot of the Workspace Notebooks tab. The notebook tab name and the plus button that starts a cloud environment for Galaxy have been highlighted.](../../images/sars-with-galaxy-in-anvil/05-start-galaxy.png)
> 2. Click on the Galaxy logo to proceed.
>    - Click on “NEXT” and “CREATE” to keep all settings as-is.
>    - ![The NEXT button among cloud environments has been highlighted.](../../images/sars-with-galaxy-in-anvil/06-galaxy-next.png)
>    - ![The CREATE button among cloud environments has been highlighted.](../../images/sars-with-galaxy-in-anvil/07-galaxy-create.png)
>    - Click on the Galaxy icon. You will see that the environment is still being set up. This will take 8-10 minutes. 
>    ![The Galaxy icon appears if the environment has been successfully launched.](../../images/sars-with-galaxy-in-anvil/08-galaxy-provisioning.png)
> 3. When it is done, click “Open”. You might need to refresh the page.
> 
{: .hands_on}

> <tip-title>Refresh</tip-title>
> Remember that you can refresh your browser or navigate away at any time. This is because the connection to the environment is in the cloud, not on your personal computer.
>
{: .tip}

You can also follow along with the first ~2 minutes of [this video](https://jhudatascience.org/AnVIL_Book_Getting_Started/starting-galaxy.html) to start Galaxy on AnVIL.

## Navigating Galaxy

Notice the three main sections.

**Tools** - These are all of the bioinformatics tool packages available for you to use.

**The Main Dashboard** - This contains flash messages and posts when you first open Galaxy, but when we are using data this is the main interface area.

**History** - When you start a project you will be able to see all of the documents in the project in the history. Now be aware, this can become very busy. Also the naming that Galaxy uses is not very intuitive, so you must make sure that you label your files with something that makes sense to you.

![Screenshot of the Galaxy landing page. The Tools and History headings have been highlighted.](../../images/sars-with-galaxy-in-anvil/10-galaxy-on-anvil.png)

The welcome page includes links to tutorials. You may try these out on your own. If you want to try a new analysis this is a good place to start.

# Exercise One: Importing Data into Galaxy

Luckily, we linked to the original data when we cloned our Workspace! We have three files we will need for our activity. These are (1) the reference genome for SARS-CoV-2, and both forward (2) and reverse (3) reads for our sample. Our sample has two sets of reads because the scientists who collected it used paired-end sequencing. The reference genome ends in “.fasta” because it has already been cleaned up by scientists. The sample we are looking at ends in ".fastq" because it is raw data from the sequencer.

> <hands-on-title>Import Data from the Workspace</hands-on-title>
>
> 1. Click on Upload Data in the Tools pane.
> 2. Click on “Choose remote files” at the bottom of the popup. Double-click the top selection, which is the Workspace folder, then “Tables/” then “reference/”. Click the reference .fasta file so that it is highlighted in green and click “Ok”.
>    ![Screenshot of the Galaxy Data upload page with the current AnVIL workspace highlighted.](../../images/sars-with-galaxy-in-anvil/11-galaxy-data-workspace.png)
> 3. Now that your reference has been added, click “Choose remote files” again to add the two sample files. Double-click the Workspace folder, then “Tables/” then “samples/”. Click the two sample `fastq` files so that they are highlighted in green and click “OK”.
> 4. Click “Start” and once complete, you can click “Close”.
> 5. Confirm your upload worked by looking at the file names in the History pane.
> 
{: .hands_on}

# Exercise Two: Examining Files in Galaxy

Now we have some data in our account we can look at it. In this exercise we will see data in fastq format. This is the typical output from an Illumina Sequencer, but also the standard format for most alignment software.

## Examining Inputs

Use your mouse and click on the eye icon {% icon galaxy-eye %} of the first file `VA_sample_forward_reads.fastq`. In the Main screen you will see something like this:

![Screenshot of a fastq file. The data includes DNA sequences but also includes many coded characters, making it hard to understand.](../../images/sars-with-galaxy-in-anvil/12-fastq-view.png)

> <question-title></question-title>
>
> 1. How many lines in a .fastq file represent an individual read?
> 2. What does each line represent?
> 3. Why is the final line for each read (the quality score) important?
>
> > <solution-title></solution-title>
> >
> > 1. Four lines represent one read.
> > 2. The lines represent: 1 - A sequence identifier, 2 - The sequence (the base calls; A, C, T, G and N), 3 - A separator (not really data), 4 - The base call quality scores.
> > 3. It can help us filter out data that is wrong and/or low quality.
> >
> {: .solution}
{: .question}

## Quality Scoring

FastQC is a tool which aims to provide simple quality control checks on raw sequence data coming from high throughput sequencing pipelines. It provides a set of analyses which you can use to get a quick impression of whether your data has any problems of which you should be aware before doing any further analysis. 

> <hands-on-title>Determine the Quality of the Samples</hands-on-title>
>
> 1. Find {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0) %} in the GENOMIC FILE MANIPULATION: FASTQ Quality Control tool folder.
>    - {% icon param-file %} *"Raw read data from your current history"*: `VA_sample_forward_reads.fastq`
> 2. Run the tool.
>
{: .hands_on}

The main dash will highlight in green if everything is okay. In the history, you will see the new files turn yellow, then green. If the job fails it will show an error. 

Click on the eye icon {% icon galaxy-eye %} in the new file in the history “FASTQC on data2 Webpage”. 

You will open up a summary report for the sequencing file:

![Screenshot of the FastQC results. The Basic Statistics and Per Base Sequence Quality sections for the report on VA_sample_forward_reads.fastq are visible.](../../images/sars-with-galaxy-in-anvil/13-fastqc-report.png)

> <question-title></question-title>
>
> 1. Explore “Basic Statistics”. How many total reads are there? Have any been flagged as poor quality? What is the sequence length?
> 2. Explore “Per base sequence quality”. Based on the Basic Statistics, is 28-40 a good or bad quality score? 
> 3. Is it okay to proceed based on the per base sequence quality?
>
> > <solution-title></solution-title>
> >
> > 1. 43,522 reads. Zero flagged as poor quality. Sequence length is 3-301 base pairs.
> > 2. 28-40 is pretty good. 
> > 3. Yes, because the per base sequence quality is good (“in the green”).
> >
> {: .solution}
{: .question}

{% snippet topics/sequence-analysis/faqs/quality_score.md %}

# Exercise Three: Alignment

Given that our data has passed some quality checks, we will try to align the data to the reference genome. In this case it is simple, a viral genome. A human sequencing project will generate much larger data sets. There are many aligners, but we will start off looking at a simple aligner BWA-MEM. This example uses paired data.

We will use our two SARS data files, which are ready for alignment.

- `VA_sample_forward_reads.fastq`
- `VA_sample_reverse_reads.fastq`

> <hands-on-title>Align to the Reference Dataset</hands-on-title>
>
> 1. Go to GENOMICS ANALYSIS and expand the Mapping menu. Select {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2) %}. This program will align your reads to your SARS reference genome. Some of our reads are >100 base pairs so we will use the MEM option. 
>    - *"Will you select a reference genome from your history or use a built-in index?"*: Use a genome from history and build index.
>    - {% icon param-file %} *"Use the following dataset as the reference sequence"*: `SARS-CoV-2_reference_genome.fasta`
>    - *"Single or Paired-end reads"*: `Paired`
>    - *"Select first set of reads"*: `VA_sample_forward_reads.fastq`
>    - *"Select second set of reads"*: `VA_sample_reverse_reads.fastq`
> 2. Run the tool.
>
{: .hands_on}

The output file is a `BAM` file, which lists where each read aligns to the reference genome and whether there are any differences. You can click the eye button to preview the results, but the results are not easy to interpret visually (much like the `fastq` files). Instead you will use a genome viewer in the next step.

> <question-title></question-title>
>
> 1. What is alignment software (for example, BWA-MEM) actually doing?
> 2. In this example, we are using paired fastq (“paired end”) data. What is an advantage of using paired data?
>
> > <solution-title></solution-title>
> >
> > 1. Alignment tools figure out the optimal positioning of the reads next to the reference genome to minimize mismatches and gaps. 
> > 2. Paired data is made up of fragments that are read twice (forward and reverse). Paired data improves data accuracy.
> >
> {: .solution}
{: .question}

# Viewing aligned data

We have aligned our data but it is currently a table of where the reads align. This is hard to read, so we will use [JBrowse](https://jbrowse.org/jbrowse1.html) to view the data.

> <hands-on-title>Visualize Reference Data</hands-on-title>
>
> 1. Scroll down in the Tools menu to STATISTICS AND VISUALIZATION. Under "Graph/Display Data", select {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.9+galaxy0) %}.
>    - {% icon param-select %} *"Reference genome to display"*: `Use a genome from history` 
>    - {% icon param-select %} *"Select the reference genome"*: `SARS-CoV-2_reference_genome.fasta`
> 2. Click "Run Tool".
> 3. You should see a new JBrowse item appear in your history. Click the eye icon {% icon galaxy-eye %} to open JBrowse. 
>
{: .hands_on}

You will need to click on the magnifying glasses to zoom in, but you should see the A,C,G, and Ts and their corresponding colors that make up the SARS-CoV-2 genome!

![Screenshot of preliminary JBrowse results. The eye icon is highlighted, as it should be used to open the JBrowse viewer. The magnifying glasses in JBrowse are also highlighted as they enable zooming in to see the individual bases.](../../images/sars-with-galaxy-in-anvil/15-galaxy-jbrowse-ref.png)

This is interesting, but it doesn’t let us compare the genome to the sample we have. We suspect there may be some differences that indicate our sample is the delta variant. 

> <hands-on-title>Visualize Aligned Data</hands-on-title>
>
> 1. Scroll down in the Tools menu to STATISTICS AND VISUALIZATION. Under "Graph/Display Data", select {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.9+galaxy0) %}.
>    - {% icon param-select %} *"Reference genome to display"*: `Use a genome from history` 
>    - {% icon param-select %} *"Select the reference genome"*: `SARS-CoV-2_reference_genome.fasta`
>    - {% icon param-repeat %} *"+ Insert Track Group"*
>    - {% icon param-repeat %} *“+ Insert Annotation Track”*
>    - {% icon param-select %} *"Track Type"*: `BAM Pileups`
>    - {% icon param-toggle %} *"Autogenerate SNP Track"*: `Yes`
>    - {% icon param-text %} *"Maximum size of BAM chunks"*: Add one more zero: `50000000`
> 2. Click "Run Tool".
>
{: .hands_on}

You should see a new JBrowse item appear in your history. Click the eye icon {% icon galaxy-eye %} to open JBrowse. Make sure that all boxes are checked on the left side: “Available Tracks”. The tracks will show up in the order that you click on them.

> <hands-on-title>Visualize Aligned Data</hands-on-title>
>
> 1. Select the tracks from the BWA-MEM tool run.
>    - {% icon param-check %} `Map with BWA-MEM on data 9, data 8, and data 7 (mapped reads in BAM format)`
>    - {% icon param-check %} `Map with BWA-MEM on data 9, data 8, and data 7 (mapped reads in BAM format) - SNPs/Coverage`
> 2. You should see a new JBrowse item appear in your history. Click the eye icon {% icon galaxy-eye %} to open JBrowse. 
>
{: .hands_on}

![Screenshot of JBrowse viewer. The Available Tracks sidebar is shown, with both sample data boxes checked.](../../images/sars-with-galaxy-in-anvil/14-galaxy-jbrowse-bams.png)

Let’s look at an example mutation in our sample. Type in the reference position “24410” and click “Go”. You should see a bunch of “A”s highlighted in green throughout our sample. The reference sequence (top line) is a “G” but all of the reads are an “A”. This means that our sample is genetically different from the established SARS-CoV-2 reference genome! Researchers often call these single base differences “SNPs” - Single Nucleotide Polymorphisms.

> <question-title></question-title>
>
> 1. How long is the SARS-CoV-2 genome? Hint: zoom out and scroll to the end of the genome.
> 2. Locate position 23,603. This is the site of an important mutation in the spike protein of the delta variant “P681R”. In this mutation, the amino acid proline is replaced by arginine. Is this mutation present at position 23,603 in our sample? Based on the evidence, do you think this sample is a delta variant?
>
> > <solution-title></solution-title>
> >
> > 1. ~29,904 bp
> > 2. Yes! C has become a G. Yes, this sample is probably a delta variant because this mutation is indicative of the delta variant.
> >
> {: .solution}
{: .question}

> <details-title>Sequencing errors</details-title>
>
> It’s possible to make mistakes in the data preparation before we get to the data analysis. Sometimes this happens when the samples are being prepared in the lab and sometimes this happens because the sequencer makes a mistake. This is one reason why quality scores are helpful. With millions of reads of data, it’s more likely that we see a “SNP” that is actually an accident. Multiple copies of the same areas of our data (“read depth”) help us be sure it’s a real SNP. When we compare across lots of aligned reads of the same area, we can determine the actual sequence by consensus. For example, we can be reasonably confident that the “G” at position 1,203 is just a sequencing or lab mistake.
>
{: .details}

# Export Your History

It’s a good idea to export your “History” so that your collaborators can see what you did. 

> <hands-on-title>Export History to Workspace</hands-on-title>
>
> 1. Click on the History Menu {% icon galaxy-history-options %} and click on “Export History to File”.
>   ![History dropdown menu button and Export History to File buttons are highlighted.](../../images/sars-with-galaxy-in-anvil/16-export-history.png)
> 2. Make sure you select “to a remote file”. Then, click to select where to export your History. On the popup menu, select your Workspace name, then select “Other Data”. Finally, select “Files”. Then click “Select this folder”. Make sure the export directory looks correct. 
> 3. Next, name your history “SARS Galaxy Variant Detection” and click “Export”.
>   ![When exporting History, make sure you select "to a remote file". The directory should be your workspace name followed by Other Data and Files. The export button is highlighted.](../../images/sars-with-galaxy-in-anvil/17-export.png)
> 4. Back at your Workspace, click on the “Data” tab, and the Files folder. You should now see the History export in your files. If you click on the file, you can download it or view it in Google Cloud Storage Browser.
>   ![Back on AnVIL, the Data tab and Files folder are selected. A file called SARS_Galaxy_Variant_Detection is now present.](../../images/sars-with-galaxy-in-anvil/18-export-verify.png)
>
{: .hands_on}

# Wrap-up

Once you are done with the activity, you’ll need to shut down your Galaxy cloud environment. This frees up the cloud resources for others and minimizes computing cost. The following steps will delete your work, so make sure you are completely finished at this point. Otherwise, you will have to repeat your work from the previous steps.

> <hands-on-title>Shut Down Galaxy in AnVIL</hands-on-title>
> 1. Return to AnVIL, and find the Galaxy logo that shows your cloud environment is running. Click on the Galaxy logo.
>   ![Screenshot of the Workspace menu. The currently running Galaxy cloud environment logo on the right of the page is highlighted.](../../images/sars-with-galaxy-in-anvil/19-galaxy-env.png)
> 2. Click "Settings".
>   ![Screenshot of the Environment details menu. The Settings button is highlighted.](../../images/sars-with-galaxy-in-anvil/20-galaxy-edit.png)
> 3. Next, scroll down and click on “DELETE ENVIRONMENT”:
>   ![Screenshot of the cloud environment pop out menu. The “DELETE ENVIRONMENT” button is highlighted.](../../images/sars-with-galaxy-in-anvil/21-galaxy-delete.png)
> 4. Finally, select “Delete everything, including persistent disk”. Make sure you are done with the activity and then click “DELETE”.
>    ![Screenshot of the cloud environment pop out menu. The “Delete everything, including persistent disk” radio button has been checked and is highlighted. The “DELETE” button is highlighted.](../../images/sars-with-galaxy-in-anvil/22-galaxy-delete-pd.png)
>
{: .hands_on}

# Conclusion
Congratulations! You have run your first analysis using Galaxy in the AnVIL platform!

tools_json:
["FastQC", "toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0", "Map with BWA-MEM", "toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2", "JBrowse", "toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.9+galaxy0"]


