# tutorials

topic: imaging
tutorial: electrophoresis-gel-bands-image-analysis
title: Quantification of electrophoresis gel bands using QuPath and Galaxy imaging tools
difficulty: intermediate
time_estimation: 1h
requirements_json: ["type: internal"]
tags_json: ["ROI selection", "Conversion", "Object feature extraction", "Multi-channel image"]
content_hash: f795a47363bc93e12f7a5f7b96294bc3
last_modified: 2026-05-27T09:01:49.078441
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/imaging/tutorials/electrophoresis-gel-bands-image-analysis/tutorial.html

content:
Electrophoresis gel images are fundamental tools in molecular biology and biochemistry for separating DNA, RNA, or protein fragments by size or charge ({% cite Green2019 %}). Quantification of these bands allow for estimating product yield, comparing amplification efficiency, verifying PCR products, calculating molecular weights, and assessing enzyme activity or protein purity ({% cite Gallagher2017 %}).

Reliable measurement of band intensity provides quantitative insights into gene expression and other biochemical properties ({% cite Gassmann2009 %}). However, manual quantification can be labor-intensive and prone to user bias.

Modern image analysis software suites such as **QuPath** ({% cite Bankhead2017 %}) provide interactive tools for defining **regions of interest (ROIs)** directly on gel images. These ROIs can be exported as **GeoJSON** files, which encode the spatial coordinates of the selected bands in a standardized, machine-readable format.

Once the ROI file is generated, bioinformatics platforms like **Galaxy** ({% cite Galaxy2024 %}) enable automated processing of these coordinates, measurement of pixel intensities for each band, and generation of transparent, reproducible quantification results.

Recent advances such as **GelGenie** ({% cite Aquilina2025 %}) further highlight the trend towards AI-powered, automated analysis pipelines for electrophoresis gels, reinforcing the need for accessible, reproducible workflows that integrate modern image processing tools.


> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data preparation

The example dataset can be downloaded from here [Electrophoresis gel dataset](https://github.com/galaxyproject/training-material/tree/main/topics/imaging/images/electrophoresis-gel-bands-image-analysis/Image.jpg).

![Electrophoresis Gel](../../images/electrophoresis-gel-bands-image-analysis/Image.jpg)

This image is from {% cite Ohgane2019 %}.

> <hands-on-title>Download and upload the image</hands-on-title>
>
> 1. Create a new history in Galaxy.  
> 2. Download the test image using the link above.  
> 3. Click the **{% icon galaxy-upload %} Upload** to add the image file into your galaxy history.  
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 4. Rename {% icon galaxy-pencil %} the dataset to `Electrophoresis-Gel.jpg`.
>
{: .hands_on}

Before we dive into the quantification steps let's convert our JPG image file into the TIFF image format. This format is preferred for analysis and quantification as it preserves the image data and metadata. 

**Note:** Skip this step if the image is already in TIFF format.

## Convert the image format

> <hands-on-title>Convert JPG to TIFF</hands-on-title>
>
> 1. {% tool [Convert image format](toolshed.g2.bx.psu.edu/repos/bgruening/graphicsmagick_image_convert/graphicsmagick_image_convert/1.3.45+galaxy0) %}  
>    - *"Image to convert"*: `Electrophoresis-Gel.jpg`  
>    - *"Output format"*: `TIFF`
>
>   > <tip-title> Why TIFF? </tip-title>
>   >
>   > TIFF files preserve full image quality, avoid compression artifacts, and are compatible with downstream tools that require uncompressed data.
>   >
>   {: .tip}
>
{: .hands_on}

## Obtain image metadata

To convert coordinates into a label map, we first need the image dimensions (width and height). In this step, we'll retrieve that information.

> <hands-on-title>Identify image dimensions</hands-on-title>
>
> 1. {% tool [Show image info](toolshed.g2.bx.psu.edu/repos/imgteam/image_info/ip_imageinfo/5.7.1+galaxy1) %}  
>    - {% icon param-file %} *"Image Input"*: Select your uploaded `Electrophoresis-Gel.tiff`
>
{: .hands_on}

The output of the tool can be visualize by clicking the {% icon galaxy-eye %} (eye) icon (**View**). See below:

<div style="text-align: center;">
  <pre style="display: inline-block; text-align: left; background: #f4f4f4; padding: 1em; border-radius: 8px; font-size: 14px; line-height: 1.4;">
Checking file format [Tagged Image File Format]
Initializing reader
TiffDelegateReader initializing /data/dnb11/galaxy_db/files/0/6/6/dataset_06685fae-ac3e-48b9-a460-d0ad39ea07a6.dat
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
Initialization took 0.569s

Reading core metadata
filename = /data/dnb11/galaxy_db/files/0/6/6/dataset_06685fae-ac3e-48b9-a460-d0ad39ea07a6.dat
Series count = 1
Series #0 :
	Image count = 1
	RGB = true (3) 
	Interleaved = false
	Indexed = false (false color)
	Width = 1278
	Height = 250
	SizeZ = 1
	SizeT = 1
	SizeC = 3 (effectively 1)
	Thumbnail size = 128 x 25
	Endianness = intel (little)
	Dimension order = XYCZT (certain)
	Pixel type = uint8
	Valid bits per pixel = 8
	Metadata complete = true
	Thumbnail series = false
	-----
	Plane #0 <=> Z 0, C 0, T 0

Reading global metadata
BitsPerSample: 8
Compression: JPEG
Document Name: temp.tiff
ImageLength: 250
ImageWidth: 1278
MetaDataPhotometricInterpretation: RGB
MetaMorph: no
NumberOfChannels: 3
PageNumber: 0 1
PhotometricInterpretation: YCbCr
PlanarConfiguration: Chunky
ReferenceBlackWhite: 0
SampleFormat: unsigned integer
SamplesPerPixel: 3
Software: GraphicsMagick 1.3.45 2024-08-27 Q8 http://www.GraphicsMagick.org/

Reading metadata
  </pre>
</div>


## Split image into separate channels

TIFF images can store multiple channels—such as different colors or fluorescence signals—in a single file. In this step, we will separate these channels by splitting the image along the channel axis (C-axis). This results in a list of single-channel images, which are required for the next processing steps.

> <hands-on-title>Split image into single channels</hands-on-title>
> 1. {% tool [Split image along axes](toolshed.g2.bx.psu.edu/repos/imgteam/split_image/ip_split_image/2.2.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Image to split"*: Select your uploaded `Electrophoresis-Gel.tiff` file
>    - *"Axis to split along"*: Select `C-axis (split the channels of an image or image sequence)` from the dropdown menu
>
{: .hands_on}

> <comment-title>Use a single-channel image for downstream compatibility and annotation</comment-title>
> While QuPath can handle multi-channel images, our other tools cannot. Therefore, after splitting the image, we will use the {% tool [Extract Dataset](__EXTRACT_DATASET__) %} tool to select one of the single-channel images. This image should represent the intensity channel we want to measure, and will be used for QuPath regions of interest (ROI) annotation.
>
{: .comment}

> <hands-on-title>Extract the TIFF file from collection</hands-on-title>
>
> 1. {% tool [Extract Dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-collection %} *"Input Collection"*: Select the collection output from the {% tool [Split image along axes](toolshed.g2.bx.psu.edu/repos/imgteam/split_image/ip_split_image/2.2.3+galaxy1) %} tool
>    - *"Which part of the Collection?"*: Choose the TIFF file that represent the channel of your interest. In our dataset example, we extract 1.tiff
>
{: .hands_on }

# Annotate and process regions of interest (ROIs)

To identify and quantify specific regions in the image, such as electrophoresis bands, we need to define them manually. This is done using the interactive tool **QuPath**, which allows you to draw rectangles over each band of interest and assign them a unique label to annotate the ROIs. The result is a **GeoJSON** file containing the coordinates of these regions.

This GeoJSON file can then be processed in Galaxy to generate a label map for image quantification.

## Prepare the ROI file with QuPath

In this step, you will open the image in QuPath, annotate the ROIs using rectangles, and export the annotations as a GeoJSON file. Each band should be marked with a rectangle and given a unique name (e.g., "1", "2", etc.). 

> <hands-on-title>Select ROIs and export GeoJSON file</hands-on-title>
> 
> 1. Open the image in **QuPath** using our interactive tool {% tool [QuPath](interactive_tool_qupath) %} with the following parameters:
>    - {% icon param-collection %} *"Input file in TIFF format"*: Select the `Electrophoresis-Gel.tiff` dataset after channel splitting.
>    
>    When the image opens in QuPath, set the image type to `Other`:  
>    ![Select image type](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-1.png)
>
> 2. Select the **Rectangle tool** from the toolbar:  
>    ![Rectangle tool selection](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-2.png)
>
> 3. Draw a rectangle around the first band of interest:  
>    ![First band selection](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-3.png)
>
> 4. Open the **Annotations** panel:  
>    ![Annotations panel](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-4.png)
>
> 5. Double-click the rectangle and set its type to `Other`:  
>    ![Set annotation type](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-5.png)
>
> 6. Duplicate the annotation to create additional rectangles:  
>    - Go to **Object → Annotations**  
>    - Select **Duplicate selected annotations**  
>    ![Duplicate annotation](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-6.png)
>
> 7. Verify the new rectangle appears in the Annotations panel:  
>    ![Second annotation](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-7.png)
>
> 8. Select the new rectangle and reposition it over the next band:  
>    ![Select second annotation](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-8.png)  
>    ![Position second annotation](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-9.png)
>
> 9. Repeat steps 6–8 for all bands you want to annotate.
>
> 10. Set a unique label for each annotation:  
>     - Right-click each rectangle, choose **Set parameters**, and enter a label (e.g., `"1"`, `"2"`)  
>     ![Name first annotation](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-10.png)
>     ![Name first annotation](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-11.png)
>     ![Name first annotation](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-12.png)
>     ![Name second annotation](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-13.png)
>
> 11. Confirm that all rectangles are labeled:  
>     ![Labeled annotations](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-14.png)
>
> 12. Select all labeled annotations:  
>     ![Select all annotations](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-15.png)
>
> 13. Export the annotations as a GeoJSON file:  
>     - Click **File → Export objects as GeoJSON**  
>     - Choose **Export as FeatureCollection**  
>     - Click **Save**  
>     ![Export process](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-16.png)  
>     ![Export process](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-17.png)  
>     ![Export process](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-18.png)
>
> 14. Exit QuPath:  
>     - Click **File → Quit**  
>     - Save changes if prompted  
>     ![Quit QuPath](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-19.png)  
>     ![Quit QuPath](../../images/electrophoresis-gel-bands-image-analysis/QuPath-protocol-20.png)
>
> 15. The exported file will appear in your Galaxy history as part of a collection within 2–3 minutes.
>
{: .hands_on}

## Extract the GeoJSON file

QuPath exports the annotations as a Galaxy collection. In this step, we extract the GeoJSON file to use it in the next stage of the analysis.

> <hands-on-title>Extract the GeoJSON file</hands-on-title>
>
> 1. {% tool [Extract Dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-collection %} *"Input Collection"*: Select the `QuPath outputs` collection
>    - *"Which part of the Collection?"*: Choose the GeoJSON file
>
> 2. {% icon galaxy-pencil %} **Rename** the extracted file to `Bands.geojson`.
>
{: .hands_on}

## Convert ROIs to label map

Now that we have the coordinate information in GeoJSON format, we can convert it into a label map. This label map is an image where each ROI is assigned a unique label, allowing it to be quantified or visualized in downstream steps. In this step, you will also need the information obtained earlier with the {% tool [Show image info](toolshed.g2.bx.psu.edu/repos/imgteam/image_info/ip_imageinfo/5.7.1+galaxy1) %} tool about the width and height of the image. 

> <hands-on-title>Convert coordinates to label map</hands-on-title>
>
> 1. {% tool [Convert coordinates to label map](toolshed.g2.bx.psu.edu/repos/imgteam/points2labelimage/ip_points_to_label/0.4.1+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"List of points in tabular or GeoJSON format"*: Select your `Bands.geojson` file
>    - *"Width of output image"*: `1278` (Use the width from the **Show image info** result)
>    - *"Height of output image"*: `250` (Use the height from the **Show image info** result)
>    - *"Tabular list of points has header?"*: `Yes`
>
{: .hands_on}

# Extract band intensity features

> <hands-on-title>Measure band intensity</hands-on-title>
>
> 1. {% tool [Extract image features](toolshed.g2.bx.psu.edu/repos/imgteam/2d_feature_extraction/ip_2d_feature_extraction/0.18.1+galaxy0) %}  
>    - *"Label map"*: Output from previous step  
>    - *"Use the intensity image"*: `Use intensity image`  
>    - *"Intensity image"*: `Electrophoresis-Gel.tiff`  
>    - *"Features to compute"*:
>      - Select: "Label from label map"
>      - Select: "Mean Intensity"
>
{: .hands_on}

## Inspecting results

The table result can be inspected by clicking on the {% icon galaxy-eye %} (eye) icon (**View**) of the tool output and used for further analysis.

# Conclusion

In this tutorial, we explored a complete workflow for quantifying electrophoresis gel bands by:

- Uploading and converting the gel image into a suitable format  
- Interactively selecting bands as regions of interest (ROIs) using QuPath  
- Exporting ROI coordinates and converting them into a label map in Galaxy  
- Measuring the intensity of each band and extracting the results in tabular format

By combining **QuPath** for annotation and **Galaxy** for analysis, this approach offers a transparent, reproducible, and user-friendly method for gel band quantification, suitable for a variety of molecular biology applications.

tools_json:
["Convert image format", "toolshed.g2.bx.psu.edu/repos/bgruening/graphicsmagick_image_convert/graphicsmagick_image_convert/1.3.45+galaxy0", "Show image info", "toolshed.g2.bx.psu.edu/repos/imgteam/image_info/ip_imageinfo/5.7.1+galaxy1", "Split image along axes", "toolshed.g2.bx.psu.edu/repos/imgteam/split_image/ip_split_image/2.2.3+galaxy1", "Extract Dataset", "__EXTRACT_DATASET__", "QuPath", "interactive_tool_qupath", "Convert coordinates to label map", "toolshed.g2.bx.psu.edu/repos/imgteam/points2labelimage/ip_points_to_label/0.4.1+galaxy1", "Extract image features", "toolshed.g2.bx.psu.edu/repos/imgteam/2d_feature_extraction/ip_2d_feature_extraction/0.18.1+galaxy0"]


topic: imaging
tutorial: iscc-suite
title: Content Tracking and Verification in Galaxy Workflows with ISCC-SUM
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["Data management", "Data handling", "Data sharing", "Validation"]
content_hash: e2e5f194e1d2b437310040aed4f797b8
last_modified: 2026-05-27T09:01:49.079960
zenodo_link: https://zenodo.org/records/18611428
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/imaging/tutorials/iscc-suite/tutorial.html

content:
# File content and integrity validation with International Standard Content Code (ISCC)

In scientific workflows, ensuring data integrity and tracking modifications of the data content is crucial for reproducibility. Traditional checksums (like MD5 or SHA) can verify if files are identical, but they cannot detect similar content or survive format conversions.

The **International Standard Content Code (ISCC)** is a content-derived identifier that provides both:
- **Identity verification**: Checksum functionality to verify exact file matches
- **Similarity detection**: Ability to detect similar content even across different file formats

The **Galaxy ISCC-suite** allows you to integrate content tracking into any Galaxy workflow, providing quality control and provenance tracking for your data analysis pipelines.

## ISCC Code structure

An ISCC-SUM code is a 55-character identifier with two main components, which are combined into one code:

- **Data-Code**: Content-based hash that allows similarity comparison
- **Instance-Code**: A fast checksum or cryptographic hash

The Instance-Code uses BLAKE3 hashing, truncated to 64 bits by default. For applications requiring cryptographic-strength verification, ISCC-SUM can output the full 256-bit hash.

For example, the ISCC hash for this file [`example_image.tiff`](https://zenodo.org/records/18611428/files/example_image.tiff) is:
```
ISCC:K4AI45QGX6J3LYNEHONZMQT2GJ6YPJDS74EIC2YMSORF4S5H5SKHQQI
```
This ISCC-CODE is built from two individual components — a **Data-Code** and an **Instance-Code**:
```
    ISCC:GADY45QGX6J3LYNEHONZMQT2GJ6YPZ4BXJQ7ZHWZ7EHRLKANDCSACWI  (Data-Code)
    ISCC:IAD2I4X7BCAWWDETUJPEXJ7MSR4EDZCTTBQS6LQQDWDHS6T4KDDPZ5A  (Instance-Code)
```
You might notice that the combined ISCC-CODE (`K4AI...`) looks completely different from these two components. That is expected: the ISCC algorithm takes shortened versions of both hashes, packs them together, and encodes the result as a new string. Think of it like combining two barcodes into a single, shorter barcode — the information is still there, just represented differently.

Files with similar content will have similar Data-Code components, but their Instance-Code will be different. Hence the Instance-Code allows to verify file integrity.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Prepare your data

For this tutorial, we'll use a simple dataset with microscope images that demonstrate different use cases. However, the ISCC SUM tools can ben used for any type of digital content.

## Get the data

> <hands-on-title>Data Upload</hands-on-title>
>
> 1. Create a new history for this tutorial in Galaxy.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Download the following image and import it into your Galaxy history.
>    - [`example_image.tiff`](https://zenodo.org/records/18611428/files/example_image.tiff)
>    - [`example_image2.tiff`](https://zenodo.org/records/18611428/files/example_image2.tiff)
>    - [`example_image3.tiff`](https://zenodo.org/records/18611428/files/example_image3.tiff)
>    - [`example_thresholded1.tiff`](https://zenodo.org/records/18611428/files/example_thresholded1.tiff)
>    
>    If you are importing the image via URL:
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
> 
{: .hands_on}

# Generate ISCC codes

The first step is generating ISCC codes for your input files. This creates a content fingerprint that can be used for later content-based identification of the file (e.g., within your workflow or within a publication).

> <hands-on-title>Generate ISCC codes for input files</hands-on-title>
>
> 1. {% tool [Generate ISCC-CODE](toolshed.g2.bx.psu.edu/repos/imgteam/iscc_sum/iscc_sum/0.1.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input File"*: Select the first example image (`example_image.tiff`.)
>  
>    Run the tool. This will generate a 55-character ISCC code for the file.
>
> 2. Expand the history item for the output of the {% tool [Generate ISCC-CODE](toolshed.g2.bx.psu.edu/repos/imgteam/iscc_sum/iscc_sum/0.1.0+galaxy1) %} tool.
>
> 3. Click on the {% icon details %} icon.
>
> 4. Scroll down to the **Job Outputs** section. Select the Dataset. You should see a single line containing the ISCC code in the output. For the first example image the code is expected to be:
> ```
> K4AI45QGX6J3LYNEHONZMQT2GJ6YPJDS74EIC2YMSORF4S5H5SKHQQI 
> ```
>
> 5. Repeat for the other example images to generate another ISCC code for comparison.
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Will the same file always generate the same ISCC code?
>
> > <solution-title></solution-title>
> >
> > 1. Yes! The same file will always generate the identical ISCC code, making it suitable for integrity verification.
> >
> {: .solution}
>
{: .question}

# Verify file integrity

During workflow execution, you may want to verify that intermediate files match expected content. The **Verify ISCC hash** tool allows you to check if a file matches a known ISCC code.

## Manual verification

> <hands-on-title>Verify a file against its ISCC code</hands-on-title>
>
> 1. Run {% tool [Verify ISCC-CODE](toolshed.g2.bx.psu.edu/repos/imgteam/iscc_sum_verify/iscc_sum_verify/0.1.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Dataset to verify"*: Select the first example image
>    - *"Expected ISCC-CODE"*: 
>        - *"Expected ISCC code"*: Paste the ISCC code you generated in the previous step
>
> 2. Expand the history item for the output of the {% tool [Verify ISCC-CODE](toolshed.g2.bx.psu.edu/repos/imgteam/iscc_sum_verify/iscc_sum_verify/0.1.0+galaxy1) %} tool.
>
> 3. Click on the {% icon details %} icon.
>
> 4. Scroll down to the **Job Output** section. Select the output to expand it, this will show you verification report, that looks like this:
> ```
> OK - ISCC-CODEs match
> Expected:  K4AI45QGX6J3LYNEHONZMQT2GJ6YPJDS74EIC2YMSORF4S5H5SKHQQI
> Generated: K4AI45QGX6J3LYNEHONZMQT2GJ6YPJDS74EIC2YMSORF4S5H5SKHQQI
> ```
>    The report shows:
>    - Status: OK (match) or FAILED (mismatch)
>    - Expected ISCC code
>    - Generated ISCC code
{: .hands_on}

## Workflow integration

A powerful use case is integrating ISCC verification directly into your workflows. Here we'll build a simple verification workflow step by step.

### Step 1 - Define the workflow inputs

To make the workflow reusable, we need to define two inputs: the image to verify and a file containing the expected ISCC code.

> <hands-on-title>Create the workflow inputs</hands-on-title>
>
> 1. Create a new workflow in the workflow editor.
>
>    {% snippet faqs/galaxy/workflows_create_new.md %}
>
> 2. Select {% icon tool %} **Input dataset** from the list of tools:
>    - {% icon param-file %} **1: Input Dataset** appears in your workflow.
>    Change the "Label" of this input to *Input image*.
> 3. Add another {% icon tool %} **Input dataset**:
>    - {% icon param-file %} **2: Input Dataset** appears in your workflow.
>    Change the "Label" of this input to *Expected ISCC code file*.
{: .hands_on}

### Step 2 - Parse the expected ISCC code

The *Generate ISCC-CODE* tool outputs the ISCC code as a text file, but the *Verify ISCC-CODE* tool expects the code as a parameter input. We use *Parse parameter value* to bridge this gap.

> <hands-on-title>Add the parameter parsing step</hands-on-title>
>
> 1. While in the workflow editor, add {% icon tool %} **Parse parameter value** from the list of tools:
>    - Connect the output of {% icon param-file %} **2: Expected ISCC code file** to the "Input file containing parameter to parse" input of {% icon tool %} **3: Parse parameter value**.
{: .hands_on}

### Step 3 - Add the verification step

Now we add the ISCC verification tool and connect all the inputs.

> <hands-on-title>Add the ISCC verification step</hands-on-title>
>
> 1. Add {% tool [Verify ISCC-CODE](toolshed.g2.bx.psu.edu/repos/imgteam/iscc_sum_verify/iscc_sum_verify/0.1.0+galaxy1) %} from the list of tools:
>    - Connect the output of {% icon param-file %} **1: Input image** to the "Dataset to verify" input of {% icon tool %} **4: Verify ISCC-CODE**.
>    - Connect the output of {% icon tool %} **3: Parse parameter value** to the "File containing expected ISCC code" input of {% icon tool %} **4: Verify ISCC-CODE**.
{: .hands_on}

The completed workflow should look like this:

![Workflow diagram showing ISCC verification integrated into a simple workflow](../../images/iscc-suite/verify_wf1.png)

### Step 4 - Run the workflow

> <hands-on-title>Run the verification workflow</hands-on-title>
>
> 1. Run the workflow with the following inputs:
>    - **Input image**: Select the first example image (`example_image.tiff`)
>    - **Expected ISCC code file**: Select the ISCC code output generated in a previous step
>
> 2. Wait for the workflow to complete. Subsequently , expand the history item for the output of the {% tool [Verify ISCC-CODE](toolshed.g2.bx.psu.edu/repos/imgteam/iscc_sum_verify/iscc_sum_verify/0.1.0+galaxy1) %} tool.
>
> 3. Click on the {% icon details %} icon.
>
> 4. Scroll down to the **Job Output** section and select the output dataset. You should see a verification report similar to the one described in the manual verification section above.
>
{: .hands_on}

When placing this verification step in a full workflow, it can help validate that your processing didn't unexpectedly alter the content.

## Image analysis workflow integration

This can be applied in an image analysis workflow to verify an image processing tool provides the expected reproducible output. In the example files we shared, a thresholded image `example_thresholded1.tiff` can be found. We will use it to verify whether the Otsu threshold result of this image can be reproduced.

> <hands-on-title>Image analysis verification workflow</hands-on-title>
>
> 1. Import and run the ready-to-use workflow:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/imaging/tutorials/iscc-suite/workflows/ISCC---image-analysis-workflow-example.ga" title="ISCC Image Analysis Verification" %}
>
> 2. Provide the inputs:
>    - **Original image**: Select `example_image.tiff` - the image to be processed
>    - **Segmented image**: Select `example_thresholded1.tiff` - the reference segmentation to compare against
>
> 3. Run the workflow.
>
> 4. Expand the history item for the output of the {% tool [Verify ISCC-CODE](toolshed.g2.bx.psu.edu/repos/imgteam/iscc_sum_verify/iscc_sum_verify/0.1.0+galaxy1) %} tool.
>
> 5. Click on the {% icon details %} icon.
>
> 6. Scroll down to the **Job Information** section to view the "Tool Standard Output" log. You should see a verification report similar to the one described in the manual verification section above.
>
> The workflow performs Otsu thresholding on the original image and verifies whether the result matches the expected segmentation using ISCC codes. This allows you to verify whether the thresholding method is working as expected and the algorithm has not been altered (e.g., in a new version).
>
> ![Workflow diagram showing ISCC verification in an image analysis pipeline](../../images/iscc-suite/verify_wf2.png)
{: .hands_on}

> <comment-title>When to use verification</comment-title>
>
> Verification is particularly useful:
> - After file transfers or storage operations
> - To confirm correct input files in complex workflows
> - As quality control checkpoints in processing pipelines
> - To detect unintended data modifications
{: .comment}

# Detect similar content

One of ISCC's unique features is detecting similar content, even across different formats. This is useful for finding duplicates, tracking content transformations, or identifying related files.

## Compare two files

> <hands-on-title>Compare two files for similarity</hands-on-title>
>
> 1. {% tool [Find datasets with similar ISCC-CODEs](toolshed.g2.bx.psu.edu/repos/imgteam/iscc_sum_similarity/iscc_sum_similarity/0.1.0+galaxy1) %} with the following parameters:
>    - *"Input type"*: `Datasets to compare`
>        - {% icon param-file %} Select multiple datasets (or a collection, see below)
>    - *"Similarity threshold (Hamming distance)"*: `12` (default)
>
> 2. The tool will create tabular output which indicates which datasets are similar.
>    The table will list all the files that has been set as input. For the files which have a similar file, below the set threshold, the similar file is listed.
{: .hands_on}

## Find similar files in collections

When working with a collection of files, you can identify all similar items. This is particularly useful when you have large datasets and want to find duplicates or track how processing affects content similarity.

> <hands-on-title>Find similar files in a collection</hands-on-title>
>
> 1. Create a dataset collection with your test images
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
>    Include all images from the tutorial: `example_image.tiff`, `example_image2.tiff`, `example_image3.tiff` and `example_thresholded1.tiff`
>
> 2. {% tool [Find datasets with similar ISCC-CODEs](toolshed.g2.bx.psu.edu/repos/imgteam/iscc_sum_similarity/iscc_sum_similarity/0.1.0+galaxy1) %} with the following parameters:
>    - *"Input type"*: `Datasets to compare`
>        - {% icon param-file %} Select your collection
>    - *"Similarity threshold (Hamming distance)"*: `12` (default)
>
> 3. Examine the output table. Each row represents a file from your collection. The columns show:
>    - The filename
>    - Its ISCC code
>    - Any similar files found (with their similarity score)
>
>    Files that share content (like `example_image.tiff` and `example_image3.tiff`, which is slightly modified ) will be grouped together, although their ISCC-SUM codes are different.
>
>    ![ISCC similarity table](../../images/iscc-suite/similarity_table.png)
{: .hands_on}

> <tip-title>Understanding the Hamming distance threshold</tip-title>
>
> The Hamming distance counts how many bits differ between two Data-Codes. For the default 64-bit Data-Code, this ranges from 0 (identical) to 64 (completely different). The tool uses a default threshold of 12, meaning files with a distance of 12 or less are considered similar.
>
> This threshold is a practical starting point — adjust it based on your use case: lower values for stricter matching, higher values to catch more distant similarities. Keep in mind that Data-Code similarity reflects byte-level similarity, not semantic content. Whether a given distance is scientifically meaningful depends on your domain and data.
{: .tip}


> <question-title></question-title>
>
> 1. Looking at the similarity results table, why do `example_image.tiff` and `example_image3.tiff` show a match while `example_thresholded1.tiff` does not?
>
> 2. What does a distance value of -1 indicate?
>
> > <solution-title></solution-title>
> >
> > 1. `example_image.tiff` and `example_image3.tiff` contain similar visual content, resulting in a Hamming distance of 5, which is below the threshold of 12. The thresholded image has undergone significant processing (binarization), changing its content substantially so it no longer matches the original within the similarity threshold.
> >
> > 2. A distance of -1 indicates that no similar file was found within the specified threshold. The file is unique compared to all other files in the collection.
> >
> {: .solution}
>
{: .question}


# Practical use cases

## Use case 1: Quality control in image analysis pipelines

When processing large microscopy datasets:

- Generate ISCC codes for raw images upon acquisition
- Detect if processing steps produce consistent outputs across batch runs
- Identify accidentally duplicated samples before analysis

## Use case 2: Data deduplication and organization

When managing growing image repositories:

- Scan collections to find duplicate uploads that waste storage
- Identify images that are near-duplicates (e.g., same sample, different export settings)
- Group related experimental replicates automatically

## Use case 3: Reproducibility and data sharing

When publishing or sharing datasets:

- Include ISCC codes in data publications for recipient verification
- Document the exact input files used in published analyses
- Enable collaborators to confirm they have identical source data

# Conclusion

In this tutorial, you learned to use the Galaxy ISCC-suite for content tracking and verification:

- **Generate ISCC-CODE**: Creates content-based identifiers for any file
- **Verify ISCC-CODE**: Confirms files match expected content at workflow checkpoints  
- **Find datasets with similar ISCC-CODEs**: Detects related or duplicate content in collections

These tools help you maintain data integrity throughout your analysis workflows, from initial data import through to final results.

# References

- ISCC - International Standard Content Code: https://iscc.codes/
- ISCC-SUM Implementation: https://github.com/iscc/iscc-sum
- ISCC-SUM Quick Start: https://sum.iscc.codes/quickstart/

tools_json:
["Generate ISCC-CODE", "toolshed.g2.bx.psu.edu/repos/imgteam/iscc_sum/iscc_sum/0.1.0+galaxy1", "Verify ISCC-CODE", "toolshed.g2.bx.psu.edu/repos/imgteam/iscc_sum_verify/iscc_sum_verify/0.1.0+galaxy1", "Find datasets with similar ISCC-CODEs", "toolshed.g2.bx.psu.edu/repos/imgteam/iscc_sum_similarity/iscc_sum_similarity/0.1.0+galaxy1"]


topic: variant-analysis
tutorial: microbial-variants
title: Microbial Variant Calling
difficulty: intermediate
time_estimation: 45m
tools_json: []
requirements_json: []
tags_json: ["prokaryote", "microgalaxy", "gmod", "jbrowse1"]
content_hash: c256400de80060bca13914bc4db6a57e
last_modified: 2026-05-27T09:01:50.017151
zenodo_link: https://doi.org/10.5281/zenodo.582600
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/microbial-variants/tutorial.html

content:
Variant calling is the process of identifying differences between two genome samples. Usually differences are limited to single nucleotide polymorphisms (SNPs) and small insertions and deletions (indels). Larger structural variation such as inversions, duplications and large deletions are not typically covered by “variant calling”.

Imagine that you have been asked to find the differences between a sample that has been sequenced and a known genome. For example: You have a new sample from a patient and you want to see if it has any differences from a well known reference genome of the same species. Typically, you would have a couple of fastq read files sent back to you from the sequencing provider and either an annotated or non annotated reference genome.

In this tutorial, we will use the tool “Snippy” (see [author development repository](https://github.com/tseemann/snippy)) to find high confidence differences (indels or SNPs) between our known genome and our reads. Snippy uses one tool to align the reads to the reference genome, and another tool to decide (“call”) if any of the resulting discrepancies are real variants or technical artifacts that can be ignored. Finally, Snippy uses another tool to check what effect these differences have on the predicted genes - truncation, frame shift or if the changes are synonymous.

For the read alignment (read mapping) step, Snippy uses BWA MEM with a custom set of settings which are very suitable to aligning reads for microbial type data. For the variant calling step, Snippy uses Freebayes with a custom set of settings. snpeff is then used to describe what the predicted changes do in terms of the genes themselves.

The Galaxy wrapper for Snippy has the ability to change some of the underlying tool settings in the advanced section but it is not recommended.

Read more about SNP calling [at Wikipedia](https://en.wikipedia.org/wiki/SNV_calling_from_NGS_data).

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get the data

The data for today is a subset of a real dataset from a *Staphylococcus aureus* bacteria.
We have a closed genome sequence and an annotation for our "wildtype" strain.
We have used a whole genome shotgun approach to produce a set of short sequence reads on an Illumina DNA sequencing instrument for our mutant strain.

- The reads are paired-end
- Each read is on average 150 bases
- The reads would cover the original wildtype genome to a depth of 19x

The files we will be using are:

- `mutant_R1.fastq` & `mutant_R2.fastq` - the read files in fastq format.
- `wildtype.fna` - The sequence of the reference strain in fasta format.
- `wildtype.gbk` - The reference strain with gene and other annotations in genbank format.
- `wildtype.gff` - The reference strain with gene and other annotations in gff3 format.

[This data is available at Zenodo](https://doi.org/10.5281/zenodo.582600).

> <hands-on-title>Get the data</hands-on-title>
>
> 1. Import all of the following files into a new history:
>     - [mutant_R1.fastq](https://zenodo.org/record/582600/files/mutant_R1.fastq)
>     - [mutant_R2.fastq](https://zenodo.org/record/582600/files/mutant_R2.fastq)
>     - [wildtype.fna](https://zenodo.org/record/582600/files/wildtype.fna)
>     - [wildtype.gbk](https://zenodo.org/record/582600/files/wildtype.gbk)
>     - [wildtype.gff](https://zenodo.org/record/582600/files/wildtype.gff)
>
>    ```
>    https://zenodo.org/record/582600/files/mutant_R1.fastq
>    https://zenodo.org/record/582600/files/mutant_R2.fastq
>    https://zenodo.org/record/582600/files/wildtype.fna
>    https://zenodo.org/record/582600/files/wildtype.gbk
>    https://zenodo.org/record/582600/files/wildtype.gff
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}

# Find variants with Snippy

We will now run the Snippy tool on our reads, comparing it to the reference.

Snippy is a tool for rapid bacterial SNP calling and core genome alignments. Snippy finds SNPs between a haploid reference genome and your NGS sequence reads. It will find both substitutions (snps) and insertions/deletions (indels).

If we give Snippy an annotated reference, it will silently run a tool called SnpEff which will figure out the effect of any changes on the genes and other features. If we just give Snippy the reference sequence alone without the annotations, it will not run SnpEff.

We have an annotated reference and so will use it in this case.

> <hands-on-title>Run Snippy</hands-on-title>
>
> 1. **Snippy** {% icon tool %} with the following parameters
>   - "Reference File" to the `wildtype.gbk` file (if the genbank file is not selectable, make sure to change its datatype to 'genbank')
>   - "Single or Paired-end reads" to `Paired`
>   - "Select first set of reads" to `mutant_R1.fastq`
>   - "Select second set of reads" to `mutant_R2.fastq`
>   - Select all outputs
>
{: .hands_on}

# Examine Snippy output

Snippy has taken the reads, mapped them against the reference using BWA MEM, looked through the resulting BAM file and found differences using some fancy Bayesian statistics (Freebayes), filtered the differences for sensibility and finally checked what effect these differences will have on the predicted genes and other features in the genome.

It produces quite a bit of output, there can be up to 10 output files.

Filename | Description
--- | ---
snps vcf file | The final annotated variants in VCF format
snps gff file | The variants in GFF3 format
snps table | A simple tab-separated summary of all the variants
snps summary  | A summary of the SNPs called
log file  | A log file with the commands run and their outputs
aligned fasta | A version of the reference but with - at position with depth=0 and N for 0 < depth < --mincov **(does not have variants)**
consensus fasta | A version of the reference genome with all variants instantiated
mapping depth | A table of the mapping depth
mapped reads bam  | A BAM file containing all of the mapped reads
outdir  | A tarball of the Snippy output directory for input into Snippy-core if required

We will now have a look at the contents of the SNP table file (`snippy on data XX, data XX and data XX table`):

```
1   2   3   4   5   6   7   8   9   10  11  12  13  14
CHROM   POS TYPE    REF ALT EVIDENCE    FTYPE   STRAND  NT_POS  AA_POS  EFFECT  LOCUS_TAG   GENE    PRODUCT
Wildtype    24388   snp A   G   G:22 A:0    CDS +   1/702   1/233   initiator_codon_variant c.1A>G p.Met1?  WILD_00022  walR    Transcriptional regulatory protein WalR
Wildtype    29479   snp T   G   G:21 T:0    CDS +   39/792  13/263  synonymous_variant c.39T>G p.Gly13Gly   WILD_00026  yycJ    Putative metallo-hydrolase YycJ
Wildtype    47299   snp T   A   A:24 T:0    CDS +   54/1758 18/585  stop_gained c.54T>A p.Cys18*    WILD_00043  mecR1   Methicillin resistance mecR1 protein
Wildtype    102969  snp G   C   C:16 G:0    CDS -   87/1281 29/426  synonymous_variant c.87C>G p.Gly29Gly   WILD_00093  spa Immunoglobulin G-binding protein A
Wildtype    103048  snp T   A   A:20 T:0    CDS -   8/1281  3/426   missense_variant c.8A>T p.Lys3Met   WILD_00093  spa Immunoglobulin G-binding protein A
Wildtype    103379  del GAA GA  GA:11 GAA:0
Wildtype    106602  snp T   G   G:21 T:0    CDS -   993/993 331/330 stop_lost&splice_region_variant c.993A>C p.Ter331Tyrext*?   WILD_00097  yfiY    putative siderophore-binding lipoprotein YfiY
Wildtype    109833  snp T   A   A:16 T:0    CDS +   1/1755  1/584   initiator_codon_variant c.1T>A p.Leu1?  WILD_00100  iucC_1  Aerobactin synthase
Wildtype    114540  del ATT AT  AT:25 ATT:0 CDS +   1717/1737   573/578 frameshift_variant c.1717delT p.Cys573fs    WILD_00102  iucA    N(2)-citryl-N(6)-acetyl-N(6)-hydroxylysine synthase
Wildtype    129881  mnp GT  AA  AA:18 GT:0  CDS +   55/708  19/235  missense_variant c.55_56delGTinsAA p.Val19Asn   WILD_00117  deoD    Purine nucleoside phosphorylase DeoD-type
Wildtype    138877  snp G   C   C:14 G:0    CDS +   1119/1545   373/514 missense_variant c.1119G>C p.Trp373Cys  WILD_00125      hypothetical protein
Wildtype    138920  snp A   G   G:10 A:0    CDS +   1162/1545   388/514 missense_variant c.1162A>G p.Lys388Glu  WILD_00125      hypothetical protein
Wildtype    160547  del GTC GC  GC:18 GTC:0
Wildtype    160552  del CTA CA  CA:20 CTA:0
Wildtype    190866  del GTT GT  GT:18 GTT:0 CDS -   28/1356 10/451  frameshift_variant c.28delA p.Asn10fs   WILD_00166  brnQ    Branched-chain amino acid transport system 2 carrier protein
```

> <question-title></question-title>
>
> 1. Which types of variants have been found?
> 2. What is the third variant called?
> 3. What is the product of the mutation?
> 4. What might be the result of such a mutation?
>
> > <solution-title></solution-title>
> >  1. In the 3rd column, you have "snp" for SNP, "del" for deletion, "mnp" for
> >  2. This is a T→A mutation, causing a stop codon
> >  3. On the 14th column, we see that The product of this gene is a methicillin resistance protein. Methicillin is an antibiotic.
> >  4. This will cause a truncation in the Methicillin gene and a loss of resistance in the organism.
> {: .solution}
{: .question}

# View Snippy output in JBrowse

We could go through all of the variants like this and read them out of a text table, but this is onerous and doesn't really give the context of the changes very well. It would be much nicer to have a visualisation of the SNPs and the other relevant data. In Galaxy we can use a tool called JBrowse.

> <hands-on-title>Run JBrowse</hands-on-title>
>
> 1. **JBrowse** {% icon tool %} with the following parameters
>    - "Reference genome to display" to `Use a genome from history`
>    - "Select the reference genome" to `wildtype.fna`
>
>       This sequence will be the reference against which annotations are displayed
>
>    - *"Output JBrowse"*: `Minimal for viewing (Documentation removed)`
>    - "Genetic Code" to `11: The Bacterial, Archaeal and Plant Plastid Code`
>    - "JBrowse-in-Galaxy Action" to `New JBrowse Instance`
>    - "Track Group"
>
>        We will now set up three different tracks - these are datasets displayed underneath the reference sequence (which is displayed as nucleotides in FASTA format). We will choose to display the sequence reads (the .bam file), the variants found by snippy (the .gff file) and the annotated reference genome (the wildtype.gff)
>
>       - **Track 1 - sequence reads**: Click on `Insert Track Group` and fill it with
>           - "Track Category" to `sequence reads`
>           - Click on `Insert Annotation Track` and fill it with
>               - "Track Type" to `BAM Pileups`
>               - "BAM Track Data" to `snippy bam file`
>               - "Autogenerate SNP Track" to `Yes`
>               - "Track Visibility" to `On for new users`
>       - **Track 2 - variants**: Click on `Insert Track Group` and fill it with
>           - "Track Category" to `variants`
>           - Click on `Insert Annotation Track` and fill it with
>               - "Track Type" to `GFF/GFF3/BED Features`
>               - "GFF/GFF3/BED Track Data" to `snippy snps gff file`
>               - "Track Visibility" to `On for new users`
>       - **Track 3 - annotated reference**: Click on `Insert Track Group` and fill it with
>           - "Track Category" to `annotated reference`
>           - Click on `Insert Annotation Track` and fill it with
>               - "Track Type" to `GFF/GFF3/BED Features`
>               - "GFF/GFF3/BED Track Data" to `wildtype.gff`
>               - "JBrowse Track Type [Advanced]" to `Canvas Features`
>               - Click on "JBrowse Styling Options [Advanced]"
>               - "JBrowse style.label" to `product`
>               - "JBrowse style.description" to `product`
>               - "Track Visibility" to `On for new users`
{: .hands_on}

A new dataset will be created in your history, containing the JBrowse interactive visualisation. We will now view its contents and play with it by clicking the {% icon galaxy-eye %} (eye) icon of the `JBrowse on data XX and data XX - Complete` dataset. The JBrowse window will appear in the centre Galaxy panel.

> <hands-on-title>Inspecting the SNPs using JBrowse</hands-on-title>
> 1. Display all the tracks and practice maneuvering around
>    1. Click on the tick boxes on the left to display the tracks
>    1. Zoom out by clicking on the `minus` button to see sequence reads and their coverage (the grey graph)
>    1. Zoom in by clicking on the `plus` button to see
>       - probable real variants (a whole column of SNPs)
>       - probable errors (single one here and there)
>
>    ![JBrowse screenshot](../../images/jbrowse1.png "Screenshot of JBrowse")
>
> 2. Look at the stop SNP
>    1. Type `47299` in the coordinates box
>    2. Click on `Go` to see the position of the SNP discussed above
>
>    ![JBrowse screenshot](../../images/jbrowse2.png "Inspection of the STOP SNP using JBrowse")
>
>     > <question-title></question-title>
>     >
>     > 1. What is the correct codon at this position?
>     > 2. What is the mutation found here?
>     >
>     > > <solution-title></solution-title>
>     > > 1. The correct codon at this position is TGT, coding for the amino acid Cysteine (middle row of the amino acid translations).
>     > > 2. The mutation of T → A turns this triplet into TGA, a stop codon.
>     > {: .solution}
>     {: .question}
>
{: .hands_on}

# Conclusion

By running a tool such as Snippy on your read files and reference genome, we can find where the biologically important changes between genomes of different strains occur and perhaps what they mean to the phenotype.


topic: variant-analysis
tutorial: baculovirus-isolate-variation
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: []
content_hash: bacaa9531a58cfe3777dd321d14b80e8
last_modified: 2026-05-27T09:01:50.014341
workflows_json: []
title:
Deciphering Virus Populations - Single Nucleotide Variants (SNVs) and Specificities in Baculovirus Isolates

url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/baculovirus-isolate-variation/tutorial.html

content:
Baculoviruses of the family *Baculoviridae* ({% cite Harrison2018 %}) are among the most intensively 
studied viruses, not only because of their widespread application in biotechnology as protein expression 
systems, medicine, and as biological insecticides ({% cite RodrguezHernndez2023 %}, 
{% cite Cox2021 %}, {% cite Grzywacz2017 %}, {% cite Eberle2012 %}, {% cite vanOers2011 %}). 
Like all members of the class *Naldaviricetes*, 
baculoviruses have large circular dsDNA genomes that, in the family *Baculoviridae*, 
range from 90 to 180 kbp and can encode up to 180 open reading frames (ORFs). 
A detailed description of the biology of the Baculoviridae family can be found in the
official report of the International Committee on Taxonomy of Viruses (ICTV) ({% cite Harrison2018 %}). A detailed summary can also be found on the [website of the ICTV](https://ictv.global/).

> <comment-title> A Baculovirus Genome </comment-title>
>
> The genome of the isolate Autographa californica multiple nucleopolyhedrovirus isolate C6 (AcMNPV-C6)
> (family *Baculoviridae*, genus *Alphabaculovirus*, species *Alphabaculovirus aucalifornicae*) is one of the 
> best-studied baculovirus genomes and is 133,894 bp long ({% cite Ayres1994 %}). 
> It is the first fully sequenced genome of baculoviruses and today hundreds of genomes are fully sequenced
> and publically available at NCBI Genbank. The genome of AcMNPV-C6 encodes for 154 open reading frames (ORF)
> in both forward and reverse orientation (see figure). 
> So far, 38 ORFs were identified, which are encoded by all baculoviruses sequenced to date and are 
> called baculovirus core genes. In addition to the core genes, genes were found 
> that are encoded by baculoviruses from the genera *Alphabaculovirus* and *Betabaculovirus*. 
> 
> ![Alternative text](https://ictv.global/sites/default/files/inline-images/7651.ODD.Baculo.Fig2.v2.png "Figure: Schematic representation of the AcMNPV-C6 genome. ORFs and their orientation are indicated by arrows. (Figure from {% cite ICTV_AcMNPV_genome_figure %})")
>
{: .comment}

The genome size makes it difficult to study genetic variation within a baculovirus isolate, 
especially since the most commonly used sequencing technique (Illumina sequencing) 
generates only short reads and requires genome assembly. 
Since an isolate represents a population of viruses and their genomes that differ genetically from one another, 
the complexity of the analysis is further increased. 
Typically, genome assemblies are performed that lead to the generation of a consensus sequence, 
which usually reflects the majority of the sequenced virus population and intends to represent the entire isolate.
The problem is that the consensus sequence can mask genetic variability.
Tools for haplotype-sensitive assembly are available but so far have been established for viruses 
with a relatively short genome but not for baculoviruses or other large dsDNA viruses. 
Using Nanopore sequencing, it is possible to sequence significiant fragments of baculovirus genomes 
to determine major haplotypes ({% cite Wennmann2024 %}). 

SNVs have been proven as a powerful tool for analyzing the genetic variability of sequenced baculovirus isolates ({% cite Fan2020 %}).
SNVs are particularly useful for the analysis of intra-isolate specific
variation, as many bioinformatic workflows and tools are established for SNV determination 
and processing. For the identification of SNVs, Illumina sequencing data is mostly used, 
as this sequencing technique provides highly accurate reads with a very low probability 
of sequencing error. The data is usually provided and stored in FASTQ format.

When a baculovirus isolate is sequenced, a dataset of sequence reads is 
obtained that consists of a multitude of fragmented genomes and the exact reconstruction 
of individual whole genomes is no longer possible. However, variable SNV positions can 
be used as markers to count the frequency of the nucleotides that occur in the SNV positions. 
The determination of the nucleotides is only possible because not one but many genomes 
were sequenced. Therefore, the SNVs reflect the genetic variability of the isolate itself. 
Although deletions and insertions can also occur, these are not covered in this tutorial.

When multiple baculovirus isolates have been sequenced, SNV positions can be determined 
that are specific to only certain isolates. These specific SNV markers can then be used 
to identify an isolate based on sequencing data only, even when the isolate is present 
in a mixture with other isolates. To enable this identification of baculovirus isolates, 
SNV positions must be determined across multiple isolates using a common reference sequence. 
This reference serves as an anchor for the detected SNV positions and must be chosen with care. 
Based on the reference and the SNVs, SNV specificities can be determined for individual isolates.

The tutorial presented here aims to explain...
- the criteria for a suitable reference genome,
- a workflow that can be used to call SNVs across multiple sequenced baculovirus isolates,
- what is meant by SNV specificities and
- how to determine the composition of an isolate based on other previously sequenced baculovirus isolates.

The workflow is based on sequencing data from the Cydia pomonella granulovirus (CpGV) 
(family *Baculoviridae*, genus *Betabaculovirus*). CpGV is used commercially in agriculture 
for the biological control of larvae of the codling moth (*Cydia pomonella*), which cause 
significant losses to fruit crops, such as apples and pears.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Dataset Preparation

Let's start at the very beginning with a new history and a detailed description 
of the sequencing datasets as well as the reference sequence. 
We will also have a closer look at the Illumina sequencing datasets, what we 
know so far about them and what we can expect from the analysis. By doing this, 
we will work our way towards our goal step by step.

> <hands-on-title>Prepare the Galaxy history</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

## Reference Genome from NCBI Genbank

Selecting the right or most suitable reference sequence is important if not critical for the entire analysis. The reference genome used in this tutorial is the Cydia pomonella granulovirus (Mexican isolate, CpGV-M). The function of the reference is to act as an anchor point to link detected SNV positions in all sequenced and analyzed isolates. Selecting a correct reference genome is critical because it must be as closely related as possible to the sequenced isolates. Therefore, it is best to use a reference from the same species that has been well analyzed by previous studies or the scientific community.  The following step will download the NCBI Accession Number `KM217575`, which corresponds to the `Cydia pomonella granulovirus isolate CpGV-M, complete genome`. It will later serve as the reference genome for the detection of SNV positions.  


> <hands-on-title> Reference genome download </hands-on-title>
>
> 1. {% tool [NCBI Accession Download](toolshed.g2.bx.psu.edu/repos/iuc/ncbi_acc_download/ncbi_acc_download/0.2.8+galaxy0) %} with the following parameters:
>    - *"Select source for IDs"*: `Direct Entry`
>        - *"ID List"*: `KM217575`
>    - *"Molecule Type"*: `Nucleotide`
>    - *"File Format"*: `FASTA` 
>    - Click Run Tool
>
>    > <warning-title>Use a single reference genome only!</warning-title>  
>    > **The NCBI Accession Download** tool accepts multiple NCBI accession numbers as input, which should be avoided, otherwise the workflow will fail. If you still want to use multiple reference genomes, then I recommend to run the entire workflow for each reference genome separately.
>    {: .warning}
>
> 2. {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0) %} with the following parameters:
>    - {% icon param-file %} *"Collection of files to collapse into single dataset"*: `NCBI Accession Download on: Downloaded files` (output of **NCBI Accession Download** {% icon tool %})
>    - Click Run Tool
>
>    > <comment-title> Collapse Genbank files </comment-title>
>    > **NCBI Accession Download** accepts multiple accession numbers and creates a list as output. 
>    > Therefore, its output can contain multiple records. For this analysis just one reference genome
>    > is required and the list needs to be converted into a single file.
>    {: .comment}
>
> 3. {% tool [Replace Text in entire line](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: the collapsed collection containing the single reference genome; output of **Collapse Collection** {% icon tool %}
>    - In *"Replacement"*:
>         - *"Find pattern"*: `^>.*$`
>         - *"Replace with:"*: `>CpGV-M `
>    - Click Run Tool
>
>    > <comment-title> What the tool does... </comment-title>
>    > The find pattern `^>.*$` searches for the `>` and all characters that follow.
>    > Then everything (the entire line) is replaced by `>CpGV-M`. 
>    {: .comment}
>
>    > <comment-title> Why we change the name... </comment-title>
>    > The reference genome is stored in a single file in FASTA format. After downloading from NCBI, the name of the reference is `>KM217575.1 Cydia pomonella granulovirus isolate CpGV-M, complete genome`. The name is very long and can cause issues during later analysis (in R or Python). Therefore, I recommend to replace the name with the **Replace text in entire line** tool.
>    {: .comment}
>
{: .hands_on}

## Paired-end Sequencing Data from NCBI SRA

The tutorial is based on Illumina data sets from several isolates of the Cydia pomonella granulovirus (CpGV). 
The CpGV is one of the most well-studied baculoviruses because many isolates have been sequenced and sequence 
data sets are available at NCBI Genbank and NCBI SRA. We will use four CpGV isolates to decipher their 
intra-isolate specific variation and try to draw conclusions about the isolate's homogeneity or heterogeneity. 
We will also encounter a mixed isolate (a more or less clean mixture of other previously sequenced CpGV isolates) 
and learn how to recognize it. In addition, we will learn how to determine the composition 
of this mixed isolate based on other sequenced CpGV isolates.

> <comment-title> Detailed CpGV Isolate Information </comment-title>
>
> For those who want to delve deeper into the data sets, I have provided a table with links to NCBI SRA and related publications.
>
> | Isolate | NCBI Genbank | NCBI SRA | Reference |
> |:------------------:|:------------------:|:------------------:|:------------------:|
> | CpGV-M | KM217575 | [SRR31589148](https://trace.ncbi.nlm.nih.gov/Traces?run=SRR31589148) | {% cite Wennmann2020 %}
> | CpGV-S | KM217573 | [SRR31589147](https://trace.ncbi.nlm.nih.gov/Traces?run=SRR31589147) | {% cite Wennmann2020 %} |
> | CpGV-E2 | KM217577 | [SRR31589146](https://trace.ncbi.nlm.nih.gov/Traces?run=SRR31589146) | {% cite Alletti2017 %} |
> | CpGV-V15 | No assembly available | [SRR31679023](https://www.ncbi.nlm.nih.gov/sra/SRX27041396) | {% cite Fan2020 %} |
{: .comment}

Follow the steps below to download the four Illumina datasets published at NCBI SRA.

> <hands-on-title>NCBI SRA Data Download</hands-on-title>
>
>    Run {% tool [Download and Extract Reads in FASTQ format from NCBI SRA](toolshed.g2.bx.psu.edu/repos/iuc/sra_tools/fastq_dump/3.1.1+galaxy1) %} with the following parameters:
>
>    - *"Select input type"*: `SRR accession`
>    - *Accession*: `SRR31589146, SRR31589147, SRR31589148, SRR31679023`
>    - *Select output format*: `gzip compressed fastq`
>    - Click Run Tool
>
>    > <comment-title> Take a cup of coffee </comment-title>
>    > Downloading the data from NCBI SRA may take a while.
>    > It is a good opportunity to have a cup of coffee or tea.
>    {: .comment}
>
{: .hands_on}

# Read Preprocessing and Quality Control

After the reference genome and the paired-end Illumina reads are loaded in the history, 
it is time to start the analysis. First, the reads need to be adapter trimmed and
quality filtered. 

> <tip-title>Learn more about quality of reads!</tip-title>
> Before and after trimming, it is usually a good idea to check the quality of the reads using FASTQC. 
> However, we will omit this step here. 
> If you are interested in learning more about read quality, I recommend the check the [Galaxy Training Quality Control]( {% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %} ) tutorial.
{: .tip}


> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Trim Galore!](toolshed.g2.bx.psu.edu/repos/bgruening/trim_galore/trim_galore/0.6.7+galaxy0) %} with the following parameters:
>    - *"Is this library paired- or single-end?"*: `Paired Collection`
>        - {% icon param-collection %} *"Select a paired collection"*: `Paired-end data (fastq-dump)` (output of **Download and Extract Reads in FASTQ format from NCBI SRA** {% icon tool %})
>    - *"Advanced settings"*: `Full parameter list`
>        - *"Trim low-quality ends from reads in addition to adapter removal (Enter phred quality score threshold)"*: `30`
>        - *"Discard reads that became shorter than length N"*: `50`
>        - *"specify if you would like to retain unpaired reads"*: `Do not output unpaired reads`
>    - *"RRBS specific settings"*: `Use defaults (no RRBS)`
>    - *"Trimming settings"*: `Use defaults`
>    - Click Run Tool
>
{: .hands_on}


# Mapping of Reads to the Reference Genome

Now the filtered reads of the individual isolates are mapped independently against the CpGV-M reference. 
The key is that each isolate is mapped separately against the identical reference genome, so that in the end we get a Binary Alignment Mapping (BAM) file for each isolate.
After that, everything we do with the sequence data is linked to the common reference.

> <hands-on-title> Read mapping using bcftools </hands-on-title>
>
> 1. {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.18) %} with the following parameters:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>        - {% icon param-file %} *"Use the following dataset as the reference sequence"*: the renamed reference genome in FASTA format, output of **Replace Text in entire line** {% icon tool %}
>    - *"Single or Paired-end reads"*: `Paired Collection`
>        - {% icon param-file %} *"Select a paired collection"*: the trimmed paired read collection, generated as output of **Trim Galore!** {% icon tool %}
>    - *"Set read groups information?"*: `Set read groups (Picard style)`
>        - *"Auto-assign"*: `Yes`
>        - *"Auto-assign"*: `Yes`
>        - *"Auto-assign"*: `Yes`
>    - *"Select analysis mode"*: `1.Simple Illumina mode`
>    - Click Run Tool
>
{: .hands_on}


# SNP Calling and Variant Detection

The key part of the analysis is the determination of variable SNV positions, which is carried out in the following step. 
Insertions/Deletions (indels) are deliberately omitted because they are not relevant for this analysis.

> <hands-on-title> Variable SNV positions </hands-on-title>
>
> 1. {% tool [bcftools mpileup](toolshed.g2.bx.psu.edu/repos/iuc/bcftools_mpileup/bcftools_mpileup/1.15.1+galaxy4) %} with the following parameters:
>    - *"Alignment Inputs"*: `Multiple BAM/CRAMs`
>        - {% icon param-file %} *"Input BAM/CRAMs"*: the BAM file collection; output of **Map with BWA-MEM** {% icon tool %}
>    - *"Choose the source for the reference genome"*: `History`
>        - {% icon param-file %} *"Genome Reference"*: the renamed reference genome in FASTA format, output of **Replace Text in entire line** {% icon tool %}
>    - In *"Indel Calling"*:
>        - *"Perform INDEL calling"*: `Do not perform INDEL calling`
>    - In *"Input Filtering Options"*:
>        - *"Max reads per BAM"*: `1024`
>        - *"Set filter by flags"*: `Do not filter`
>        - *"Quality Options"*: `defaults`
>        - *"Select read groups to include or exclude"*: `use defaults`
>    - In *"Restrict to"*:
>        - *"Regions"*: `Do not restrict to Regions`
>        - *"Targets"*: `Do not restrict to Targets`
>    - In *"Output options"*:
>        - *"Optional tags to output"*: `DP (Number of high-quality bases)`
>        - *"Optional tags to output"*: `DPR (Number of high-quality bases for each observed allele)`
>    - *"Output type"*: `uncompressed VCF`
>    - Click Run Tool
>
>    > <comment-title> What bcftools mpileup does... </comment-title>
>    > bcftools mpileup calculates the nucleotide coverage for each position in the reference genome based on the input BAM files. It enables preparation for variant analysis. 
>    {: .comment}
>
> 2. {% tool [bcftools call](toolshed.g2.bx.psu.edu/repos/iuc/bcftools_call/bcftools_call/1.15.1+galaxy5) %} with the following parameters:
>    - {% icon param-file %} *"VCF/BCF Data"*: the uncompressed VCF file; output of **bcftools mpileup** {% icon tool %}
>    - In *"Restrict to"*:
>        - *"Regions"*: `Do not restrict to Regions`
>    - In *"Consensus/variant calling Options"*:
>        - *"Calling method"*: `Multiallelic and rare-variant caller`
>            - *"Constrain"*: `Do not constrain`
>                - *"Targets"*: `Do not restrict to Targets`
>    - In *"File format Options"*:
>        - *"Regions"*: `Do not restrict to Regions`
>    - In *"Input/output Options"*:
>        - *"Keep alts"*: `Yes`
>        - *"Output variant sites only"*: `Yes`
>    - *"Output type"*: `uncompressed VCF`
>    - Click Run Tool
>
>    > <comment-title> What bcftools call does... </comment-title>
>    > bcftools call performs the actual variant detection. It removes non variant and keeps variant sites only.
>    {: .comment}
>
{: .hands_on}

What you get is a file in Variant Call Format (VCF), which can be difficult to understand at first glance. It is important to note that the file begins with a large header, lines that begin with two hashtags (##), which contain details of the analysis and the data set. After the header, there is a table with a line that contains the column names. This line begins with just one hashtag (#). After that, the results are displayed in a tab-separated format. In our VCF example, some columns have the following abbreviations and meanings (I just selected a few that are important to us in this tutorial):

| Fixed fields  | Description  |
|---:|:---|
|**CHROM**  |Chromosome (name of the reference, in our case CpGV-M)  |
|**POS**  |Position (position in the reference genome)   |
|**REF**  |Nucleotide of the Referance at the corresponding position (POS)   |
|**ALT**   |Alternative nucleotides detected at this position in the sequencing data.   |
|**FORMAT**   |Describes the content of the *sample column* reported in the VCF file. In our case GT:PL:DP:DPR.           |
|**SRR31589146**   |1st isolate: The results for the sequence data SRR31589146 in the format specified in FORMAT.   |
|**SRR31589147**   |2nd isolate: The results for the sequence data SRR31589147 in the format specified in FORMAT.   |
|**SRR31589148**   |3rd isolate: The results for the sequence data SRR31589148 in the format specified in FORMAT.   |
|**SRR31679023**   |4th isolate: The results for the sequence data SRR31679023 in the format specified in FORMAT.   |


To understand how the data is stored, we have to look at FORMAT in detail. This is where two values are of great importance: DP and DPR.

|Genotype&nbsp;field|Description   |
|---:|:---|
|**DP**   |Read depth (number of nucleotides) at this position for this isolate.   |
|**DPR**   |The read depth for each allele. Here, the first value corresponds to the reference nucleotide (REF), the second to the first possible allele (ALT1), the second to the second possible allele (ALT2) and the third value to the last possible allele (ALT3).

> <comment-title>DPR functional, but deprecated</comment-title>
> If you start **bcftools mpileup** with the *Optional tags to output* `DPR (Number of high-quality bases for each observed allele)` option, you will see a warning in the information panel of the VCF file: `[warning] tag DPR functional but deprecated. Please switch to AD in future.` At the time of writing this tutorial, I was using DPR. It works the same way with *Optional tags to output* option `AD` (instead of DPR), but you will need to adjust something later in the workflow. The tutorial will be switched to AD in the near future. For now, we stick with DPR.
{: .comment}

> <tip-title>Learn more about variant analysis on diploid and non-diploid data!</tip-title>
> Also take a look at other tutorials that deal with non-diploid (but also diploid) data sets to perform a variant analysis. Broadening your horizons is always important. I recommend the check the tutorials on ... 
> * [Calling variants in diploid systems]( {% link topics/variant-analysis/tutorials/dip/tutorial.md %} ) and.
> * [Calling variants in non-diploid systems]( {% link topics/variant-analysis/tutorials/non-dip/tutorial.md %} ).
> 
> Maybe later? Then let's continue.
{: .tip}

# VCF to Table Transformation

Now we come to an exciting part, because we have all the information we need to analyse the genetic variation within the sequenced virus isolates. The data is only hidden in the VCF file and is difficult for the beginner in bioinformatics to see. We have the positions (`POS`), which were detected as variable in the virus isolates. In addition, we know the number of all reads (and thus also nucleotides) in these positions (represented by `DP`). By using `DPR`, we obtain information on how often the alleles (the four possible nucleotides) occur at a particular position. To analyse `DP` and `DPR`, we first have to access it because the information is hidden in each *sample column* in the `FORMAT` genotype data. If we look at the first position `POS = 246` in sample column `SRR31589146`, the following data is visible:     
`1/1:255,134,0,255,255,255,255,255,255,255:886:107,773,6,0`.  
The information provided by the FORMAT field explains the division of the data by colons: `GT:PL:DP:DPR`.

We can break it down like this...

| FORMAT genotype field  | Value                                 |
|-----------------------:|:--------------------------------------|
|GT                      |1/1                                    |
|PL                      |255,134,0,255,255,255,255,255,255,255  |
|DP                      |886                                    |
|DPR                     |107,773,6,0                            |

GT = Genotype information, which cannot be used with virus isolates!  
PL = Phred-scaled genotype likelihood (also not useable for us, because we do not have a diploid organism!)

DPR can be broken down even further, since the individual values, which are separated by commas this time, can be assigned to the reference or the alternative nucleotides. In a virus isolate, four nucleotides (A, T, G and C) can theoretically occur at each position. One nucleotide defines the reference, leaving three alternatives: first (`ALT1`), second (`ALT2`) and third (`ALT3`) alternative. 



|CHROM   |POS   |DP   |DPR           | ALT1  |ALT2    |ALT3  |  
|:------:|:----:|:---:|:------------:|:-----:|:------:|:----:|
|CpGV-M  |246   |886  |107,773,6,0   |773    |6       |0     |

If we now divide the absolute frequencies of `ALT1 = 773`, `ALT2 = 6` and `ALT3 = 0` by `DP = 886`, we get the relative frequencies (`REL.ALT`):

|ALLELE     |REL.ALT     |
|:---------:|:----------:|
|ALT1       |0.87246     |
|ALT2       |0.00677201  |
|ALT3       |0           |


> <hands-on-title> Transfrom VCF to tab-delimited table </hands-on-title>
>
> 1. {% tool [VCFtoTab-delimited:](toolshed.g2.bx.psu.edu/repos/devteam/vcf2tsv/vcf2tsv/1.0.0_rc1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Select VCF dataset to convert"*: the filtered VCF file; output of **bcftools call** {% icon tool %}
>    - *"Fill empty fields with"*: `NULL`
>    - Click Run Tool
>
>    > <comment-title> An easier-to-read table </comment-title>
>    > This tool creates a table from the VCF file. The columns are tab-delimited. Take a look at the table and see if it is now easier to read. It can also be imported to R/RStudio/Excel more easily.
>    {: .comment}
>
> 2. {% tool [Text reformatting with awk](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_awk_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: the tab-delimited VCF table; output of **VCFtoTab-delimited** {% icon tool %}
>    - *"AWK Program"*: *Paste the code from the code box below.*
> > <code-in-title>awk</code-in-title>
> > ```
> > BEGIN { FS="\t"; OFS="\t" }
> > NR == 1 {
> > 
> >     # Rename "SAMPLE" to "ISOLATE" in the header
> >     for (i = 1; i <= NF; i++) {
> >         if ($i == "SAMPLE") $i = "ISOLATE";
> >     }
> >     # Identify column indices based on header names
> >     for (i = 1; i <= NF; i++) {
> >         if ($i == "POS") pos_col = i;
> >         if ($i == "ISOLATE") isolate_col = i;
> >         if ($i == "DP") dp_col = i;
> >         if ($i == "DPR") dpr_col = i;
> >     }
> >     # Check if all required columns are found
> >     if (!(pos_col && isolate_col && dp_col && dpr_col)) {
> >         print "Error: One or more required columns (POS, ISOLATE, DP, DPR) are missing." > "/dev/stderr";
> >         exit 1;
> >     }
> >     # Print the new header
> >     print $0, "ALLELE", "DPR.ALLELE", "REL.ALT", "REL.ALT.0.05";
> >     next;
> > }
> > {
> >     # Create a unique key for position and isolate
> >     pos_isolate_key = $(pos_col) "_" $(isolate_col);
> > 
> >     if (last_pos != $(pos_col)) {  # When a new position starts
> >         delete isolate_count;      # Reset the isolate counter
> >         last_pos = $(pos_col);     # Update the current position
> >     }
> > 
> >     # Split the DPR column into values
> >     split($(dpr_col), dpr_values, ",");
> > 
> >     # Count occurrences of each isolate per position
> >     isolate_count[$(isolate_col)]++;
> >     dpr_index = isolate_count[$(isolate_col)] + 1;  # Index for the alternate allele (starting at 2)
> > 
> >     # Check if the index is valid
> >     if (dpr_index <= length(dpr_values)) {
> >         allele = "ALT" (dpr_index - 1);            # Determine the allele
> >         dpr_value = dpr_values[dpr_index];         # Get the corresponding DPR value
> >         rel_alt = (dpr_value / $(dp_col));         # Calculate REL.ALT (DPR.ALLELE / DP)
> >         rel_alt_filtered = (rel_alt >= 0.05) ? rel_alt : 0;  # Filter REL.ALT values < 0.05
> >         print $0, allele, dpr_value, rel_alt, rel_alt_filtered;  # Output with new columns
> >     }
> > }
> > ```
> {: .code-in}
>
>    - Click Run Tool
> 
>    > <comment-title> Create ALLELE and REL.ALT columns </comment-title> 
>    > The **Text reformatting with awk** tool allows the use of AWK code. AWK is for text processing and to performing complex tasks on data sets (such as tab-delimited files). We will come across AWK scripts again later. Here is a brief summary of what it does: 
>    > * Splitting `DPR` data into separate allele counts.   
>    > * Calculating the relative allele frequencies (`REL.ALT`) for each allele.  
>    > * Filtering out relative frequencies below a threshold of 0.05 and storing the result in `REL.ALT.0.05`.  
>    {: .comment}
>
{: .hands_on}


The output table is complex and shows the relative frequency (`REL.ALT`) for each of the three alternative alleles/nucleotides (`ALT1`, `ALT2` and `ALT3`) in each position (`POS`) and sequenced CpGV isolate (`ISOLATE`). In column `REL.ALT.0.05`, values of the `REL.ALT < 0.05` were set to 0 to set a threshold. We will see later why this is sometimes important.

Below is the table with selected relevant columns only. `REL` and `ALT` show the reference and alternative nucleotide, respectively.    

| CHROM  | POS | REF | ALT | ISOLATE     | DP  | DPR         | ALLELE | DPR.ALLELE | REL.ALT    | REL.ALT.0.05 |
|--------|-----|-----|-----|-------------|-----|-------------|--------|------------|------------|--------------|
| CpGV-M | 246 | C   | T   | SRR31589146 | 886 | 107,773,6,0 | ALT1   | 773        | 0.87246    | 0.873446     |
| CpGV-M | 246 | C   | G   | SRR31589146 | 886 | 107,773,6,0 | ALT2   | 6          | 0.00677201 | 0            |
| CpGV-M | 246 | C   | A   | SRR31589146 | 886 | 107,773,6,0 | ALT3   | 0          | 0          | 0            |
| CpGV-M | 246 | C   | T   | SRR31589147 | 881 | 5,875,1,0   | ALT1   | 875        | 0.99319    | 0.994305     |
| CpGV-M | 246 | C   | G   | SRR31589147 | 881 | 5,875,1,0   | ALT2   | 1          | 0.00113507 | 0            |
| CpGV-M | 246 | C   | A   | SRR31589147 | 881 | 5,875,1,0   | ALT3   | 0          | 0          | 0            |
| CpGV-M | 246 | C   | T   | SRR31589148 | 935 | 800,133,1,1 | ALT1   | 133        | 0.142246   | 0.142398     |
| CpGV-M | 246 | C   | G   | SRR31589148 | 935 | 800,133,1,1 | ALT2   | 1          | 0.00106952 | 0            |
| CpGV-M | 246 | C   | A   | SRR31589148 | 935 | 800,133,1,1 | ALT3   | 1          | 0.00106952 | 0            |
| CpGV-M | 246 | C   | T   | SRR31679023 | 839 | 42,797,0,0  | ALT1   | 797        | 0.94994    | 0.950296     |
| CpGV-M | 246 | C   | G   | SRR31679023 | 839 | 42,797,0,0  | ALT2   | 0          | 0          | 0            |
| CpGV-M | 246 | C   | A   | SRR31679023 | 839 | 42,797,0,0  | ALT3   | 0          | 0          | 0            |


## Replace SRA Names with Virus Abbreviations

One thing that stands out are the NCBI SRA numbers in the ISOLATE column, which have been automatically extracted from the NCBI SRA datasets. Since it is difficult to remember which virus isolate is behind which number, we can replace these accession numbers with proper names, e.g. virus isolate abbreviations. This makes the table even easier to read and later we can use the information directly to display the SNV positions. 

> <hands-on-title> Replace SRA accession numbers by virus isolate abbreviations </hands-on-title>
>
> 1. {% tool [Replace Text in a specific column](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: the reformatted VCF table; output of **Text reformatting with awk:** {% icon tool %}
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"in column"*: `c25`
>            - *"Find pattern"*: `SRR31589148`
>            - *"Replace with"*: `CpGV-M`
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"in column"*: `c25`
>            - *"Find pattern"*: `SRR31589147`
>            - *"Replace with"*: `CpGV-S`
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"in column"*: `c25`
>            - *"Find pattern"*: `SRR31589146`
>            - *"Replace with"*: `CpGV-E2`
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"in column"*: `c25`
>            - *"Find pattern"*: `SRR31679023`
>            - *"Replace with"*: `CpGV-V15`
>    - Click Run Tool
>
>    > <question-title>What is replaced by what?</question-title>
>    > 1. Can you say which SRA number was replaced by which isolate abbreviation?
>    >    > <solution-title></solution-title>
>    >    > * SRR31589148 was replaced by CpGV-M  
>    >    > * SRR31589147 was replaced by CpGV-S  
>    >    > * SRR31589146 was replaced by CpGV-E2  
>    >    > * SRR31679023 was replaced by CpGV-V15  
>    >    {: .solution}
>    {: .question}
>
{: .hands_on}

## Reduce Complexity of SNV Table to First Alternative

Based on the SNV table, we can see that three possible nucleotides (alleles) occur in most positions. After filtering with a minimum threshold of REL.ALT > 0.05, many relative frequencies of ALT2 and ALT3 are set to zero. At least it is visible that in the vast majority of positions only the first alternative has a value of > 0.05. Thus, two nucleotides usually occur in the detected SNV positions: the reference nucleotide (REF) and the nucleotide of the first alternative (ALT1). We want to reduce the complexity of the data set and only consider ALT1 in the further course of this tutorial.

> <hands-on-title>Keep only ALT1 and remove ALT2 and ALT3</hands-on-title>
>
> 1. {% tool [Filter data on any column using simple expressions](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: the VCF table with the replaced isolate abbreviations; output of **Replace Text in a specific column** {% icon tool %}
>    - *"With following condition"*: `c30=='ALT1'`
>    - *"Numbers of header ines to skip"*: `1`
>    - Click Run Tool
>
>    > <question-title>How can you check whether only two nucleotides occur (mostly) per SNV position?</question-title>
>    > Think about an option how the occurance of one, two, three or four nucleotides per SNV position can be analyzed.
>    >    > <solution-title></solution-title>
>    >    > When looking only at a table with several columns and numbers, it is difficult to understand that mainly a reference and one alternative nucleotide were detected in the SNV positions. 
>    >    > One way to examine the frequency of a second or third alternative nucleotide is to filter the table with `c30=='ALT2'` or `c30=='ALT3'` instead of `c30=='ALT1'`.
>    >    > The resulting table can be used to create a distribution over the `REL.ALT` values in column 32 (`c32`).
>    >    >
>    >    > You can perform this analysis by using the following tools: **Filter data on any column using simple expressions** {% icon tool %}, **Cut columns from a table** {% icon tool %} to extract column 32 (`c32`), which corresponds to `REL.ALT`
>    >    > and **Histogram with ggplot2** {% icon tool %}.
>    >    >
>    >    > This task is left to you as a small challenge, if you like. It is up to you to familiarise yourself with the individual tools,
>    >    > but it could be worth it, as it could help you understand why we do not consider ALT2 and ALT3 further in the next steps.
>    >    {: .solution}
>    {: .question}
>
>    > <warning-title>Be careful when removing data!</warning-title>  
>    > The data sets used here have already been analysed extensively and for this tutorial only the first alternative nucleotide (allele) is sufficient to explain the further workflow. Always be sure what you are doing when you remove data! Always ask yourself if you are allowed to do it and what the consequences might be. 
>    {: .warning}
>
{: .hands_on}

# Visualizing SNV Variability Across Isolates

The final table should be much easier to read and contain all the information we need to perform an analysis of the intra-isolate specific variability. Here is a short overview of the final table (selected columns) and its first three SNV positions:  

| CHROM  | POS | REF | ALT | ISOLATE  | DP  | DPR         | ALLELE | DPR.ALLELE | REL.ALT  | REL.ALT.0.05 |
|--------|-----|-----|-----|----------|-----|-------------|--------|------------|----------|--------------|
| CpGV-M | 246 | C   | T   | CpGV-E2  | 886 | 107,773,6,0 | ALT1   | 773        | 0.87246  | 0.87246      |
| CpGV-M | 246 | C   | T   | CpGV-S   | 881 | 5,875,1,0   | ALT1   | 875        | 0.99319  | 0.99319      |
| CpGV-M | 246 | C   | T   | CpGV-M   | 935 | 800,133,1,1 | ALT1   | 133        | 0.142246 | 0.142246     |
| CpGV-M | 246 | C   | T   | CpGV-V15 | 839 | 42,797,0,0  | ALT1   | 797        | 0.94994  | 0.94994      |
| CpGV-M | 249 | A   | C   | CpGV-E2  | 888 | 106,780,1   | ALT1   | 780        | 0.878378 | 0.878378     |
| CpGV-M | 249 | A   | C   | CpGV-S   | 881 | 2,879,0     | ALT1   | 879        | 0.99773  | 0.99773      |
| CpGV-M | 249 | A   | C   | CpGV-M   | 938 | 799,139,0   | ALT1   | 139        | 0.148188 | 0.148188     |
| CpGV-M | 249 | A   | C   | CpGV-V15 | 843 | 48,793,2    | ALT1   | 793        | 0.940688 | 0.940688     |
| CpGV-M | 564 | T   | C   | CpGV-E2  | 964 | 98,856,9,1  | ALT1   | 856        | 0.887967 | 0.887967     |
| CpGV-M | 564 | T   | C   | CpGV-S   | 947 | 1,940,6,0   | ALT1   | 940        | 0.992608 | 0.992608     |
| CpGV-M | 564 | T   | C   | CpGV-M   | 976 | 821,154,1,0 | ALT1   | 154        | 0.157787 | 0.157787     |
| CpGV-M | 564 | T   | C   | CpGV-V15 | 880 | 33,846,1,0  | ALT1   | 846        | 0.961364 | 0.961364     |

We can now start the first visualisation and create a plot for each CpGV isolate (`ISOLATE`), plotting the position of the SNV (`POS`) against the relative frequency of the alternative nucleotide (`REL.ALT`). Note that we are not using the REL.ALT threshold values because we want to look at the unfiltered data.

> <hands-on-title>SNV plot</hands-on-title>
>
> 1. {% tool [Scatterplot with ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/3.4.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input in tabular format"*: the ALT1-filtered VCF table; output of **Filter data on any column using simple expressions** {% icon tool %}
>    - *"Column to plot on x-axis"*: `2`
>    - *"Column to plot on y-axis"*: `32`
>    - *"Plot title"*: `SNV plot`
>    - *"Label for x axis"*: `Reference Genome Position of CpGV-M`
>    - *"Label for y axis"*: `Relative Nucleotide Frequency `
>    - In *"Advanced options"*:
>        - *"Type of plot"*: `Points only (default)`
>            - *"Data point options"*: `Default`
>        - *"Plotting multiple groups"*: `Plot multiple groups of data on individual plots`
>            - *"column differentiating the different groups"*: `25`
>        - *"Axis title options"*: `Default`
>        - *"Axis text options"*: `Default`
>        - *"Plot title options"*: `Default`
>        - *"Grid lines"*: `Hide major and minor grid lines`
>        - *"Axis scaling"*: `Automatic axis scaling`
>    - In *"Output Options"*:
>        - *"Unit of output dimensions"*: `Centimeters (cm)`
>        - *"width of output"*: `20.0`
>        - *"height of output"*: `12.0`
>        - *"dpi of output"*: `200.0`
>    - Click Run Tool
>
>    > <comment-title> Column names and numbers </comment-title>
>    > To provide ggplot2 with the data for the X- and Y-axis, column numbers must be passed to the function. You have to check which column numbers the columns `POS` and `REL.ALT` have. In addition, ggplot2 offers the option of splitting the data according to `ISOLATE`, to obtain a separate plot for each value in ISOLATE. 
>    > * `POS` = column 2  
>    > * `REL.ALT` = column 32  
>    > * `ISOLATE` = column 25  
>    > ![VCF tab-delimited table with column names and numbers](../../images/baculovirus-isolate-variation/galaxy_vcf_table_column_numbers_names.png "Section of the VCF table with column numbers and names. ")
>    {: .comment}
>
{: .hands_on}

As a result, we get a SNV plot that shows the relative frequency of the first alternative nucleotide in all variable SNV positions. Before we have a closer look at the data, we have to remember that all the sequencing data (= the reads) were mapped during this workflow against the reference genome of CpGV-M. For that reason, all isolates are compared to the genome of CpGV-M. We see that CpGV-E2 has a heterogeneous SNV pattern. CpGV-M appears homogeneous but with relatively few genetic variation – but keep in mind that the sequence data of CpGV-M was mapped against its own reference, which is why one might assume less variability. For CpGV-S, the variability is also homogeneous, but many SNV positions have a nucleotide frequency close to one. This could indicate that the isolate is homogeneous but different from the reference CpGV-M. For isolate CpGV-V15, a SNV cloud occurs at approximately 0.5 (50%), which could indicate a mixed isolate. By “mixed”, it is meant that two or more (homogenous) isolates occur in a certain ratio in this isolate.

![SNV plot](../../images/baculovirus-isolate-variation/snp_plot.png "Single nucleotide variant (SNV) plot of all variable positions in the sequenced CpGV isolates CpGV-E2 (top left), CpGV-M (top right), CpGV-S (bottom left) and CpGV-V15 (bottom right). Each point represents the relative frequency of the first alternative nucleotide (allele).")

# SNV Specificity Determination

Now we come to the last but most complex section of this tutorial - determining SNV specificities. We have seen that CpGV-V15 is a mixed isolate. Now we want to find out which isolates were mixed and how this SNV pattern can be explained. In the next step, we will consider  SNV positions as markers and look for SNV positions that are only variable for one or more sequenced isolates of CpGV. If a SNV position is only variable for CpGV-S (i.e. the relative frequency is greater than 0; `REL.ALT > 0`) but is equal to zero for the other isolates, then this position is a marker for CpGV-S, or specific for CpGV-S. Since we cannot perform this determination by hand for all positions, I have written a small program implemented in the **Text reformatting with awk** tool that does the work for us. 

Before we get started, I would like to explain specificity in more detail using an example. First, we decide that we want to determine the specificities for the SNV positions for the following isolates:
* CpGV-M
* CpGV-S
* CpGV-E2.  

Isolate CpGV-V15 is not included in the analysis of SNV specificity determination because we want to explain its composition later with the other data sets.  

The principle behind SNV specificities and how they are determined can be explained by the table below. In the first position 249, all isolates (CpGV-E2, CpGV-S and CpGV-M) have a relative nucleotide frequency > 0. This means that all three sequenced isolates have an alternative nucleotide at this position, with frequencies of 14.7%, 88% and >99% for CpGV-M, CpGV-E2 and CpGV-S, respectively. The remaining percentages are the reference nucleotide. Note that I am using the `REL.ALT.0.05` values, which are based on the threshold of 0.05 (values < 0.05 were set to zero). By this, the tutorial is simplified. Position 249 is thus variable for all three sequenced isolates and therefore specific for `CpGV-E2 + CpGV-S + CpGV-M`.
In the second position 603, only CpGV-E2 has an alternative nucleotide, its `REL.ALT.0.05 > 0`. The isolates of CpGV-S and CpGV-M show no variability in position 603 and are therefore identical to the reference. Thus, `position 603 is specific for CpGV-E2`. If we apply the same logic to positions 1278 and 6393, we can see that `position 1278 is specific for CpGV-S` and `position 6393 is specific for CpGV-E2 and CpGV-S`.  
If we now wanted to detect specifically isolate CpGV-S and CpGV-E2 in a mixture, positions 603 and 1278 could serve as markers.    

|  POS   | REL.ALT0.05 (CpGV-E2) | REL.ALT0.05 (CpGV-S)  | REL.ALT0.05 (CpGV-M)  | SNV Specificity               |  
|--------|-----------------------|-----------------------|-----------------------|-------------------------------|
|249     | 0.878378              | 0.99773               | 0.148188              | CpGV-E2 + CpGV-S + CpGV-M     |
|603     | 1                     | 0                     | 0                     | CpGV-E2                       |
|1278    | 0                     | 0.998963              | 0                     | CpGV-S                        |
|6393    | 0.98847               | 0.997783              | 0                     | CpGV-E2 + CpGV-S              |

Let us run the tool below to determine the SNV specificities of our dataset.  

> <hands-on-title> SNV specificity determination </hands-on-title>
>
> 1. {% tool [Text reformatting with awk](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_awk_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: the ALT1-filtered VCF table; output of **Filter data on any column using simple expressions** {% icon tool %}
>    - *"AWK Program"*: *Paste the code from the code box below.*
> > <code-in-title>awk</code-in-title>
> > ```
> > BEGIN { FS="\t"; OFS="\t" }
> > NR == 1 {
> >     # Dynamically find column indices based on header names
> >     for (i = 1; i <= NF; i++) {
> >         if ($i == "POS") pos_col = i;
> >         if ($i == "ALLELE") allele_col = i;
> >         if ($i == "ISOLATE") isolate_col = i;
> >         if ($i == "REL.ALT.0.05") rel_alt_col = i;
> >     }
> > 
> >     # Check if all required columns were found
> >     if (!(pos_col && allele_col && isolate_col && rel_alt_col)) {
> >         print "Error: Required columns (POS, ALLELE, ISOLATE, REL.ALT.0.05) are missing." > "/dev/stderr";
> >         exit 1;
> >     }
> > 
> >     # Print the header and add the new SPEC column
> >     print $0, "SPEC";
> >     next;
> > }
> > 
> > {
> >     # When reaching a new position, prepare the specificity
> >     if ($(pos_col) != current_pos) {
> >         # Assign the calculated specificity to all rows of the current position
> >         for (i in pos_lines) {
> >             # Add "SNV specificity: " before the specificity value
> >             final_spec = (specificity == "" ? "0" : "SNV specificity: " specificity);
> >             print pos_lines[i], final_spec;
> >         }
> >         # Reset variables for the new position
> >         delete pos_lines;
> >         specificity = "";
> >         current_pos = $(pos_col);
> >     }
> > 
> >     # Save the current row for later
> >     pos_lines[NR] = $0;
> > 
> >     # Conditions for calculating specificity
> >     if ($(allele_col) == "ALT1" && 
> >         ($(isolate_col) == "CpGV-E2" || $(isolate_col) == "CpGV-S" || $(isolate_col) == "CpGV-M") && 
> >         $(rel_alt_col) > 0) {
> >         # Concatenate isolate names with " + " if REL.ALT.0.05 > 0
> >         specificity = (specificity == "" ? $(isolate_col) : specificity " + " $(isolate_col));
> >     }
> > }
> > 
> > END {
> >     # Assign the specificity to the last position
> >     for (i in pos_lines) {
> >         # Add "SNV specificity: " before the specificity value
> >         final_spec = (specificity == "" ? "0" : "SNV specificity: " specificity);
> >         print pos_lines[i], final_spec;
> >     }
> > }
> > ```
> {: .code-in}
>
>    - Click Run Tool
> 
>
{: .hands_on}

# Visualization of Variable SNV Positions by Specificity

Now, the VCF table has an additional column called `SPEC`, which indicates the SNV specificity for each position. This allows us to create a special SNV plot for each isolate. The special thing about this is that a separate plot is created for each isolate and each specificity. Since we can only look at each isolate individually, we first have to filter out the data for one isolate, we start with CpGV-S. After that, a plot is created for each SNV specificity for the CpGV-S isolate.  

> <hands-on-title> Extract data for one isolate only </hands-on-title>
>
> 1. {% tool [Filter data on any column using simple expressions](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: the VCF table with determined SNV specificities; output of **Text reformatting with awk** {% icon tool %}
>    - *"With following condition"*: `c25=='CpGV-S'`
>    - *"Number of header lines to skip"*: `1`
>    - Click Run Tool
>
>    > <comment-title> Filtering the CpGV-S data </comment-title>
>    >
>    > The VCF SNV table is reduced to the data of CpGV-S. This is done by filtering on column 25 (ISOLATE).
>    {: .comment}
>
>
> 2. {% tool [Scatterplot with ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/3.4.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input in tabular format"*: the VCF table for CpGV-S; output of **Filter data on any column using simple expressions** {% icon tool %}
>    - *"Column to plot on x-axis"*: `2`
>    - *"Column to plot on y-axis"*: `32`
>    - *"Plot title"*: `Isolate CpGV-S`
>    - *"Label for x axis"*: `Reference Genome Position of CpGV-M`
>    - *"Label for y axis"*: `Relative Nucleotide Frequency`
>    - In *"Advanced options"*:
>        - *"Type of plot"*: `Points only (default)`
>            - *"Data point options"*: `Default`
>        - *"Plotting multiple groups"*: `Plot multiple groups of data on individual plots`
>            - *"column differentiating the different groups"*: `34`
>        - *"Axis title options"*: `Default`
>        - *"Axis text options"*: `Default`
>        - *"Plot title options"*: `Default`
>        - *"Grid lines"*: `Hide major and minor grid lines`
>        - *"Axis scaling"*: `Automatic axis scaling`
>    - In *"Output Options"*:
>        - *"Unit of output dimensions"*: `Centimeters (cm)`
>        - *"width of output"*: `30.0`
>        - *"height of output"*: `12.0`
>        - *"dpi of output"*: `200.0`
>    - Click Run Tool
>
{: .hands_on}

Let us now look at the result for isolate CpGV-S. We can see that all CpGV-E2 specific SNV positions have a frequency equal or close to zero. This makes sense, since we are looking at the isolate CpGV-S and the CpGV-E2 specific SNV markers should not indicate the presence of CpGV-E2. The situation is different for the SNV positions that are solely or in combination with another isolate specific for CpGV-S: here the frequency is close to 1. This also makes sense, since these markers indicate the presence of CpGV-S, which is the case. The few SNV positions that are specific for CpGV-M also do not indicate the presence of CpGV-M.  
![SNV specificity plot for CpGV-S](../../images/baculovirus-iso

tools_json:
["NCBI Accession Download", "toolshed.g2.bx.psu.edu/repos/iuc/ncbi_acc_download/ncbi_acc_download/0.2.8+galaxy0", "Collapse Collection", "toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0", "Replace Text in entire line", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.3+galaxy1", "Download and Extract Reads in FASTQ format from NCBI SRA", "toolshed.g2.bx.psu.edu/repos/iuc/sra_tools/fastq_dump/3.1.1+galaxy1", "Trim Galore!", "toolshed.g2.bx.psu.edu/repos/bgruening/trim_galore/trim_galore/0.6.7+galaxy0", "Map with BWA-MEM", "toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.18", "bcftools mpileup", "toolshed.g2.bx.psu.edu/repos/iuc/bcftools_mpileup/bcftools_mpileup/1.15.1+galaxy4", "bcftools call", "toolshed.g2.bx.psu.edu/repos/iuc/bcftools_call/bcftools_call/1.15.1+galaxy5", "VCFtoTab-delimited:", "toolshed.g2.bx.psu.edu/repos/devteam/vcf2tsv/vcf2tsv/1.0.0_rc1+galaxy0", "Text reformatting with awk", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_awk_tool/9.3+galaxy1", "Replace Text in a specific column", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1", "Filter data on any column using simple expressions", "Filter1", "Scatterplot with ggplot2", "toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/3.4.0+galaxy1"]


topic: variant-analysis
tutorial: beacon_cnv_query
title: Querying the University of Bradford GDC Beacon Database for Copy Number Variants (CNVs)
difficulty: introductory
time_estimation: 30M
tools_json: ["Beacon2 CNV", "toolshed.g2.bx.psu.edu/repos/iuc/beacon2_cnv/beacon2_cnv/2.1.1+galaxy0"]
requirements_json: []
tags_json: []
content_hash: 868e84ae1c495a3f073eabf4e676ffa0
last_modified: 2026-05-15T22:28:17.715102
zenodo_link: https://zenodo.org/records/10658688
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/beacon_cnv_query/tutorial.html

content:
The concept of a genomics "Beacon" refers to facilitating connection between genomic data suppliers, developers, and researchers interested in acquiring genetic variation data. The Beacon system was intended to be simple: an API that allows users to query genomic data collections for the presence of specified genetic variations and receive a simple "Yes" or "No" response. The term "Beacon" was chosen to represent the goal of illuminating the hitherto opaque world of genetic data sharing through widespread engagement. The Beacon Project, which became one of the initial Global Alliance for Genomics and Health (GA4GH) Driver Projects, was warmly welcomed by both members of the GA4GH developer community and genomic resource providers {% cite Rambla2022 %}.

By 2016, over 35 institutions worldwide had adopted the Beacon protocol across more than 90 genomic datasets. At this point, ELIXIR, the European bioinformatics infrastructure organization, became interested in moving the community initiative toward a defined definition. Their efforts were focused on improving usability to encourage wider adoption of the Beacon system within the biomedical genomics industry.

The Beacon v1.0 protocol was formally accepted as a GA4GH standard in 2018 {% cite Fiume2019 %}. Although this version included new features, such as limited support for structural variant inquiries, quantitative replies, and interaction with external protocols, it mostly stuck to the concept of querying variations and obtaining aggregate results. However, it quickly became clear that the procedure required significant extension, particularly to fulfil the objectives of clinical applications in rare diseases and cancer genomics. This understanding prompted the start of the Beacon v2 design process, which attempted to reconfigure the protocol to support a wider range of use cases.

Copy number variations (CNVs) are genomic segments with different numbers of copies. These variations, including DNA sequence amplifications and deletions, have been found in every domain of life, including bacteria, archaea, plants, and mammals. CNVs contribute significantly to genomic variety and can play an important role in promoting fast adaptive evolution and the development of inherited and somatic human illnesses such as cancer {% cite Lauer2019 %}.

Subsequently, The classification of CNV gains and losses into more specific categories: copy number gain, low-level copy number gain, high-level copy number gain, focal genome amplification, copy number loss, low-level copy number loss, high-level copy number loss, and complete genomic deletion. This detailed categorization enhances the precision in identifying and describing CNVs. More information can be found on the [ELIXIR hCNVs resource page](https://cnvar.org/resources/CNV-annotation-standards/).

In this tutorial, we will use the Beacon2 Query tool to explore a Beacon built from the [GDC Public Access CNVs Data](https://portal.gdc.cancer.gov/) to identify Copy Number Variants (CNVs). We will also perform targeted queries on this dataset using various search criteria to pinpoint specific data points. This tutorial will showcase the tool's versatility and effectiveness in exploring and analyzing genomic data.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Querying the Beacon Database to Retrieve and Display CNV Records for Specific Genomic Locations

A common use case for querying Copy Number Variation (CNV) records is when researchers are interested in identifying CNVs within a targeted chromosome or at a specific genomic location within that chromosome.

To conduct this type of query, you need to specify parameters that precisely define the desired genomic region in the Beacon database.

Those parametars are, "CHROMOSOME", "Start", and "End".


> <hands-on-title>Query the Beacon MongoDB</hands-on-title>
> 1. For this tutorial, make a new history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Use {% tool [Beacon2 CNV](toolshed.g2.bx.psu.edu/repos/iuc/beacon2_cnv/beacon2_cnv/2.1.1+galaxy0) %} to query the Beacon **genomicVariations** collection
>   - *"DATABASE HOST"*: `20.108.51.167`
>   - *"DATABASE PORT"*: `27017`
>   - *"DATABASE"*: `beacon`
>   - *"COLLECTION"*: `genomicVariations`
>   - *"CHROMOSOME"*: `1`
>   - *"START"*: `1574102`
>   - *"CHROMOSOME"*: `1674102`
>
> The search function will query the Beacon database to retrieve and display results located on chromosome 1, specifically within the genomic range between positions 1,574,102 and 1,674,102.
> After the query, review the output file to determine how many records match these specific criteria.
>
> > <question-title></question-title>
>    >
>    > What types of information can be extracted from records?
>    >
>    > ```json
>    > {'_id': ObjectId('66c466431ea6cb4184ee0f2f'),
>    > 'assemblyId': 'GRCh38',
>    > 'biosampleId': 'MP2PRT-PARNFH-TMP1-A, MP2PRT-PARNFH-NM1-A',
>    > 'definitions': {'Location': {'chromosome': '17',
>    >                              'end': 43170245,
>    >                              'start': 43044295}},
>    > 'diseaseType': 'acute lymphoblastic leukemia',
>    > 'gene': 'BRCA1',
>    > 'geneID': 'ENSG00000012048.23',
>    > 'id': 'refvar-66c466431ea6cb4184ee0f2f',
>    > 'info': {'caseID': 'MP2PRT-PARNFH, MP2PRT-PARNFH',
>    >          'cnCount': 3,
>    >          'fileName': 'f11b7fb7-a610-4978-b5c4-523450a0fd5f.wgs.ASCAT.gene_level.copy_number_variation.tsv',
>    >          'legacyId': 'DUP:chr17:43044295-43170245',
>    >          'projectID': 'MP2PRT-ALL',
>    >          'sampleType': 'Blood Derived Cancer - Bone Marrow, Blood Derived '
>    >                        'Cancer - Bone Marrow, Post-treatment'},
>    > 'primarySite': 'hematopoietic and reticuloendothelial systems',
>    > 'updated': '2024-08-19T21:23:09.374531',
>    > 'variantInternalId': '17:43044295-43170245:EFO:0030071',
>    > 'variantState': {'id': 'EFO:0030071', 'label': 'low-level gain'},
>    > 'variantType': 'DUP'}
>    > ```
>    >
>    > > <solution-title></solution-title>
>    > > 1. Identifiers and IDs:
>    > > - _id: The unique identifier for the record, typically used in databases.
>    > > - assemblyId: The reference genome assembly (e.g., 'GRCh38').
>    > > - biosampleId: IDs of the biological samples involved.
>    > > - geneID: Identifier for the gene (e.g., 'ENSG00000012048.23').
>    > > - id: Another unique identifier, possibly for external reference (e.g., 'refvar-66c466431ea6cb4184ee0f2f').
>    > > - caseID: Identifier(s) for the case(s) associated with this record.
>    > > - projectID: Identifier for the project (e.g., 'MP2PRT-ALL').
>    > > - variantInternalId: Internal identifier for the variant (e.g., '17:43044295-43170245:EFO:0030071').
>    > > 2. Genomic Information:
>    > > - gene: The gene involved (e.g., 'BRCA1').
>    > > - variantType: The type of genetic variant (e.g., 'DUP' for duplication).
>    > > - variantState: Describes the state of the variant, including an ID and label (e.g., 'low-level gain').
>    > > 3. Location Information:
>    > > - Location: Contains the chromosomal location of the variant:
>    > > - chromosome: Chromosome number (e.g., '17').
>    > > - start: Start position on the chromosome.
>    > > - end: End position on the chromosome.
>    > > 4. Disease and Clinical Information:
>    > > - diseaseType: The type of disease associated with the record (e.g., 'acute lymphoblastic leukemia').
>    > > - primarySite: The primary site of the disease (e.g., 'hematopoietic and reticuloendothelial systems').
>    > > 5. Sample Information:
>    > > - sampleType: Description of the sample type (e.g., 'Blood Derived Cancer - Bone Marrow').
>    > > - cnCount: Copy number count, which may indicate the number of copies of the variant.
>    > > 6. File and Legacy Information:
>    > > - fileName: Name of the associated file (e.g., 'f11b7fb7-a610-4978-b5c4-523450a0fd5f.wgs.ASCAT.gene_level.copy_number_variation.tsv').
>    > > - legacyId: A legacy identifier, possibly for reference to previous systems or records (e.g., 'DUP:chr17:43044295-43170245').
>    > > 7. Metadata:
>    > > - updated: Timestamp indicating when the record was last updated (e.g., '2024-08-19T21:23:09.374531').
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}



# Querying the Beacon Database Collection for a specific CNVs.

Suppose we are searching for specific variants in the database—for instance, records related to a low-level gain in a particular genomic location.

Building on the previous example, we will add the "VARIANT STATE" parameter to the query filter to refine our search and retrieve the exact variants we are interested in.


> <hands-on-title>Query the Beacon MongoDB</hands-on-title>
>
> 1. Use {% tool [Beacon2 CNV](toolshed.g2.bx.psu.edu/repos/iuc/beacon2_cnv/beacon2_cnv/2.1.1+galaxy0) %} to query the Beacon **genomicVariations** collection
>   - *"DATABASE HOST"*: `20.108.51.167`
>   - *"DATABASE PORT"*: `27017`
>   - *"DATABASE"*: `beacon`
>   - *"COLLECTION"*: `genomicVariations`
>   - *"CHROMOSOME"*: `1`
>   - *"START"*: `1574102`
>   - *"END"*: `1674102`
>   - *"*"VARIANT STATE"*"*: `low-level gain`
>
> This will print out the low-level gain CNV records located on chromosome 1 within the genomic range between positions 1,574,102 and 1,674,102.
{: .hands_on}


# Querying the Beacon Database Collection for Specific CNVs in a Targeted Gene.

Suppose we are interested in identifying specific Copy Number Variations (CNVs) within a particular gene in the Beacon database. For example, we might be looking for records related to deletions or duplications within a gene known to be associated with a specific disease or condition.

To achieve this, we will refine our query by focusing on a targeted gene and using the relevant parameters—such as "GENE NAME" and "VARIANT TYPE"—to filter the database records. This approach will help us retrieve CNV records that match our specific criteria, allowing us to analyze variations within the targeted gene effectively.


> <hands-on-title>Query the Beacon MongoDB</hands-on-title>
>
> 1. Use {% tool [Beacon2 CNV](toolshed.g2.bx.psu.edu/repos/iuc/beacon2_cnv/beacon2_cnv/2.1.1+galaxy0) %} to query the Beacon **genomicVariations** collection
>   - *"DATABASE HOST"*: `20.108.51.167`
>   - *"DATABASE PORT"*: `27017`
>   - *"DATABASE"*: `beacon`
>   - *"COLLECTION"*: `genomicVariations`
>   - *"VARIANT STATE"*: `low-level gain`
>   - *"GENE NAME"*: `ASMT`
>
> This query will retrieve and print CNV records associated with low-level gain within the ASMT gene from the Beacon database. By specifying the targeted gene name and variant state, the query will focus on CNVs that are particularly relevant to conditions associated with this gene.
{: .hands_on}


# Querying the Beacon Database Collection for Specific CNVs in a Targeted Gene Based on Primary Site or Disease Type.

In genomic research, it is often essential to focus on specific Copy Number Variations (CNVs) that are not only located within a particular gene but also associated with certain clinical features, such as a primary site of a tumour or a specific disease type. For instance, researchers might be interested in identifying CNVs in the BRCA1 gene that are linked to breast cancer or in the TP53 gene associated with various types of cancers.

To conduct such a detailed analysis, we can query the Beacon database by narrowing down the search to CNVs within a targeted gene, while simultaneously filtering the results based on relevant clinical parameters such as the primary site of the disease or the type of disease. This approach allows us to extract highly specific data that can provide deeper insights into the genetic underpinnings of particular conditions or cancer types, facilitating targeted research and potential clinical applications.


> <hands-on-title>Query the Beacon MongoDB</hands-on-title>
>
> 1. **Import the Excel sheet** containing information about the primary sites and diseases from the Beacon database, available at this [Zenodo link](https://zenodo.org/records/13348132/files/Beacon2-list-of-files.xlsx):
>
>    ```
>    https://zenodo.org/records/13348132/files/Beacon2-list-of-files.xlsx
>    ```
>
>    Please review this list to identify the primary sites and diseases that have available CNV (Copy Number Variation) records. In some cases, the same dataset can be found in the Galaxy shared data library. Ask the instructor for more details about this. The dataset can also be downloaded to local storage.
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="xlsx" %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Use the {% tool [Beacon2 CNV](toolshed.g2.bx.psu.edu/repos/iuc/beacon2_cnv/beacon2_cnv/2.1.1+galaxy0) %} to query the Beacon **genomicVariations** collection:
>    - *"DATABASE HOST"*: `20.108.51.167`
>    - *"DATABASE PORT"*: `27017`
>    - *"DATABASE"*: `beacon`
>    - *"COLLECTION"*: `genomicVariations`
>    - *"VARIANT STATE"*: `low-level loss`
>    - *"GENE NAME"*: `BRCA1`
>    - *"PRIMARY SITE"*: `breast`
>    - *"DISEASE TYPE"*: `adnexal and skin appendage neoplasms` *(optional)*
>
>    This query will retrieve and print CNV records associated with a low-level loss within the BRCA1 gene from the Beacon database. By specifying the gene name, variant state, and filtering based on the primary site as "breast," the query will focus on CNVs that are highly relevant to breast cancer. Additionally, you can refine the search further by including the disease type, such as "adnexal and skin appendage neoplasms," to narrow down the results to those most pertinent to your research objectives.
{: .hands_on}




In this tutorial, we used our University of Bradford Beacon server. To query a specific Beacon, update your credentials in the Galaxy user preferences and enter your server's details by adjusting the "DATABASE HOST" and "DATABASE PORT" fields accordingly.

> <comment-title>Use Credentials to Access Specific Beacon</comment-title>
> 1. Make sure you are logged in to Galaxy.
> 2. Go to **User > Preferences** in the top menu bar.
> 3. To add beacon database credentials, click on **Manage Information** and fill in the Beacon2 Account empty fields `db_auth_source`, `db_user` and `db_password`.
> 4. Make the changes and click on the **Save** button at the bottom.
{: .comment}

# Conclusion

Now, you have a general knowledge of Beacons and how to query them.

You can apply what you learned in this tutorial to query Beacon made by your institution or other institutions.

We hope you find this tutorial helpful!


topic: variant-analysis
tutorial: tb-variant-analysis
title: M. tuberculosis Variant Analysis
difficulty: intermediate
time_estimation: 2h
requirements_json: []
tags_json: ["prokaryote", "one-health", "microgalaxy", "tuberculosis"]
content_hash: 041af69a74ca6a98ec21586117f89c73
last_modified: 2026-05-27T09:01:50.019882
zenodo_link: https://doi.org/10.5281/zenodo.3496437
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/tb-variant-analysis/tutorial.html

content:
Tuberculosis (TB) is an infectious disease caused by the bacterium _Mycobacterium tuberculosis_. According to the [WHO](https://www.who.int/tb/publications/global_report/en/), in 2018 there were 10.0 million new cases of TB worldwide and 1.4 million deaths due to the disease, making TB the world's most deadly infectious disease. The [publication](https://www.ncbi.nlm.nih.gov/pubmed/9634230) of the genome of _M. tuberculosis H37Rv_ in 1998 gave researchers a powerful new tool for understanding this pathogen. This genome has been revised since then, with the latest version being available
as RefSeq entry [NC_000962.3](https://www.ncbi.nlm.nih.gov/nuccore/NC_000962.3/). The genome comprises a single circular chromosome of some 4.4 megabases. The H37Rv strain that the genome was sequenced from is a long-preserved laboratory strain, originally [isolated](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2132400) from a patient in 1905 and [named](https://journals.sagepub.com/doi/abs/10.3181/00379727-33-8330P) as H37Rv in 1935. It is notably different in some genomic [regions](https://www.sciencedirect.com/science/article/pii/S0888754317300617?via%3Dihub) from some modern clinical strains but remains the standard reference sequence for _M. tuberculosis_ (Mtb). In a larger context, _M. tuberculosis_ is a prominent member of the Mycobacterium Tuberculosis Complex (MTBC).

This group of related species comprises the [10](https://wwwnc.cdc.gov/eid/article/30/3/23-1466_article) [lineages](https://www.ncbi.nlm.nih.gov/pubmed/29456241) of human-infecting _M. tuberculosis_ as well as predominantly animal-infecting species such as *M. bovis* and *M. pinnipedii*. Two other close relatives of Mtb, _M. leprae_ and _M. lepromatosis_ circulate between humans, causing the disease leprosy. Finally, amongst the Mycobacteria there are several other species that live in the environment and can cause human disease. These are the [Nontuberculous Mycobacteria](https://www.ncbi.nlm.nih.gov/pubmed/28345639).

Variation in the genome of _M. tuberculosis_ (Mtb) is associated with changes in phenotype, for example, [drug resistance](https://genomemedicine.biomedcentral.com/articles/10.1186/s13073-019-0660-8) and [virulence](https://www.jci.org/articles/view/173156). It is also useful for [outbreak investigation](https://www.frontiersin.org/articles/10.3389/fpubh.2019.00087/full) as the single nucleotide polymorphisms (SNPs) in a sample can be used to build a phylogeny.

This tutorial will focus on identifying genomic variation in Mtb and using that to explore drug resistance and other aspects of the bacteria.

# Get your data

The data for today is a sample of _M. tuberculosis_ [collected](https://www.ncbi.nlm.nih.gov/bioproject/PRJEB18529) from a [southern African patient](https://bmcmedicine.biomedcentral.com/articles/10.1186/s12916-017-0834-4). In addition to the bacterial sequence sample, we will work with a Genbank format version of the genome of the [inferred](https://www.nature.com/articles/ng.590) most recent common [ancestor](https://zenodo.org/record/3497110) of the M. tuberculosis complex which is combined with the annotation of the H37Rv reference sequence. This ancestral genome only differs from the H37Rv version 3 genome ([NC_000962.3](https://www.ncbi.nlm.nih.gov/nuccore/NC_000962.3)) by the insertion of SNPs to try and model the ancestor of all lineages of Mtb.

> <hands-on-title>Get the data</hands-on-title>
>
> 1. {% tool [Import](upload1) %} the following files from [Zenodo](https://doi.org/10.5281/zenodo.3960260) or from the shared data library
>```
>https://zenodo.org/record/3960260/files/004-2_1.fastq.gz
>https://zenodo.org/record/3960260/files/004-2_2.fastq.gz
>https://zenodo.org/record/3960260/files/Mycobacterium_tuberculosis_ancestral_reference.gbk
>https://zenodo.org/record/3960260/files/Mycobacterium_tuberculosis_h37rv.ASM19595v2.45.chromosome.Chromosome.gff3
>```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Create a paired collection named `Paired Reads` containing the `004-2_1.fastq.gz` and `004-2_2.fastq.gz` datasets.
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
> 
{: .hands_on}

# Quality control

This step serves the purpose of identifying possible issues with the raw
sequenced reads input data before embarking on any "real" analysis steps.

Some of the typical problems with NGS data can be mitigated by preprocessing
affected sequencing reads before trying to map them to the reference genome.
Detecting some other, more severe problems early on may at least save you a lot
of time spent on analyzing low-quality data that is not worth the effort.

Here, we will perform a standard quality check on our input data and only point
out a few interesting aspects of that data. For a more thorough explanation
of NGS data quality control, you may want to have a look at the dedicated
tutorial on ["Quality control"]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).

> <hands-on-title>Quality control of the input datasets</hands-on-title>
>
> 1. Run {% tool [Flatten collection](__FLATTEN__) %} with the following parameters:
>    - *"Input collection"*: `Paired Reads`
>   
> 2. Rename the flatten collection: `Flat Collection`
>   
> 3. Run {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy1) %} with the following parameters:
>    - {% icon param-collection %} *"Raw read data from your current history"*: `Flat Collection` (Flattened paired end read dataset collection)
>
>    The **FastQC** {% icon tool %} input form looks like this. You only need to pay attention to the top part
>    where *Raw read data from your current history* is selected. Leave all the other parameters at their default
>    values and click *Execute*.
>
>    ![FastQC input and dependencies](../../images/mt_qc.png)
>
>    When you start this job, four new datasets (one with the calculated raw
>    data, another one with an html report of the findings for each input
>    dataset) will get added to your history.
>
{: .hands_on}

While one could examine the quality control report for each set of reads (forward and reverse) independently it can be quite useful to inspect them side by side using the **MultiQC** tool.

> <hands-on-title>Combining QC results</hands-on-title>
>
> 1. Use {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy4) %} to aggregate the raw **FastQC** data of all input datasets into one report
>      - In *"Results"*
>        - *"Which tool was used generate logs?"*: `FastQC`
>        - In *"FastQC output"*
>           - *"Type of FastQC output?"*: `Raw data`
>           - {% icon param-collection %} *"FastQC output"*: *RawData* collection output of **FastQC** {% icon tool %}
>
> 2. Using the {% icon galaxy-eye %} button, inspect the *Webpage* output produced by the tool
>
>    > <question-title></question-title>
>    >
>    > 1. Based on the report, do you think preprocessing of the reads
>    >    (trimming and/or filtering) will be necessary before mapping?
>    >
>    > 2. What is the average GC content of the data (known as GC%) in the `004-2_1` dataset?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Sequence quality is quite good overall. If anything you might
>    > >    consider trimming the 3' ends of reads (base qualities decline
>    > >    slightly towards the 3' ends) or to filter out the small fraction
>    > >    of reads with a mean base quality < 5.
>    > >    We will run **fastp** {% icon tool %} on the
>    > >    fastq datasets in the next step
>    > >
>    > > 2. The GC% is 66%, which is close to the 65.6% that one expects from a _M. tuberculosis_ sample.
>    > >    Examining the GC% is a quick way to check that the sample you have sequenced contains reads
>    > >    from the organism that you expect.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

As these reads look like they need a bit of trimming, we can turn to the **fastp** tool to clean up our data.

> <hands-on-title>Quality trimming</hands-on-title>
> 
> 1. Use {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/1.0.1+galaxy3) %} to clean up the reads and remove the poor quality sections.
>       - *"Single-end or paired-end reads?"*: `Paired Collection`
>       - *"Select paired collection(s) "*: `Paired Reads`
>
> 2. Inspect the output produced by **fastp**
>
>    > <question-title></question-title>
>     >
>    > 1. Was there a difference in file size of the reads before and after trimming? Is the difference the same between both sets of reads?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. After processing by **fastp** the output files are about 80 MB smaller than the inputs. The change in size is similar for both files.
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

The next section, on looking for contamination in our data using Kraken2, takes a long time to run and can be skipped and perhaps
run later if you prefer.

{% include _includes/cyoa-choices.html option1="Skip Kraken2" option2="Run Kraken2" default="Skip Kraken2" %}

<div class="Skip-Kraken2" markdown="1" >
</div>

<div class="Run-Kraken2" markdown="1" >

# Look for contamination with Kraken2

We should also look for contamination in our reads. Sometimes, other sources of DNA accidentally or inadvertently get mixed in with our sample. Any reads from non-sample sources will confound our SNP analysis. **Kraken2** is an effective way of looking at which species is represented in our reads so we can easily spot possible contamination of our sample. Unfortunately, the tool uses a lot of RAM (typically 50GB when used with the *Standard* database), so you might want to skip this step if your environment doesn't have enough computing nodes able to process such jobs. For an example of a probably-contaminated sample that does not use **Kraken2** as part of its analysis, see the optional section on analysing *SRR12416842* at the end of this tutorial.

> <hands-on-title>Run Kraken2</hands-on-title>
>
> 1. Execute {% tool [Kraken2](toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.17.1+galaxy0) %} with the following parameters
>   - *"Single or paired reads"*: `Paired`
>       - *"Collection of paired reads"*: `fastp on X: Paired-end output`
>
>   - *"Print scientific names instead of just taxids"*: `Yes`
>   - *"Enable quick operation"*: `Yes`
>   - *"Print a report with aggregrate counts/clade to file"*: `Yes`
>   - *"Select a Kraken2 database"*: `Standard`
>
> 2. Inspect the report produced by Kraken
>
>    > <question-title></question-title>
>    >
>    > 1. Was there any significant contamination of the sample?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Over 85% of the reads here have been positively identified as *Mycobacterium* (the precise % will differ depending on which version of the Kraken database you are using). The others found were bacteria from the same kingdom. There were no contaminating human or viral sequences detected.
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

</div>

# Find variants with Snippy

We will now run the Snippy tool on our reads, comparing them to the reference.

[Snippy](https://github.com/tseemann/snippy) is a tool for rapid bacterial SNP calling and core genome alignments. Snippy finds SNPs between a haploid reference genome and your NGS sequence reads. It will find both substitutions (SNPs) and insertions/deletions (indels).

If we give Snippy an annotated reference in Genbank format, it will run a tool called SnpEff which will figure out the effect of any changes on the genes and other features. If we just give Snippy the reference sequence alone without the annotations, it will not run SnpEff.

We have an annotated reference built from the inferred _M. tuberculosis_ [ancestral reference genome](https://zenodo.org/record/3497110) and the
gene annotation from the [H37Rv strain](https://www.ncbi.nlm.nih.gov/nuccore/NC_000962.3) so will use it in this case.

> <hands-on-title>Run Snippy</hands-on-title>
>
> 1. {% tool [Snippy](toolshed.g2.bx.psu.edu/repos/iuc/snippy/snippy/4.6.0+galaxy0) %} with the following parameters
>   - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>   - *"Use the following dataset as the reference sequence"*: `Mycobacterium_tuberculosis_ancestral_reference.gbk`
>   - *"Input type"*: `Paired end reads in a collection`
>       - *"Select a paired collection"*: `fastp on X: Paired-end output`
>
>   - Under *"Advanced parameters"*
>       - *"Minimum proportion for variant evidence"*: `0.1` (This is so we can see possible rare variants in our sample)
>   - Under *"Output selection"* select the following:
>       - *"The final annotated variants in VCF format"*
>       - *"A simple tab-separated summary of all the variants"*
>       - *"The alignments in BAM format"*
>       - Deselect any others.
>
> 2. Inspect the Snippy VCF output
>
>    > <question-title></question-title>
>    >
>    > 1. What type of variant is the first one in the list?
>    >
>    > 2. What was the effect of this variant on the coding region it was found in?
>    >
>    > 3. How many variants were found?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Substitution of a `C` to a `T`. This variant is supported by 197 reads.
>    > >
>    > > 2. According to SnpEff, it's a Synonymous change in Rv0002.
>    > >
>    > > 3. 1098 variants are found. To count variants, look at how many non-comment lines are in the snippy VCF output or how many lines (excluding the header) there are in the VCF file. This is quite typical for _M. tuberculosis_.
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

**RECAP**: So far we have taken our sample reads, cleaned them up a bit, checked for taxonomic assocation, compared the reads with our reference sequence and then called variants (SNPs and indels) between our sample and the reference genome. We have tried to mitigate a few errors along the way:

1. Sequencing errors: these were addressed by the quality trimming step
2. Sample contamination: we used **Kraken2** to assess the extent of this problem in our sample
3. Appropriate choice of a reference genome: we used a genome that is inferred to be ancestral to all _M. tuberculosis_ for our analysis and the diversity within Mtb is limited enough for us to rely on a single reference genome for the entire species.
4. Quality filtering in the mapping and variant calling stage: Internally `snippy` uses tools like `bwa-mem` and `freebayes` that judge the quality of their predictions. `snippy` then uses this information to perform some filtering on variant calling predictions.

# Further variant filtering and drug resistance profiling

We still cannot entirely trust the proposed variants. In particular, there are regions of the _M. tuberculosis_ genome that are difficult to effectively map reads to. These include the PE/PPE/PGRS genes, which are highly repetitive, and the IS (insertion sequence sites). Secondly, when an insertion or deletion (indel) occurs in our sample relative to the reference it can cause apparent, but false, single nucleotide variants to appear near the indel. Finally, where few reads map to a region of the reference genome, either because of a sequence deletion or because of a high GC content in the genomic region, we cannot be confident about the quality of variant calling in the region. The `TB Variant Filter` can help filter out variants based on a variety of criteria, including those listed above.

> <hands-on-title>Run TB Variant Filter</hands-on-title>
> 1. {% tool [TB Variant Filter](toolshed.g2.bx.psu.edu/repos/iuc/tb_variant_filter/tb_variant_filter/0.4.0+galaxy0) %} with the following parameters
>   - *"VCF file to be filter"*: `snippy on collection XX snps vcf file`
>   - *"Filters to apply"*: Select `Filter variants by region` and `Filter sites by read alignment depth`.
>
> 2. Open the new VCF file.
>
>    > <question-title></question-title>
>    >
>    > 1. How many of the original variants have now been filtered out?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. `131` (The difference in the number of lines between the snippy vcf file and the filtered vcf file.)
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

> <details-title>Which filters to apply?</details-title>
>
> TB Variant Filter tries to provide reasonable defaults for filtering
> variants predicted in the M. tuberculosis
> genome, using multiple different strategies.
> Firstly, certain regions of the Mtb genome
> contain repetitive sequences, e.g. from
> the PE/PPE gene family. Historically all of the genomic regions corresponding to
> those genes were filtered out but
> the new default draws on work from
> Maximillian Marin and others. This
> list of "refined low confidence" (RLC)
> regions is the current region filter in
> TB Variant Filter for reads over 100 bp.
> If you are using shorter reads (e.g. from Illumina iSeq) the "Refined Low Confidence and Low Mappability" region list should be used instead.
> For more on how these regions were calculated read the [paper](https://academic.oup.com/bioinformatics/article-abstract/38/7/1781/6502279?login=false) or [preprint](https://www.biorxiv.org/content/10.1101/2021.04.08.438862v3.full).
>
> In addition to region filters, filters for variant type, allele frequency, coverage depth and distance from indels are provided.
> Older variant callers struggled to accurately
> call insertions and deletions (indels) but more recent tools (e.g. GATK v4 and the variant caller used in Snippy, Freebayes) no longer have this weakness. One remaining reason to filter SNVs/SNPs near indels is that they might have a different
> evolutionary history to "free standing" SNVs/SNPs, so the "close to indel filter" is still available in TB Variant Filter in case such SNPs/SNVs should be filtered out.
{: .details}

Now that we have a collection of *high-quality variants* we can search them against variants known to be associated with drug resistance. The *TB Profiler* tool does this using a database of variants curated by Dr Jody Phelan at the London School of Hygiene and Tropical Medicine. It can do its own mapping and variant calling but also accepts mapped reads in BAM format as input. It does its own variant calling and filtering.

Finally, TB Variant Report uses the COMBAT-TB [eXplorer](https://explorer.sanbi.ac.za) [database](https://academic.oup.com/bioinformatics/advance-article/doi/10.1093/bioinformatics/btz658/5554700) of _M. tuberculosis_ genome annotation to annotate variants in Mtb. It also takes the output of *TB Profiler* and produces a neat report that is easy to browse and search.

> <hands-on-title>Run TB Profiler and TB Variant Report</hands-on-title>
> 1. {% tool [TB-Profiler profile](toolshed.g2.bx.psu.edu/repos/iuc/tbprofiler/tb_profiler_profile/6.6.4+galaxy0) %} with the following parameters
>   - *"Input File Type"*: `BAM`
>       - *"Bam"*: `snippy on collection XX mapped reads (bam)`
>
>
>       **TB Profiler** produces 3 output files, it's own VCF file, a report about the sample including it's likely lineages and any AMR found. There is also a `.json` formatted results file.
>
> 2. When *snippy* is run with Genbank format input it prepends `GENE_` to gene names in the VCF annotation. This causes a problem for *TB Variant report*, so we need to edit the output with sed.
>
>     {% tool [Text transformation with sed](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sed_tool/9.5+galaxy2) %} with the following parameters:
>
>       - *"File to process"*: `TB Variant Filter on data XX`
>       - *"SED Program"*: `s/GENE_//g`
>
> 3. {% tool [TB Variant Report](toolshed.g2.bx.psu.edu/repos/iuc/tbvcfreport/tbvcfreport/1.0.1+galaxy0) %} with the following parameters
>   - *"Input SnpEff annotated M.tuberculosis VCF(s)"*: `Text transformation on data XX`
>   - *"TBProfiler Drug Resistance Report (Optional)"*: `TB-Profiler Profile on data XX: Results.json`
>
> 4. Open the drug resistance and variant report html files.
>
>    > <question-title></question-title>
>    >
>    > 1. What was the final lineage of the sample we tested?
>    >
>    > 2. Were there any drug resistances found?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. `4` with sublineage `4.4.1.1.1`.
>    > >
>    > > 2. Yes, resistance to isoniazid, rifampicin, ethambutol, pyrazinamide and streptomycin as well as to the flouroquinolines (amikacin, capreomycin and kanamycin) is predicted from mutations in the katG, rpoB, embB, pncA, rpsL and rrs (ribosomal RNA) genes respectively.
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

At this point we have a drug resistance report from **TB Profiler**, aligned reads from **snippy** and variants both in VCF format
and in the easier-to-read format produced by **TB Variant report**. Often this is enough information on a sample, and we might end
our analysis here, especially if we are processing many samples together, as is described towards the end of this tutorial.

We will, however, spend some time examining this data in more detail in the next section.

# View Snippy output in JBrowse

We could go through all of the variants in the VCF files and read them out of a text table, but this is onerous and doesn't really give the context of the changes very well. It would be much nicer to have a visualisation of the SNPs and the other relevant data. A genome viewer, such as JBrowse, can be used within Galaxy to display the _M. tuberculosis_ genome and the data from our analysis.

> <hands-on-title>Run JBrowse</hands-on-title>
>
> <!-- GTN:IGNORE:006 -->
> 1. Use {% tool [seqret](toolshed.g2.bx.psu.edu/repos/devteam/emboss_5/EMBOSS:%20seqret84/5.0.0) %} to convert the Genbank format reference (`Mycobacterium_tuberculosis_ancestral_reference.gbk`) to FASTA format. Use the following parameters:
>    - *"Sequences"*: `Mycobacterium_tuberculosis_ancestral_reference.gbk`
>    - *"Use feature information"*: `No`
>    - *"Read one sequence and stop"*: `Yes`
>    - *"Output sequence file format"*: `FASTA (m)`
>
> 2. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1) %} with the following parameters
>    - *"Reference genome to display"*: `Use a genome from history`
>       - *"Select the reference genome"*: `seqret output from the previous step`
>
>       This sequence will be the reference against which annotations are displayed
>
>    - *"Genetic Code"*: `11: The Bacterial, Archaeal and Plant Plastid Code`
>    - *"JBrowse-in-Galaxy Action"*: `New JBrowse Instance`
>    - *"Track Group"*
>
>        We will now set up three different tracks - these are datasets displayed underneath the reference sequence (which is displayed as nucleotides in FASTA format). We will choose to display the sequence reads (the .bam file), the variants found by snippy (the .gff file) and the annotated reference genome (the wildtype.gff)
>
>       - **Track 1 - sequence reads**: Click on `Insert Track Group` and fill it with
>           - "Track Category" to `sequence reads`
>           - Click on `Insert Annotation Track` and fill it with
>               - "Track Type" to `BAM Pileups`
>               - "BAM Track Data" to `snippy on collection XX mapped reads (bam)`
>               - "Autogenerate SNP Track" to `No`
>               - "Track Visibility" to `On for new users`
>       - **Track 2 - variants**: Click on `Insert Track Group` and fill it with
>           - "Track Category" to `variants`
>           - Click on `Insert Annotation Track` and fill it with
>               - "Track Type" to `VCF SNPs`
>               - "SNP Track Data" to `TB Variant Filter on data XX`
>               - "Track Visibility" to `On for new users`
>       - **Track 3 - annotated reference**: Click on `Insert Track Group` and fill it with
>           - "Track Category" to `annotated reference`
>           - Click on `Insert Annotation Track` and fill it with
>               - "Track Type" to `GFF/GFF3/BED Features`
>               - "GFF/GFF3/BED Track Data" to `Mycobacterium_tuberculosis_h37rv.ASM19595v2.45.chromosome.Chromosome.gff3`
>               - "JBrowse Track Type [Advanced]" to `Canvas Features`
>               - Click on "JBrowse Styling Options [Advanced]"
>               - "JBrowse style.label" to `product`
>               - "JBrowse style.description" to `product`
>               - "Track Visibility" to `On for new users`
{: .hands_on}

A new dataset will be created in your history, containing the JBrowse interactive visualisation. We will now view its contents and play with it by clicking the {% icon galaxy-eye %} (eye) icon of the `JBrowse on data XX and data XX - Complete` dataset. The JBrowse window will appear in the centre Galaxy panel.

What you should see is something like this:

{% snippet topics/visualisation/faqs/visualizations_jbrowse.html loc="Chromosome:761005..761306" datadir="data" %}

You can now click on the names of the tracks to add them in, try the vcf file and gff file. You can see where the variants are located and which genes they are in. If you click on the BAM file you can zoom right in to see the read alignments for each variant if you wish.

> <question-title>Using JBrowse to examine a known variant</question-title>
>
> 1. Paste `Chromosome:761009..761310` into the JBrowse location bar and click **Go**. Ensure that the BAM and VCF tracks are set to visible. What is the meaning of the red column that shows up in the middle of the display of reads (the BAM track)?
>
> 2. What change happened at position 761155 in the genome, according to the read and variant data displayed?
>
> 3. How did this change impact the *rpoB* gene, the gene that is found on the forward strand overlapping this position?
>
> > <solution-title></solution-title>
> >
> > 1. The red column is a variant that is present in almost all of the reads aligning to the genome at position 761155.
> >
> > 2. Examining the VCF track the variant is shown as a mutation from C to T.
> >
> > 3. The "frame" of the codons in this gene can be determined by navigating to where the *rpoB* gene, the second horizontal bar in the annotation track, starts (position 759807). This gene is in frame 0 on the forward strand, the top frame in the display. Navigating back to position 761155, the amino acid encoded in the reference genome is S, that is serine. The mutation from a C to a T changes the codon at this position to L, leucine, and is associated with resistance to the drug rifampicin.
> >
> >  While the rifampicin resistance mutation being examined here is already reported by **TB-Profiler**, JBrowse allows us to examine the evidence underlying this mutation in more detail.
> {: .solution}
{: .question}

An alternative to running JBrowse within Galaxy is to install [IGV](https://igv.org/) and use Galaxy's built-in support for visualising BAM files with IGV.

{% snippet  topics/galaxy-interface/faqs/visualisations_igv.md %}

This requires more setup for the user, for example loading the FASTA file of the genome into IGV and downloading any annotation GFF3 files you want to use with IGV. On the other hand, running IGV locally is often faster than using JBrowse in Galaxy.

# Different samples, different stories (optional)

In [Zenodo](https://doi.org/10.5281/zenodo.3960260) we have included sample *18-1* from the same study (aka. [ERR1750907](https://www.ebi.ac.uk/ena/browser/view/ERR1750907)). This is also a southern African
_M. tuberculosis_ sample, but in some ways quite different from the sample we have analysed in the tutorial thus
far.

> <hands-on-title>Take a closer look at sample 18-1</hands-on-title>
>
> 1. Fetch the data from Zenodo
>```
>https://zenodo.org/record/3960260/files/018-1_1.fastq.gz
>https://zenodo.org/record/3960260/files/018-1_2.fastq.gz
>```
>
> 2. Examine the sequence quality with {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy1) %}.
>
> 3. Examine the sample composition with {% tool [Kraken2](toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.17.1+galaxy0) %}.
>
>    > <question-title></question-title>
>    >
>    > 1. What problems were discovered with sequence quality?
>    >
>    > 2. What did the **Kraken2** report show? How does this impact your assessment of variants discovered from this sample?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The quality of the sequence drops sharply towards the end of the sequences. Even more concerning, the sequence content changes across the length of the sample, which is not what we would expect at all. Finally, the sample seems to contain sequencing adapters, an artefact of the sequencing process that should be trimmed out before any sequence analysis.
>    > >
>    > > 2. Less than 60% of the sequence reads are associated with the genus *Mycobacterium*. Perhaps the quality problems in the sequence reads contribute to this poor classification? They certainly will make variant calling less reliable. You might get too many (false positive) or too few (false negative) variants reported compared to what is actually present in the sample.
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

As you can see, the quality of sequence data strongly determines how useful it is for subsequent analysis. This is why quality control is always a first step before trying to call and interpret variants. What we do with a sample like this will depend on what resources we have available. Can we discard it and use other data for our analysis? Can we re-sequence? Can we clean it up, remove the adapters (using **Trimmomatic**, **fastp** or **cutadapt**) and perhaps use the **Kraken2** output to decide which reads to keep? These are all possible strategies and there is no one answer for which is the correct one to pursue.

The next example is *SRR12416842* from an Indonesia [study](https://www.microbiologyresearch.org/content/journal/jmm/10.1099/jmm.0.001221) of multi-drug resistant (MDR) tuberculosis.

> <hands-on-title>Take a closer look at sample SRR12416842</hands-on-title>
>
> 1. Fetch the data from EBI European Nucleotide Archive
>```
>ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR124/042/SRR12416842/SRR12416842_1.fastq.gz
>ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR124/042/SRR12416842/SRR12416842_2.fastq.gz
>```
>
> 2. Perform quality trimming with {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/1.0.1+galaxy3) %} and examine it's *HTML* output to see quality before and after trimming.
>
> 3. Map the samples to the _M. tuberculosis_ reference genome with {% tool [Snippy](toolshed.g2.bx.psu.edu/repos/iuc/snippy/snippy/4.6.0+galaxy0) %}. Make sure to select the BAM output as one of the outputs.
>
>    > <question-title></question-title>
>    >
>    > 1. Was the sequence quality good?
>    >
>    > 2. How many variants were discovered by snippy?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The **fastp** result shows that while there is some dropoff in sequence quality (especially towards the end of the reads from the second dataset), the sequences are of good enough quality to analyse.
>    > >
>    > > 2. **snippy** discovered more than 15,000 variants. This is unusual for a _M. tuberculosis_ sample where we expect at most a few thousand variants across the length of the genome.
>    > {: .solution}
>    {: .question}
>
> 4. Run {% tool [samtools stats](toolshed.g2.bx.psu.edu/repos/devteam/samtools_stats/samtools_stats/2.0.8) %} on the *snippy on collection XX mapped reads (bam)* file. In the output, pay attention to the *sequences*, *reads mapped* and *reads unmapped* results.
>
> 5. Run the {% tool [BAM Coverage Plotter](toolshed.g2.bx.psu.edu/repos/iuc/jvarkit_wgscoverageplotter/jvarkit_wgscoverageplotter/20201223+galaxy0) %} on the mapped reads BAM file that you got from **snippy** using the FASTA format reference you made with **seqret** as the reference.
>
>    > <question-title></question-title>
>    >
>    > 1. What percentage of reads mapped to the reference genome?
>    >
>    > 2. If you could run the **BAM Coverage Plotter** tool, was the coverage even across the genome?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Less than 110000 out of 7297618, that is 1.5%, of the reads mapped to the reference genome.
>    > >
>    > > 2. The image from the **BAM Coverage Plotter** tool shows just a few vertical bars, suggestion that almost no reads mapped to the reference genome.
>    > >
>    > > ![BAM Coverage Plot of SRR12416842 showing few reads mapped](../../images/mtb_poor_mapping.png)
>    > >
>    > > By contrast, reads from the `004-02` map evenly across the _M. tuberculosis_ genome, with an average depth of over 100 reads, as shown in this output from **BAM Coverage Plotter**:
>    > >
>    > > ![BAM Coverage Plot of 004-02 showing reads mapped evenly](../../images/mtb_good_mapping.png)
>    > >
>    > > If you wish to investigate further, analyse the SRR12416842 sample with **Kraken2**.
>    > {: .solution}
>    {: .question}
{: .hands_on}

There is something clearly wrong with sample SRR12416842, perhaps indicating sample contamination. This example of a sample that doesn't map to the reference genome illustrates that even when sequence quality is good, sequence data problems can become apparent in later steps of analysis and it is important to always have a sense of what results to expect. You can develop a better sense of what quality control results to expect by first practicing techniques with known data before analysing new samples.

# Processing many samples at once: Collections and Workflows (optional)

In the tutorial thus far we have focused on processing single samples, where two read datasets (forward and reverse reads) are associated with a single sample. In practice, sequence analysis typically involves analysing batches of samples and we run the same
analysis steps for each sample in the batch. Galaxy supports working with batches using collections and workflows.


If you have followed all of the steps of this tutorial, you will have two useable samples, one named *004-2* and another *018-1*. Create a list of dataset pairs from these samples and call the collection `samples`.

{% snippet faqs/galaxy/collections_build_list_paired.md %}

The workflow that we are going to use is published on [WorkflowHub.eu](https://workflowhub.eu/workflows/1035) and you can run it directly on Galaxy servers using the link below:

{% snippet faqs/galaxy/workflows_run_wfh.md title="TB Variant Analysis v1.0" wfhub_id="1035" version="1" %}

If the text above doesn't show a text to run or if you are not on one of the public "usegalaxy.*" servers that integrate with WorkflowHub.eu:

1. Copy the URL of [this link]({{ site.baseurl }}{{ page.dir }}workflows/tb-variant-analysis.ga) (e.g. via right click) or save it to your computer and
2. Import the workflow into Galaxy
3. Click the run icon (it is a small white right-pointing triangle on a blue background)

{% snippet faqs/galaxy/workflows_import.md %}

> <hands-on-title>Analysing samples with the TB Variant Reporting workflow</hands-on-title>
>
> 1. **Run the TB Variant Analysis workflow** {% icon workflow %} using the following parameters
>    - {% icon param-collection %} *Reads* the `samples` collection of your input reads
>
>    - {% icon param-file %} *Reference Genome* the `Mycobacterium_tuberculosis_ancestral_reference.gbk` reference genome
>
> 2. Click the **Run Workflow button** at the top-right of the screen
>
{: .hands_on}

The workflow will produce a series of collections, with the most important outputs being tagged with dataset tags.

* The **MultiQC report** tagged *qc_report* contains information on `fastp` read trimming and `BamQC` mapping statistics.

* The filtered VCF is transformed with `sed` and tagged as *annotated_vcf*

* The TB-Profiler drug resistance report is tagged with *drug_resistance_report*

* The TB VCF Report outputs are tagged with *variant_report*

* For each sample a consensus genome is created by inserting the single nucleotide variants into the reference genome. This is tagged as *consensus_genome* and is intended for use in building a phylogenetic tree.

* The Kraken2 report (to check for contamination) is tagged with *kraken_report*

The workflow also produces a Workflow Invocation Report that summarises the outputs of the workflow. This can be found on the Workflow Invocation menu which is either in the User menu or on the History menu depending on which version of Galaxy you are using.

# Conclusion

Thank you for coming to the end of this tutorial. After completing this tutorial you might want to follow some of the other tutorials in the Galaxy Training Network on analysing _M. tuberculosis_ data. Just search for *tuberculosis* in the search bar. Enjoy the rest of your day!

tools_json:
["Import", "upload1", "Flatten collection", "__FLATTEN__", "FastQC", "toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy1", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy4", "fastp", "toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/1.0.1+galaxy3", "Kraken2", "toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.17.1+galaxy0", "Snippy", "toolshed.g2.bx.psu.edu/repos/iuc/snippy/snippy/4.6.0+galaxy0", "TB Variant Filter", "toolshed.g2.bx.psu.edu/repos/iuc/tb_variant_filter/tb_variant_filter/0.4.0+galaxy0", "TB-Profiler profile", "toolshed.g2.bx.psu.edu/repos/iuc/tbprofiler/tb_profiler_profile/6.6.4+galaxy0", "Text transformation with sed", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sed_tool/9.5+galaxy2", "TB Variant Report", "toolshed.g2.bx.psu.edu/repos/iuc/tbvcfreport/tbvcfreport/1.0.1+galaxy0", "seqret", "toolshed.g2.bx.psu.edu/repos/devteam/emboss_5/EMBOSS:%20seqret84/5.0.0", "JBrowse", "toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.11+galaxy1", "samtools stats", "toolshed.g2.bx.psu.edu/repos/devteam/samtools_stats/samtools_stats/2.0.8", "BAM Coverage Plotter", "toolshed.g2.bx.psu.edu/repos/iuc/jvarkit_wgscoverageplotter/jvarkit_wgscoverageplotter/20201223+galaxy0"]


topic: variant-analysis
tutorial: dip
title: Calling variants in diploid systems
difficulty: intermediate
time_estimation: 3h
requirements_json: []
tags_json: []
content_hash: 4fc0b2d3cc622deebb39466e527b02cc
last_modified: 2026-05-27T09:01:50.016231
zenodo_link: https://doi.org/10.5281/zenodo.60520
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/dip/tutorial.html

content:
Today we hear a lot about personalized medicine. Yet the *personalization* is defined by the genetic make up of the individual. In this tutorial we will discuss how this information can be uncovered from the genomic sequencing data.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Calling variants

Variant calling is a complex field that was significantly propelled by advances in DNA sequencing and efforts of large scientific consortia such as the [1000 Genomes](http://www.1000genomes.org). Here we summarize basic ideas central to Genotype and Variant calling. First, let's contrast the two things although they often go together:

- **Variant calling** - identification of positions where the sequenced sample is different from the reference sequence (or [reference genome graph](https://github.com/vgteam/vg))
- **Genotype calling** - identifying individual's genotype at variable sites.

A typical workflow for variation discovery involves the following steps (e.g., see Nielsen et al. [2011](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3593722/)):

1. Mapping reads against the reference genome
2. Thresholding BAM datasets by, for example, retaining paired, properly mapped reads
3. Performing quality score recalibration
4. Performing realignment
5. Performing variant calling/genotype assignment
6. Performing filtering and genotype quality score recalibration
7. Annotating variants and performing downstream analyses

However, continuing evolution of variant detection methods has made some of these steps obsolete. For instance, omitting quality score recalibration and re-alignment (steps 3 and 4 above) when using haplotype-aware variant callers such as [FreeBayes](https://github.com/ekg/freebayes) does not have an effect on the resulting calls (see Brad Chapman's methodological comparisons at [bcbio](https://bit.ly/1S9kFJN)). Before going forward with an actual genotype calling in Galaxy let's take a look as some basic ideas behind modern variant callers.

## How does SNP calling and genotyping work?

Consider a set of sequencing reads derived from a diploid individual:

```
REFERENCE: atcatgacggcaGtagcatat
--------------------------------
READ1:     atcatgacggcaGtagcatat
READ2:         tgacggcaGtagcatat
READ3:     atcatgacggcaAtagca
READ4:            cggcaGtagcatat
READ5:     atcatgacggcaGtagc
```

The capitalized position contains a G &#8594; A [transition](https://en.wikipedia.org/wiki/Transition_(genetics)). So, in principle this can be a heterozygous site with two alleles **G** and **A**. A commonly used naïve procedure would define a site as *heterozygous* if there is a non-reference allele with frequency between 20% and 80%. In this case **A** is present in 1/5 or 20% of the cases, so we can say that this is a heterozygous site. Yet it is only represented by a single read and thus is hardly reliable. Here are some of the possibilities that would explain this *variant*. It can be:

- a true variant
- an experimental artifact, e.g. a library preparation error
- a base calling error
- an analysis error, e.g. a misalignment (though unlikely in the above example)

The modern variant callers attempt to assign a reliability estimate for each genotype call. This is done using Bayes reasoning (for a great visual explanation see [blog](https://oscarbonilla.com/2009/05/visualizing-bayes-theorem/) by Oscar Bonilla). Here we present a SNP-relevant "translation" on this explanation (with inspiration from [Erik Garrison](https://github.com/ekg)).

Suppose in a population you have $$A$$ individuals (not to be confused with nucleotide **A**; in this case $$A$$ is a number of individuals) with a variant. You are performing re-sequencing and observe a variant in $$B$$ (again, a number) of your sequencing reads. We want to estimate the probability of having the real polymorphism in the population given our observations in sequencing reads. The logic is as follows:

- The probability of having polymorphism *A* in the population is $$P(A) = \lvert A \rvert / \lvert U \rvert$$
- The probability of seeing a variant given our identification approach (i.e., sequencing) is $$P(B) = \lvert B \rvert / \lvert U \rvert$$

Now, the probability of having a variant and it being observed in our sequencing data is the overlap between $$A$$ and $$B$$ sets: $$P(AB) = \lvert AB \rvert / \lvert U \rvert$$. This is presented graphically below:

| Polymorphisms | Variant Calls | Polymorphisms and Variant Calls |
|:-------------:|:-------------:|:-------------------------------:|
| ![P(A) polymorphism](../../images/pA.png) | ![P(B) variant calls](../../images/pB.png) | ![P(AB) polymorphism and variant calls](../../images/pAB.png) |
| $$P(A)$$ | $$P(B)$$ | $$P(AB)$$ |

Now we can ask the following question: *What is the probability of a having a real polymorphism $$A$$ given our observation of variants in reads $$B$$?* In other words, *what is the probability of $$A$$ given $$B$$?* Or, as stated in the original [blog](https://oscarbonilla.com/2009/05/visualizing-bayes-theorem/): "*given that we are in region $$B$$, what is the probability that we are in the region $$AB$$?*"

$$P(A \vert B) = \frac{\lvert AB \rvert}{\lvert B \rvert}$$

Dividing both numerator and denominator by $$\lvert U \rvert$$:

$$P(A \vert B) = \frac{\frac{\lvert AB \rvert}{\lvert U \rvert}}{\frac{\lvert B \rvert}{\lvert U \rvert}}$$

Because we know that $$P(AB) = \frac{\lvert AB \rvert}{\lvert U \rvert}$$ and $$P(B) = \frac{\lvert B \rvert}{\lvert U \rvert}$$, we can rewrite the equation above as:

$$P(A \vert B) = \frac{P(AB)}{P(B)}$$

Now, let's ask an opposite question. Given a true polymorphism $$A$$, what are the chances that we do detect it (i.e. find ourselves in $$AB$$)? It will be:

$$P(B \vert A) = \frac{P(AB)}{P(A)}$$

So, because we know that $$P(A \vert B) = \frac{P(AB)}{P(B)}$$ and we just reasoned that $$P(B \vert A) = \frac{P(AB)}{P(A)}$$, we can say that $$P(A \vert B)P(B) = P(B \vert A)P(A)$$, leading us to the Bayes formula:

$$P(A \vert B) = \frac{P(B \vert A)P(A)}{P(B)}$$

Translating this into "genomics terms", the probability of having a genotype $$G$$ given sequencing reads $$S$$ is: $$P(G \vert S) = \frac{P(S \vert G)P(G)}{P(S)}$$. Because in a given calculation of $$P(G \vert S)$$ the reads are fixed, we can re-write the Bayes formula in the following way:

$$P(G \vert S) \sim P(S \vert G)P(G)$$

with $$P(S)$$ becoming a constant. This leaves us with the need to estimate two things:

1. $$P(S \vert G)$$, the data likelihood
2. $$P(G)$$, the prior probability for the variant.

In the simplest case we can estimate these as follows:

### $$P(S \vert G)$$

Suppose $$S_i$$ is a base in read $$i$$ corresponding to a genome position with genotype $$G$$. The probability of seeing $$S_i$$ given $$G$$, or $$P(S_i \vert G)$$, is given by the quality score of $$S_i$$ (the quality scores are given by base calling software and reported as [phred scores](https://en.wikipedia.org/wiki/Phred_quality_score)). Thus the genotype likelihood $$P(S \vert G)$$ is the product of $$P(S_i \vert G)$$ over all $$i$$. In reality however there are many other sources of uncertainty (in addition to base qualities) that are incorporated in the calculation of data likelihoods including NGS technology-related issues, dependency of error rates on substitution type (e.g., transitions versus transversions), sequencing context etc...

### $$P(G)$$ - a single sample case

One can assign an equal probability to all possible genotypes, or to source this information based on previously obtained knowledge containing in a database, such as [dbSNP](https://www.ncbi.nlm.nih.gov/SNP/). In this case (as exemplified in [Nielsen et al. 2011](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3593722/)) we may, for instance, have a site with a **G/T** polymorphism and genotypes **GG**, **TT**, and **GT** having frequencies of 0.45, 0.45, 0.09, respectively. We will use these values as priors.

### $$P(G)$$ - a multi-sample case

Genotype calling reliability can be significantly improved when analyzing multiple samples jointly. In this case genotype frequencies can be inferred from allele frequencies using Hardy-Weinberg equilibrium ([HWE](https://en.wikipedia.org/wiki/Hardy%E2%80%93Weinberg_principle)). The following example (again from [Nielsen et al. 2011](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3593722/)) illustrates this idea: suppose you are calling genotypes for a single individual using a combination of multiple samples. There are two genotypes, **AT** and **AA**, with equally large genotype likelihoods. If, however, in our collection of multiple samples the frequency of **A** is 1% ($$p = 0.01$$; $$q = 1 - p = 0.99$$), then from the HWE we have:

| **AA** ($$p^2$$) | **AT** ($$2pq$$) | **TT** ($$q^2$$) |
|---------|---------|--------|
| 0.0001 | 0.0198 | 0.9801 |

This makes it highly unlikely that **AA** is a true genotype of this individual.

## Calling with FreeBayes

[FreeBayes](https://github.com/ekg/freebayes) is an open source variant caller that has been battle-tested by the 1000 Genomes community and is extensively used today (also see [bcbio](https://bcbio.wordpress.com/)). It has a number of features that simplify variant discovery workflows. These include (from FreeBayes github page):

- **Indel realignment is accomplished internally** using a read-independent method, and issues resulting from discordant alignments are dramatically reduced through the direct detection of haplotypes
- **The need for base quality recalibration is avoided** through the direct detection of haplotypes. Sequencing platform errors tend to cluster (e.g. at the ends of reads), and generate unique, non-repeating haplotypes at a given locus
- **Variant quality recalibration is avoided** by incorporating a number of metrics, such as read placement bias and allele balance, directly into the Bayesian model
- **Ability to incorporate non-diploid cases** such as pooled datasets or data from polyploid samples.

Freebayes is a *haplotype-based* variant caller. This implies that instead of looking at an individual positions within an alignment of reads to the reference genome, it looks at a haplotype window, length of which is dynamically determined (see section 3.2. in [FreeBayes manuscript](https://arxiv.org/pdf/1207.3907v2.pdf)):

![Haplotype-based calling](../../images/freebayes.png "Looking at a haplotype window makes misalignments tolerable. In this case a low complexity poly(A) stretch is misaligned. As a result looking at individual positions will result in calling multiple spurious variants. In the case of FreeBayes looking at a haplotype identifies two alleles (this is a diploid example) A(7) and A(6), while A(8) is likely an error. Image by Erik Garrison")

### The data

In this example we will perform variant calling and annotation using [genome in the bottle data](https://jimb.stanford.edu/giab/). Specifically, we will use Ashkenazim Father-Mother-Son trio data from the Personal Genome Project:

- HG002 - NA24385 - huAA53E0 (son)
- HG003 - NA24149 - hu6E4515 (father)
- HG004 - NA24143 - hu8E87A9 (mother)

Yet for a quick tutorial these datasets are way too big, so we created a [downsampled dataset](https://doi.org/10.5281/zenodo.60520). This dataset was produced by mapping the trio reads against the `hg19` version of the human genome, merging the resulting bam files together (we use readgroups to label individual reads so they can be traced to each of the original individuals), and restricting alignments to a small portion of chromosome 19 containing the [*POLRMT*](https://www.ncbi.nlm.nih.gov/gene?cmd=Retrieve&dopt=Graphics&list_uids=5442) gene.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this variant calling exercise
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files named `GIAB-Ashkenazim-Trio.txt` (tabular format) and `GIAB-Ashkenazim-Trio-hg19` (BAM format) from [Zenodo](https://doi.org/10.5281/zenodo.60520) or a data library:
> 3. Specify the used genome for mapping:
>     1. Click on the {% icon galaxy-pencil %} **pencil icon** for the BAM dataset to edit its attributes
>     2. Select `Human Feb 2009` on **Database/Build**
>     3. Click the **Save** button
{: .hands_on}

### Generating and post-processing FreeBayes calls

> <hands-on-title>Generating FreeBayes calls</hands-on-title>
>
> 1. {% tool [FreeBayes](toolshed.g2.bx.psu.edu/repos/devteam/freebayes/freebayes/1.3.9+galaxy0) %} with the following parameters:
>    - *"Choose the source for the reference genome"*: `locally cached`
>    - *"BAM dataset"*: the uploaded `GIAB-Ashkenazim-Trio-hg19` BAM dataset
>    - *"Using reference genome"*: `Human (Homo sapiens): hg19`
>    - *"Choose parameter selection level"*: `5. Full list of options`
>    - *"Algorithmic features"*: `Set algorithmic features`
>    - *"Calculate the marginal probability of genotypes and report as GQ in each sample field in the VCF output"*: `Yes` (This would help us evaluating the quality of genotype calls)
{: .hands_on}

This will produce a dataset in [VCF](http://www.1000genomes.org/wiki/Analysis/variant-call-format) format containing 35 putative variants. Before we can continue, we need to post-process this dataset by breaking compound variants into multiple independent variants.

> <hands-on-title>Simplify variant representation</hands-on-title>
>
> 1. {% tool [VcfAllelicPrimitives](toolshed.g2.bx.psu.edu/repos/devteam/vcfallelicprimitives/vcfallelicprimitives/1.0.0_rc3+galaxy0) %} with:
>    - *"Select VCF dataset"*: the VCF output of **FreeBayes** {% icon tool %}
>    - *"Maintain site and allele-level annotations when decomposing"*: `Yes`
>    - *"Maintain genotype-level annotations when decomposing"*: `Yes`
{: .hands_on}

{% tool [VcfAllelicPrimitives](toolshed.g2.bx.psu.edu/repos/devteam/vcfallelicprimitives/vcfallelicprimitives/1.0.0_rc3+galaxy0) %} generates a VCF files containing 37 records (the input VCF only contained 35). This is because a multiple nucleotide polymorphism (`TAGG|CAGA`) at position 618851 have been converted to two:

| Before | After |
|--------|---------|
| `chr19 618851 . TAGG CAGA 81.7546` | `chr19 618851 . T C 81.7546`<br>`chr19 618854 . G A 81.7546` |

# Annotating variants with SnpEff

At this point we are ready to begin annotating variants using [**SnpEff**](http://snpeff.sourceforge.net/SnpEff.html). SnpEff "*...annotates and predicts the effects of variants on genes (such as amino acid changes)...*" and so is critical for functional interpretation of variation data.

> <hands-on-title>Annotating variants</hands-on-title>
> 1. {% tool [SnpEff eff](toolshed.g2.bx.psu.edu/repos/iuc/snpeff/snpEff/5.2+galaxy0) %} with:
>    - *"Sequence changes (SNPs, MNPs, InDels)"*: the VCF output of **VcfAllelicPrimitives** {% icon tool %}
>    - *"Genome source"*: `Download on demand`
>    - *"Snpff Genome Version Name"*: `GRCh37.75`
{: .hands_on}

SnpEff will generate two outputs:

1. an annotated VCF file
2. an HTML report

The report contains a number of useful metrics such as distribution of variants across gene features:

![SnpEff output chart](../../images/snpeff_chart.png)

or changes to codons:

![SnpEff output codon changes](../../images/snpeff_codons.png)

# Manipulating variation data with GEMINI

Now that we have an annotated VCF file it is time to peek inside our variation data. [Aaron Quinlan](http://quinlanlab.org/), creator of [GEMINI](http://gemini.readthedocs.org/en/latest/index.html), calls it *Detective work*.

## Loading data into GEMINI

The first step is to convert a VCF file we would like to analyze into a GEMINI database. For this we will use **GEMINI Load** tool. GEMINI takes as input a VCF file and a [PED](https://www.cog-genomics.org/plink2/formats#ped) file describing the relationship between samples. In our case the PED file looks like this (second imported file):

| #family_id | sample_id | paternal_id | maternal_id | sex | phenotype | ethnicity |
|------------|-----------|-------------|-------------|-----|-----------|-----------|
| family1 | HG004_NA24143_mother | -9 | -9 | 2 | 1 | CEU |
| family1 | HG003_NA24149_father | -9 | -9 | 1 | 1 | CEU |
| family1 | HG002_NA24385_son | HG003_NA24149_father | HG004_NA24143_mother | 1 | 2 | CEU |
{: .table .table-responsive}

> <hands-on-title>Loading data into GEMINI</hands-on-title>
>
> 1. {% tool [GEMINI load](toolshed.g2.bx.psu.edu/repos/iuc/gemini_load/gemini_load/0.20.1+galaxy2) %} with:
>    - *"VCF file to be loaded in the GEMINI database"*: the VCF output of **SnpEff** {% icon tool %}
>    - *"Sample information file in PED+ format"*: the uploaded `GIAB-Ashkenazim-Trio.txt` tabular
>    - *"Choose a gemini annotation database"*: the most recent available release
>
>    This will create an SQLite database in your history.
> 2. Run {% tool [GEMINI database info](toolshed.g2.bx.psu.edu/repos/iuc/gemini_db_info/gemini_db_info/0.20.1) %} to see the content of the database:
>    - *"GEMINI database"*: the output of **GEMINI load** {% icon tool %}
>
>    This produces a list of all database tables and their columns according to the [latest version of the GEMINI database schema](https://gemini.readthedocs.org/en/latest/content/database_schema.html).
{: .hands_on}

## Querying the GEMINI database

The GEMINI database can be queried using the versatile [SQL language](https://swcarpentry.github.io/sql-novice-survey). In Galaxy this is done using the **GEMINI query** tool. Within this tool SQL commands are typed directly into the **The query to be issued to the database** text box. Let's begin getting information from some of the tables we discovered using the **GEMINI db_info** tool above.

> <comment-title>GEMINI tutorials</comment-title>
>
> The examples below are taken from "[Introduction to GEMINI](https://s3.amazonaws.com/gemini-tutorials/Intro-To-Gemini.pdf)" tutorial. For extensive documentation see "[Querying the GEMINI database](https://gemini.readthedocs.org/en/latest/content/querying.html)".
{: .comment}

> <hands-on-title>Selecting "novel" variants that are not annotated in dbSNP database</hands-on-title>
>
> 1. {% tool [GEMINI query](toolshed.g2.bx.psu.edu/repos/iuc/gemini_query/gemini_query/0.20.1+galaxy2) %} with:
>    - *"GEMINI database"*: the output of {% tool [GEMINI load](toolshed.g2.bx.psu.edu/repos/iuc/gemini_load/gemini_load/0.20.1+galaxy2) %}
>    - *"Build GEMINI query using"*: Advanced query constructor
>    - *"The query to be issued to the database"*: `SELECT count(*) FROM variants WHERE in_dbsnp == 0`
>
>    As we can see in the output dataset, there are 21 variants that are not annotated in dbSNP.
{: .hands_on}

> <hands-on-title>Find variants within the POLRMT gene</hands-on-title>
>
> 1. {% tool [GEMINI query](toolshed.g2.bx.psu.edu/repos/iuc/gemini_query/gemini_query/0.20.1+galaxy2) %} with:
>    - *"GEMINI database"*: the output of **GEMINI load** {% icon tool %}
>    - *"Build GEMINI query using"*: Advanced query constructor
>    - *"The query to be issued to the database"*: `SELECT rs_ids, aaf_esp_ea, impact, clinvar_disease_name, clinvar_sig FROM variants WHERE filter is NULL and gene = 'POLRMT'`
>
>    Since the `variants` table has a large number of columns, in the query above we had to select only the most interesting columns. The output shows the variants found within the *POLRMT* gene.
{: .hands_on}

## Querying genotypes

GEMINI provides access to genotype, sequencing depth, genotype quality, and genotype likelihoods for each individual (`subjectID`):

- `gt_types.subjectID` - three types of genotype types: `HOM_REF`, `HET`, `HOM_ALT`
- `gt_quals.subjectID` - genotype quality
- `gt_depths.subjectID` - total number of reads in this subject at position
- `gt_ref_depths.subjectID` -  number of reference allele reads in this subject at position
- `gt_alt_depths.subjectID` - number of alternate allele reads in this subject at position

> <question-title></question-title>
>
> 1. At how many sites does child have a non-reference allele?
> 2. At how many sites both father and son have non reference alleles?
> 3. List genotypes for father and son where they have non-reference alleles.
>
> > <solution-title></solution-title>
> > 1. To answer this question you can run the **GEMINI query** {% icon tool %} tool with:
> >    - *"The query to be issued to the database"*: `SELECT * from variants`
> >    - *"Restrictions to apply to genotype values"*: `gt_types.HG002_NA24385_son <> HOM_REF`
> > 2. To answer this question you can run the **GEMINI query** {% icon tool %} tool with:
> >    - *"The query to be issued to the database"*: `SELECT * from variants`
> >    - *"Restrictions to apply to genotype values"*: `(gt_types.HG002_NA24385_son <> HOM_REF AND gt_types.HG003_NA24149_father <> HOM_REF)`
> > 3. To answer this question you can run the **GEMINI query** {% icon tool %} tool with:
> >    - *"The query to be issued to the database"*: `SELECT gts.HG002_NA24385_son, gts.HG003_NA24149_father from variants`
> >    - *"Restrictions to apply to genotype values"*: `(gt_types.HG002_NA24385_son <> HOM_REF AND gt_types.HG003_NA24149_father <> HOM_REF)`
> {: .solution }
{: .question}

## Using wildcards

Wildcards simply writing SQL expressions when searching across multiple terms. The syntax for genotype filter wildcards is:

```
(COLUMN).(SAMPLE_WILDCARD).(SAMPLE_WILDCARD_RULE).(RULE_ENFORCEMENT)
```

Let's look at some examples.

> <question-title></question-title>
>
> At which variants are every sample heterozygous?
>
> > <solution-title></solution-title>
> >
> > To answer this question you can run the {% tool [GEMINI query](toolshed.g2.bx.psu.edu/repos/iuc/gemini_query/gemini_query/0.20.1+galaxy2) %} tool with:
> > - *"The query to be issued to the database"*: `SELECT chrom, start, end, ref, alt, gene, impact, (gts).(*) FROM variants`
> > - *"Restrictions to apply to genotype values"*: `(gt_types).(*).(==HET).(all)`
> >
> > Here we use wildcards for the query (`(gts.(*))` means get genotypes for **all** samples) and genotype filtering (`(gt_types).(*).(==HET).(all)`, the [all operator](https://gemini.readthedocs.org/en/latest/content/querying.html#the-all-operator) implies that we want results for **all** affected individuals).
> >
> {: .solution }
{: .question}

## Going further

This short tutorial should give you an overall idea on how generate variant data in Galaxy and process it with GEMINI. Yet there is much more to learn. Below we list GEMINI tutorials and links to Galaxy libraries with relevant data:

| Tutorial | PDF | Galaxy library | Galaxy history |
|:---------|:---:|:--------------:|:--------------:|
| Introduction | [{% icon references %}](https://s3.amazonaws.com/gemini-tutorials/Intro-To-Gemini.pdf) | [{% icon galaxy-library %}](https://usegalaxy.org/library/folders/F0283ca691a41c352) | [{% icon galaxy-history %}]( https://usegalaxy.org/u/aun1/h/gemini-introduction) |
| Identifying *de novo* mutations underlying Mendelian disease | [{% icon references %}](https://s3.amazonaws.com/gemini-tutorials/Gemini-DeNovo-Tutorial.pdf) | [{% icon galaxy-library %}](https://usegalaxy.org/library/folders/F775008f45cbbf010) | [{% icon galaxy-history %}]( https://usegalaxy.org/u/aun1/h/gemini-de-novo-mutations) |
| Identifying autosomal recessive variants underlying Mendelian disease | [{% icon references %}](https://s3.amazonaws.com/gemini-tutorials/Gemini-Recessive-Tutorial.pdf) | [{% icon galaxy-library %}](https://usegalaxy.org/library/folders/F35b262f5ac8aa63a) | [{% icon galaxy-history %}](https://usegalaxy.org/u/aun1/h/gemini-autosomal-recessive) |
| Identifying autosomal dominant variants underlying Mendelian disease | [{% icon references %}](https://s3.amazonaws.com/gemini-tutorials/Gemini-Dominant-Tutorial.pdf) | [{% icon galaxy-library %}](https://usegalaxy.org/library/folders/F1c4722ad56892a31) | [{% icon galaxy-history %}]( https://usegalaxy.org/u/aun1/h/gemini-autosomal-dominant ) |

### How to use these tutorials?

* Right click on the **PDF** {% icon references %} icon and open tutorial in a new browser tab
* Right click on **Galaxy history** {% icon galaxy-history %} icon and open Galaxy history in another new browser tab
* When the Galaxy history interface opens, you will need to click the **Import history** link highlighted with a red outline in the following figure:

  ![Import history](../../images/import_history.png)

* If you have a wide screen arrange browsers tabs side by side:

  ![Arranging browsers side by side](../../images/side-by-side.png)

* Proceed with the tutorial. For example, to repeat the following command from GEMINI tutorial:

  ![GEMINI query command](../../images/gemini_command.png)

  use Galaxy's {% tool [GEMINI query](toolshed.g2.bx.psu.edu/repos/iuc/gemini_query/gemini_query/0.20.1+galaxy2) %} tool as below:

  ![GEMINI_query tool interface](../../images/galaxy_command.png)

* and so on....

tools_json:
["FreeBayes", "toolshed.g2.bx.psu.edu/repos/devteam/freebayes/freebayes/1.3.9+galaxy0", "VcfAllelicPrimitives", "toolshed.g2.bx.psu.edu/repos/devteam/vcfallelicprimitives/vcfallelicprimitives/1.0.0_rc3+galaxy0", "SnpEff eff", "toolshed.g2.bx.psu.edu/repos/iuc/snpeff/snpEff/5.2+galaxy0", "GEMINI load", "toolshed.g2.bx.psu.edu/repos/iuc/gemini_load/gemini_load/0.20.1+galaxy2", "GEMINI database info", "toolshed.g2.bx.psu.edu/repos/iuc/gemini_db_info/gemini_db_info/0.20.1", "GEMINI query", "toolshed.g2.bx.psu.edu/repos/iuc/gemini_query/gemini_query/0.20.1+galaxy2"]


topic: variant-analysis
tutorial: sars-cov-2-variant-discovery
title: Mutation calling, viral genome reconstruction and lineage/clade assignment
difficulty: intermediate
time_estimation: 3H
requirements_json: ["type: internal", "collections", "type: internal", "sars-cov-2"]
tags_json: ["covid19", "virology", "one-health"]
content_hash: 2b98a9cebf131b9ee3c88d3f4ccb7953
last_modified: 2026-05-27T09:01:50.018049
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/sars-cov-2-variant-discovery/tutorial.html

content:
Sequence-based monitoring of global infectious disease crises, such as the COVID-19 pandemic, requires capacity to generate and analyze large volumes of sequencing data in near real time. These data have proven essential for surveilling the emergence and spread of new viral variants, and for understanding the evolutionary dynamics of the virus.

The tutorial [SARS-CoV-2 sequencing data analysis]({% link topics/variant-analysis/tutorials/sars-cov-2/tutorial.md %}) shows in detail how you can identify mutations in SARS-CoV-2 samples from paired-end whole-genome sequencing data generated on the Illumina platform.

For versatile and efficient genome surveillance, however, you would want to:
- be able to analyze data of different origin

  Besides WGS paired-end Illumina data, different labs are generating also single-end Illumina and ONT data, and are combining these platforms with various tiled-amplicon approaches upstream of sequencing.

- go beyond per-sample mutation calls in variant call format (VCF)

  To keep track of large numbers of samples sequenced in batches (as, nowadays, produced routinely by SARS-CoV-2 genome surveillance initiatives across the globe), you need concise reports and visualizations of results at the sample and batch level.

- use sample mutation patterns to construct sample consensus genomes

- use the consensus genomes to assign the samples to SARS-CoV-2 lineages as defined by major lineage classification systems ([Nextstrain clades](https://nextstrain.org/blog/2022-04-29-SARS-CoV-2-clade-naming-2022) and [PANGO](https://www.pango.network/))

- decrease hands-on time and data manipulation errors by combining analysis steps into workflows for automated execution

The purpose of this tutorial is to demonstrate how a set of workflows developed by the [Galaxy Covid-19 project](https://galaxyproject.org/projects/covid19/) can be combined and used together with a handful of additional tools to achieve all of the above. Specifically, we will cover the analysis flow presented in figure 1.

![Analysis flow in the tutorial](../../images/sars-cov-2-variant-discovery/schema.png "Analysis flow in the tutorial")

Depending on the type of sequencing data **one of four variation analysis workflows** can be run to discover mutations in a batch of input samples. Outputs of any of these workflows can then be processed further with two additional workflows: the **variation reporting workflow** generates a per-sample report of mutations, but also batch-level reports and visualizations, while the **consensus construction workflow** reconstructs the full viral genomes of all samples in the batch by modifying the SARS-CoV-2 reference genome with each sample's set of mutations.

A few highlights of these workflows are:

- All the variation analysis workflows are more sensitive than they need to be for consensus sequence generation, i.e. they can not only be used to capture fixed or majority alleles, but can also be used on their own to address less routine questions such as co-infections with two viral lineages or shifting intrahost allele-frequencies in, for example, immunocompromised, longterm-infected patients.
- The reporting workflow produces a batch-level overview plot of mutations and their observed allele-frequencies that enables spotting of batch-effects like sample cross-contamination and outlier samples that are different from the rest of the batch.
- The consensus workflow can express uncertainty about any base position in the generated consensus sequence by N-masking the position according to user-defined thresholds.
- All of the workflows are openly developed and available in the form of defined releases through major public workflow registries.

In this tutorial you will learn to
- obtain releases of the workflows from public registries
- set up input data for different types of sequencing protocols
- run and combine the workflows
- understand the various outputs produced by the workflows and to extract insight about viral samples from them

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Prepare Galaxy and data

The suggested input for this tutorial is a special batch of data that is of particular interest as it represents a turning point in the COVID-19 pandemic.
It is a subset (16 samples) of the first sequencing data reported from South Africa at the end of November 2021 for the then novel, fast-spreading SARS-CoV-2 variant that would later be named Omicron.
This data has been Illumina paired-end sequenced after amplification with the ARTIC v4 set of tiled-amplicon primers.

> <comment-title>Bringing your own data</comment-title>
> Alternatively, you can also follow this tutorial using your own SARS-CoV-2 sequencing data (you need at least two samples) as long as it is of one of the following types:
>
> - Single-end data derived from Illumina-based whole-genome sequencing experiments
> - Paired-end data derived from Illumina-based whole-genome sequencing experiments
> - Paired-end data generated with Illumina-based tiled-amplicon (e.g. ARTIC) protocols
> - ONT FASTQ files generated with Oxford nanopore (ONT)-based tiled-amplicon (e.g. ARTIC) protocols
>
> {% icon warning %} If you are using your own *tiled-amplicon* data, you are also expected to know the primer scheme used at the amplification step.
>
{: .comment}

## Prepare a new Galaxy history

Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Prepare the Galaxy history</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

## Get sequencing data

> <comment-title>Importing your own data</comment-title>
> If you are going to use your own sequencing data, there are several possibilities to upload the data depending on how many datasets you have and what their origin is:
>
> - You can import data
>
>   - from your local file system,
>   - from a given URL or
>   - from a shared data library on the Galaxy server you are working on
>
>   In all of these cases you will also have to organize the imported data into a dataset collection like explained in detail for the suggested example data.
>
>   > <details-title>Data logistics</details-title>
>   >
>   > A detailed explanation of all of the above-mentioned options for getting your data into Galaxy and organizing it in your history is beyond the scope of this tutorial.
>   > If you are struggling with getting your own data set up like shown for the example data in this section, please:
>   > - Option 1: Browse some of the material on [Using Galaxy and Managing your Data]({% link topics/galaxy-interface %})
>   > - Option 2: Consult the FAQs on [uploading data]({% link faqs/galaxy/index.md %}#data%20upload) and on [collections]({% link faqs/galaxy/index.md %}#collections)
>   > - Option 3: Watch some of the related brief videos from the [{% icon video %} Galactic introductions](https://www.youtube.com/playlist?list=PLNFLKDpdM3B9UaxWEXgziHXO3k-003FzE) playlist.
>   >
>   {: .details}
>
> - Alternatively, if your data is available from [NCBI's Sequence Read Archive (SRA)](https://www.ncbi.nlm.nih.gov/sra), you can import it with the help of a dedicated tool, which will organize the data into collections for you.
>
>   > <details-title>Getting data from SRA</details-title>
>   >
>   > The simpler [SARS-CoV-2 sequencing data analysis tutorial]({% link topics/variant-analysis/tutorials/sars-cov-2/tutorial.md %}) uses and explains this alternative way of importing.
>   >
>   {: .details}
{: .comment}

For the suggested batch of early Omicron data we suggest downloading it via URLs from the [European Nucleotide Archive (ENA)](https://www.ebi.ac.uk/ena/browser/home). In case your Galaxy server offers that same data through a shared data library, this represents a faster (data is already on the server) alternative, so we offer instructions for this scenario as well.

> <hands-on-title>Import the sequencing data</hands-on-title>
>
> - Option 1: Import from the ENA
>
>   ```
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/002/SRR17054502/SRR17054502_1.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/002/SRR17054502/SRR17054502_2.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/003/SRR17054503/SRR17054503_1.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/003/SRR17054503/SRR17054503_2.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/004/SRR17054504/SRR17054504_1.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/004/SRR17054504/SRR17054504_2.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/005/SRR17054505/SRR17054505_1.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/005/SRR17054505/SRR17054505_2.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/006/SRR17054506/SRR17054506_1.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/006/SRR17054506/SRR17054506_2.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/007/SRR17054507/SRR17054507_1.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/007/SRR17054507/SRR17054507_2.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/008/SRR17054508/SRR17054508_1.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/008/SRR17054508/SRR17054508_2.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/009/SRR17054509/SRR17054509_1.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/009/SRR17054509/SRR17054509_2.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/010/SRR17054510/SRR17054510_1.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/010/SRR17054510/SRR17054510_2.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/033/SRR17051933/SRR17051933_1.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/033/SRR17051933/SRR17051933_2.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/034/SRR17051934/SRR17051934_1.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/034/SRR17051934/SRR17051934_2.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/035/SRR17051935/SRR17051935_1.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/035/SRR17051935/SRR17051935_2.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/036/SRR17051936/SRR17051936_1.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/036/SRR17051936/SRR17051936_2.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/037/SRR17051937/SRR17051937_1.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/037/SRR17051937/SRR17051937_2.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/038/SRR17051938/SRR17051938_1.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/038/SRR17051938/SRR17051938_2.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/039/SRR17051939/SRR17051939_1.fastq.gz
>   ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/039/SRR17051939/SRR17051939_2.fastq.gz
>   ```
>
>   1. Copy the links above
>   2. Open the {% tool [Upload](upload1) %} Manager
>   3. In the top row of tabs select **Collection**
>   4. Configure the drop-down select boxes on that tab like this:
>      - *"Collection Type"*: `List of Pairs`
>      - *"File Type"*: `fastqsanger.gz`
>   5. Click on **Paste/Fetch data** and paste the links you copied into the empty text box
>   6. Press **Start**
>   7. Wait for the **Build** button to become enabled, then click it
>   8. In the lower half of the next dialogue, Galaxy already suggests a mostly reasonable pairing of the inputs.
>
>      As you can see, however, this auto-pairing would retain a *.fastq* suffix attached to each pair of forward and reverse reads. To correct this
>      1. Click **Unpair all** above the suggested pairings to undo all of them.
>      2. Change the following default values in the upper half of the window:
>         - *"unpaired forward"*: `_1.fastq.gz` (instead of *_1*)
>         - *"unpaired reverse"*: `_2.fastq.gz` (instead of *_2*)
>      3. Click **Auto-pair**
>   9. At, the bottom of the window, enter a suitable **Name**, like `Sequencing data`, for the new collection
>   10. Click on **Create collection**
>
> - Option 2: Import from a shared data library
>
>   {% snippet faqs/galaxy/datasets_import_from_data_library.md astype="as a Collection" collection_type="List of Pairs" collection_name="Sequencing data" tohistory="the history you created for this tutorial" path="GTN - Material / Variant analysis / Mutation calling, viral genome reconstruction and lineage/clade assignment from SARS-CoV-2 sequencing data / Omicron_batch_analysis" box_type="none" %}
>
{: .hands_on}

## Import reference sequence and auxiliary datasets

Besides the sequenced reads data, we need at least two additional datasets for calling variants and annotating them:

- the SARS-CoV-2 reference sequence [NC_045512.2](https://www.ncbi.nlm.nih.gov/nuccore/NC_045512.2?report=fasta) to align and compare our sequencing data against

- a tabular dataset defining aliases for viral gene product names, which will let us translate NCBI RefSeq Protein identifiers (used by the SnpEff annotation tool) to the commonly used names of coronavirus proteins and cleavage products.

> <hands-on-title>Get reference sequence and feature mappings</hands-on-title>
>
> 1. Get the SARS-CoV-2 reference sequence
>
>    A convenient public download link for this sequence is best obtained from the ENA again, where the sequence is known under its [INSDC](https://www.insdc.org/) alias [MN908947.3](https://www.ebi.ac.uk/ena/browser/view/MN908947.3):
>    ```
>    https://www.ebi.ac.uk/ena/browser/api/fasta/MN908947.3?download=true
>    ```
>
>    1. {% tool [Upload](upload1) %} the reference to your history via the link above and make sure the dataset format is set to `fasta`.
>
>       {% snippet faqs/galaxy/datasets_import_via_link.md format="fasta" %}
>    2. {% tool [Replace Text in entire line](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.3+galaxy1) in entire line %} to simplify the reference sequence name
>       - {% icon param-file %} *"File to process"*: the uploaded reference sequence from the ENA
>       - In {% icon param-repeat %} *"1. Replacement"*:
>         - *"Find pattern"*: `^>.+`
>         - *"Replace with"*: `>NC_045512.2`
>
>       While the identifiers `MN908947.3` and `NC_045512.2` really refer to the same sequence, one of the tools we are going to use during the analysis (SnpEff) requires the NCBI RefSeq identifier.
>    3. When the Replace Text tool run is finished, **rename** the output dataset to make it clear that this is the SARS-CoV-2 reference dataset to use in the analysis.
>
>       {% snippet faqs/galaxy/datasets_rename.md name="SARS-CoV-2 reference" format="fasta" %}
> 2. {% tool [Upload](upload1) %} the mapping for translation product identifiers in tabular format
>
>    This mapping really consists of just a few lines of text. Each line lists the NCBI Protein Refseq identifier of a SARS-CoV-2 translation product (which the tool SnpEff knows about and will use for annotating mutation effects), followed by a more commonly used name for that product (which we would like to see in final mutation reports). The last line specifies a "mapping" for the **.** annotation, which SnpEff uses for mutations that do not affect any viral open-reading frame. We do not have a better name for it so we specify that we want to retain this annotation unaltered in reports.
>
>    ```
>    YP_009725297.1    leader
>    YP_009725298.1    nsp2
>    YP_009725299.1    nsp3
>    YP_009725300.1    nsp4
>    YP_009725301.1    3Cpro
>    YP_009725302.1    nsp6
>    YP_009725303.1    nsp7
>    YP_009725304.1    nsp8
>    YP_009725305.1    nsp9
>    YP_009725306.1    nsp10
>    YP_009725307.1    RdRp
>    YP_009725308.1    helicase
>    YP_009725309.1    ExoN
>    YP_009725310.1    endoR
>    YP_009725311.1    MethTr
>    YP_009725312.1    nsp11
>    GU280_gp02        S
>    GU280_gp03        orf3a
>    GU280_gp04        E
>    GU280_gp05        M
>    GU280_gp06        orf6
>    GU280_gp07        orf7a
>    GU280_gp08        orf7b
>    GU280_gp09        orf8
>    GU280_gp10        N
>    GU280_gp11        orf10
>    .                 .
>    ```
>
>    Two remarks on this content:
>    - Since the feature mapping dataset is expected to be in tabular format, but the above display uses spaces to separate columns, please make sure, you have **Convert spaces to tabs** checked when creating the dataset from the copied content!
>    - If you prefer other names for certain translation products than the ones defined above (*e.g.* you might be used to call the first peptide *nsp1* instead of *leader*), you are, of course, free to change those names in the pasted content before uploading it to Galaxy. What needs to be kept is only the Refseq identifiers in the first column as these are fixed by the SnpEff tool.
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md name="SARS-CoV-2 feature mapping" format="tabular" convertspaces="true" %}
>
{: .hands_on}

Another two datasets are needed only for the analysis of ampliconic, e.g. ARTIC-amplified, input data:

- a BED file specifying the primers used during amplification and their binding sites on the viral genome
- a custom tabular file describing the amplicon grouping of the primers (currently NOT used for tiled-amplicon ONT data)

> <details-title>Using your own tiled-amplicon data? Provide the correct primer scheme and amplicon info.</details-title>
>
> The instructions below assume that you are going to analyze viral samples amplified using **version 4 of the ARTIC network's SARS-CoV-2 set of primers**, which is the case for the suggested Omicron batch of data.
>
> If you have decided to analyze your own tiled-amplicon sequencing data in this tutorial, and if your samples have been amplified with a **different** set of primers, you are supposed, at the following step, to upload a primer scheme file and corresponding amplicon information that describes this set of primers.
>
> The Galaxy Project maintains a [collection of such files for some commonly used sets of primers](https://doi.org/10.5281/zenodo.4555734).
> If the files describing your set of primers are part of this collection, you can simply upload them using their Zenodo download URLs (analogous to what is shown for the ARTIC v4 primers below).
>
> For sets of primers *not* included in the collection, you will have to create those files yourself (or obtain them from other sources).
> The expected format for the primer scheme file is 6-column BED format, while the amplicon info file is a simple tabular format that lists all primer names (which must be identical to the ones used in the primer scheme file) that contribute to formation of the same amplicon on a single tab-separated line.
> If in doubt, download the ARTIC v4 files through the URLs provided below, and use them as a template for your own custom files.
>
{: .details}

> <hands-on-title>Get primer scheme and amplicon info</hands-on-title>
>
> 1. Get the ARTIC v4 primer scheme file from
>
>    ```
>    https://zenodo.org/record/5888324/files/ARTIC_nCoV-2019_v4.bed
>    ```
>
>    and upload it to Galaxy as a dataset of type `bed`.
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="bed" %}
>
> 2. Get the ARTIC v4 amplicon info file from
>
>    ```
>    https://zenodo.org/record/5888324/files/ARTIC_amplicon_info_v4.tsv
>    ```
>
>    and upload it to Galaxy as a dataset of type `tabular`.
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="tabular" %}
>
{: .hands_on}

At this point you should have the following items in your history:

1. A collection of sequenced reads for all samples to be analyzed
   - This collection should be organized as a list with one element per sample that you are planning to analyze.

     If you are going to analyze the suggested batch of Omicron data, the list should have 16 elements.

   - For paired-end sequenced data, like the suggested batch, each of the list elements should itself be a paired collection of a *forward* and a *reverse* reads dataset, each in *fastqsanger.gz* format.

2. The SARS-CoV-2 reference as a single *fasta* dataset
3. The SARS-CoV-2 feature mappings as a single *tabular* dataset
4. Only required if you are analyzing *tiled-amplicon* data (which is the case for the suggested batch):
   - a primer scheme as a single *bed* or *bed6* dataset
   - amplicon information as a single *tabular* dataset (*not* required for ONT data)

If any of these items are still missing, head back to the corresponding section(s) and upload them now.

If any of these items have not been assigned the correct format (expand the view of each dataset to reveal the format Galaxy has recorded for it), please fix them now.

{% snippet faqs/galaxy/datasets_change_datatype.md %}

If everything looks fine, you are ready to start the actual data analysis.


# Analysis

As shown in figure 1, the analysis will consist mainly of running three workflows - one for calling the mutations for each input sample, one for reporting the findings and one for generating sample consensus sequences based on the identified mutations.

You will also run a handful of individual tools, in particular for generating and exploring lineage assignments for the input samples.

Along the way you will interpret key outputs of each used workflow and tool to get a complete picture of what can be learnt about the genomes of the input samples.

## From sequencing data to annotated mutations per sample

To identify mutations from sequencing data, the first workflow performs steps including

- sequencing data quality control with filtering and trimming of low-quality reads
- mapping
- filtering and further processing of mapped reads (including primer-trimming in case of tiled-amplicon data)
- calling of mutations
- annotation of identified mutations with their effects on open-reading frames (ORFs) and proteins

Four flavors of this workflow are available that are optimized for different types of input data so your first task is to obtain the correct workflow for your type of input data.

> <hands-on-title>Import the workflow for your data into Galaxy</hands-on-title>
>
> All workflows developed as part of the Galaxy Covid-19 project can be retrieved via either of the two popular workflow registries [Dockstore](https://dockstore.org/) and [WorkflowHub](https://workflowhub.eu/), the choice is up to you, and Galaxy makes this process really easy for you.
>
> {% snippet faqs/galaxy/workflows_import_search.md search_query='organization:"iwc" name:"sars-cov-2"' disable_image="True" disable_video="True" box_type="none" %}
>
> *IWC* (the intergalactic workflow commission) is the organization that the Galaxy Covid-19 project uses to publish its workflows, and the name restriction makes sure we are only getting workflows from that organization that deal with SARS-CoV-2 data analysis.
>
> Depending on your input data you will need to select the appropriate workflow from the list of hits returned by the workflow registry server.
> This would be:
> - **sars-cov-2-pe-illumina-artic-variant-calling/COVID-19-PE-ARTIC-ILLUMINA**
>
>   if you are working with the suggested batch of samples for this tutorial, or if your own data is tiled-amplicon data sequenced on the Illumina platform in paired-end mode
>
>   Once imported into Galaxy, this workflow appear under the name: **COVID-19: variation analysis on ARTIC PE data**.
> - **sars-cov-2-ont-artic-variant-calling/COVID-19-ARTIC-ONT**
>
>   if you are working with tiled-amplicon data sequenced on the ONT platform
>
>   Once imported into Galaxy, this workflow appear under the name: **COVID-19: variation analysis of ARTIC ONT data**.
> - **sars-cov-2-pe-illumina-wgs-variant-calling/COVID-19-PE-WGS-ILLUMINA**
>
>   if you are working with WGS (i.e. non-ampliconic) data obtained on the Illumina platform in paired-end mode
>
>   Once imported into Galaxy, this workflow appear under the name: **COVID-19: variation analysis on WGS PE data**.
> - **sars-cov-2-se-illumina-wgs-variant-calling/COVID-19-SE-WGS-ILLUMINA**
>
>   if you are working with WGS data obtained on the Illumina platform in single-end mode
>
>   Once imported into Galaxy, this workflow appear under the name: **COVID-19: variation analysis on WGS SE data**.
>
> In all cases, the latest version of the workflow should be fine to use in this tutorial.
>
{: .hands-on}

> <details-title>About the workflows</details-title>
>
> Differences between the four different *variation analysis* workflow flavors are as follows:
> - All workflows for analysis of Illumina data use a core analysis flow through **bwa-mem** (or **bowtie2** for single-end data) as the read mapper and **lofreq** as a sensitive variant caller (which is capable of identifying also non-majority variant alleles), while the workflow for analysis of ONT data uses **minimap2** and **medaka** for read mapping and variant calling, respectively.
> - Workflows for analysis of tiled-amplicon data differ from WGS data analysis ones in that they do not perform mapped reads deduplication (because apparent duplicates are a side-effect of sequencing defined amplicons instead of random genome fragments).
>   In addition, the tiled-amplicon data analysis workflows handle primer trimming and amplicon-based filtering through the **ivar** suite of tools and additional processing logic.
>
> All four workflows use **fastp** for quality control, raw reads filtering and trimming, and **SnpEff**, specifically its 4.5covid19 version, for annotation of mutation effects.
>
{: .details}

{% include _includes/cyoa-choices.html option1="tiled-amplicon Illumina paired-end" option2="tiled-amplicon ONT" option3="WGS Illumina paired-end" option4="WGS Illumina single-end" default="tiled-amplicon Illumina paired-end" text="Now that you have imported the data and the corresponding workflow of your choice, please select the type of your input data so that we can adjust a few parts of this tutorial that are dependent on the nature of your data:" %}

> <hands-on-title>From sequencing data to annotated mutations</hands-on-title>
>
> <div class="tiled-amplicon-Illumina-paired-end" markdown="1">
>
> 1. Run the **COVID-19: variation analysis on ARTIC PE data** {% icon workflow %} workflow using the following parameters:
>
>    - {% icon param-collection %} *"Paired Collection"*: your paired collection of input sequencing data
>    - {% icon param-file %} *"NC_045512.2 FASTA sequence of SARS-CoV-2"*: the `SARS-CoV-2 reference` sequence
>    - {% icon param-file %} *"ARTIC primer BED"*: the uploaded primer scheme in **bed** format
>    - {% icon param-file %} *"ARTIC primers to amplicon assignments"*: the uploaded amplicon info in **tabular** format
>
>      A common mistake here is to mix up the last two datasets: the *primer BED* dataset is the one with the positions of the primer binding sites listed in it. The *amplicon assignments* dataset contains only (grouped) names of primers.
>
>    The additional workflow parameters *"Read removal minimum AF"*, *"Read removal maximum AF"*, *"Minimum DP required after amplicon bias correction"* and *"Minimum DP_ALT required after amplicon bias correction"* can all be left at their default values. The tiled-amplicon Illumina workflow goes to great lengths in order to calculate accurate allele-frequencies even in cases of viral mutations that affect the binding of some of the tiled-amplicon primers. The four settings above are for fine-tuning this behavior, but their defaults have been carefully chosen and you should modify them only if you know exactly what you are doing. 
> </div>
> <div class="tiled-amplicon-ONT" markdown="1">
>
> 1. Run the **COVID-19: variation analysis of ARTIC ONT data** {% icon workflow %} workflow using the following parameters:
>
>    - {% icon param-collection %} *"ONT-sequenced reads"*: your collection of input sequencing data
>    - {% icon param-file %} *"NC_045512.2 FASTA sequence of SARS-CoV-2"*: the `SARS-CoV-2 reference` sequence
>    - {% icon param-file %} *"Primer binding sites info in BED format"*: the uploaded primer scheme in **bed** format
>
>    The optional workflow parameters *"Minimum read length"* and *"Maximum read length"* should be chosen according to the tiled-amplicon primer scheme's amplicon sizes.
>    The [ARTIC network's recommendations for excluding obviously chimeric reads](https://artic.network/ncov-2019/ncov2019-bioinformatics-sop.html) (see the section "Read filtering" on that page) are a good starting point.
>
>    The workflow defaults are appropriate for ARTIC network primers, but you may have to modify them if your sample material has been amplified with another primer scheme. As suggested on the above page: try to set *"Minimum read length"* to the size of the smallest amplicon in your primer scheme, and *"Maximum read length"* to the size of the largest amplicon plus 200 nts.
>
> </div>
> <div class="WGS-Illumina-paired-end" markdown="1">
>
> 1. Run the **COVID-19: variation analysis on WGS PE data** {% icon workflow %} workflow using the following parameters:
>
>    - {% icon param-collection %} *"Paired Collection"*: your paired collection of input sequencing data
>    - {% icon param-file %} *"NC_045512.2 FASTA sequence of SARS-CoV-2"*: the `SARS-CoV-2 reference` sequence
>
> </div>
> <div class="WGS-Illumina-single-end" markdown="1">
>
> 1. Run the **COVID-19: variation analysis on WGS SE data** {% icon workflow %} workflow using the following parameters:
>
>    - {% icon param-collection %} *"Single End Collection"*: your collection of input sequencing data
>    - {% icon param-file %} *"NC_045512.2 FASTA sequence of SARS-CoV-2"*: the `SARS-CoV-2 reference` sequence
>
> </div>
>
>    > <tip-title>Running a workflow</tip-title>
>    >
>    > {% snippet faqs/galaxy/workflows_run.md box_type="none" %}
>    >
>    > Note: the {% icon galaxy-gear %} icon next to **Run Workflow** offers the option to *Send results to a new history*.
>    > This is very useful if you are planning to analyze the data in your current history in multiple different ways, and you would like to have each analysis end up in its own dedicated history.
>    > Here, however, we only want to do one analysis of our batch of data so we are fine with results of the workflow run getting added to the current history.
>    {: .tip}
>
{: .hands_on}

Scheduling of the workflow will take a while. One of the last datasets that will be added to your history will be called **Preprocessing and mapping reports**. It is the first overview report that you will obtain in this tutorial, and the only one produced by the variation analysis workflow.

Once this dataset is ready take a moment to explore its content. It contains potentially valuable information about coverage of reads mapped to the reference genome and quality statistics for these reads. Technical issues with any samples in a batch are typically visible in this report already and spotting them early can often prevent overinterpretation of later reports.

If you are following along with the suggested batch of samples, we also have prepared a simple question for you.

> <question-title></question-title>
>
> 1. There are at least three problematic samples in the batch. What are their identifiers and what is the issue with them?
>
> > <solution-title></solution-title>
> >
> > 1. The three samples are **SRR17054505**, **SRR17054506** and **SRR17054508**.
> >
> >    The problem with all three is poor coverage as can be seen from the "General Statistics" section.
> >    When you inspect the "Cumulative genome coverage" plot, you can see that a fourth sample, **SRR17054502**, is not that much better than two of these three.
> >
> >    Coverage of the worst sample, **SRR17054505**, is critically low and we cannot expect very usable mutation calls from it.
> >    For the other three we may hope for some results, but we should not expect those to be perfect.
> >
> {: .solution}
{: .question}

## From mutations per sample to reports and visualizations

The main output of the first workflow is a collection with annotated mutation calls per sample. This collection will serve as input to the *reporting* workflow.

> <hands-on-title>Import the variant analysis reporting workflow into Galaxy</hands-on-title>
>
> Just like the variation analysis workflows before, also the *reporting workflow* developed by the Galaxy Covid-19 project can be retrieved from *Dockstore* or *WorkflowHub*:
>
> {% snippet faqs/galaxy/workflows_import_search.md search_query='organization:"iwc" name:"sars-cov-2"' workflow_name="sars-cov-2-variation-reporting/COVID-19-VARIATION-REPORTING" disable_image="True" disable_video="True" box_type="none" %}
>
> Again, you can just select the latest version of the workflow, and, once imported, it should appear in your list of workflows under the name: **COVID-19: variation analysis reporting**.
>
{: .hands-on}

> <hands-on-title>From mutations per sample to reports and visualizations</hands-on-title>
>
> 1. Run the **COVID-19: variation analysis reporting** {% icon workflow %} workflow with the following parameters:
>
>    <div class="tiled-amplicon-Illumina-paired-end" markdown="1">
>
>    - *"Variation data to report"*: `Final (SnpEff-) annotated variants`
>    </div>
>    <div class="tiled-amplicon-ONT" markdown="1">
>
>    - *"Variation data to report"*: `Final (SnpEff-) annotated variants`
>    </div>
>    <div class="WGS-Illumina-paired-end" markdown="1">
>
>    - *"Variation data to report"*: `Final (SnpEff-) annotated variants with strand-bias soft filter applied`
>    </div>
>    <div class="WGS-Illumina-single-end" markdown="1">
>
>    - *"Variation data to report"*: `Final (SnpEff-) annotated variants with strand-bias soft filter applied`
>    </div>
>
>      The collection with variation data in VCF format; output of the previous workflow
>
>      > <comment-title>Use the right collection of annotated variants!</comment-title>
>      > The variation analysis workflow should have generated *two* collections of annotated variants - one called `Final (SnpEff-) annotated variants`, the other one called `Final (SnpEff-) annotated variants with strand-bias soft filter applied`.
>      >
>      > <div class="tiled-amplicon-Illumina-paired-end" markdown="1">
>      > For tiled-amplicon data, please consider the strand-bias filter experimental and proceed with the `Final (SnpEff-) annotated variants` collection as input here.
>      > </div>
>      > <div class="tiled-amplicon-ONT" markdown="1">
>      > For tiled-amplicon data, please consider the strand-bias filter experimental and proceed with the `Final (SnpEff-) annotated variants` collection as input here.
>      > </div>
>      > <div class="WGS-Illumina-paired-end" markdown="1">
>      > For WGS (i.e. non-ampliconic) data, use the `Final (SnpEff-) annotated variants with strand-bias soft filter applied` collection as input here to eliminate some likely false-positive variant calls.
>      > </div>
>      > <div class="WGS-Illumina-single-end" markdown="1">
>      > For WGS (i.e. non-ampliconic) data, use the `Final (SnpEff-) annotated variants with strand-bias soft filter applied` collection as input here to eliminate some likely false-positive variant calls.
>      > </div>
>      >
>      {: .comment}
>
>    - *"gene products translations"*: the uploaded `SARS-CoV-2 feature mapping` dataset
>
>      Remember, this mapping defines the gene names that will appear as affected by given mutations in the reports.
>    - *"Number of Clusters"*: `3`
>
>      The variant frequency plot generated by the workflow will separate the samples into this number of main clusters.
>
>      3 is an appropriate value for the suggested example batch of 16 samples.
>      If you are using your own data, and it consists of a lot less or more samples than the example batch, you might want to experiment with lower or higher values, respectively.
>      Generating an optimal plot sometimes requires some experimenting with this value, but once the workflow run is complete, you can always just rerun the one job that generates the plot and adjust this setting for that rerun.
>
>    The remaining workflow parameters *"AF Filter"*, *"DP Filter"*, and *"DP_ALT_FILTER"* define criteria for reporting mutations as reliable. They can all be left at their default values.
>
{: .hands_on}

The three key results datasets produced by the Reporting workflow are:

1. **Combined Variant Report by Sample**

   This table combines the key statistics for each mutation call in each individual sample. Each line in the dataset represents one mutation detected in one specific sample.

   This report is meant as a faster and more legible way for studying the collection of per-sample mutations in VCF format directly.

   > <details-title>Structure of the by-sample mutation report</details-title>
   >
   > Column | Field | Meaning
   > --- | --- | ---
   >  1  | `Sample` | Sample ID
   >  2  | `POS` | Position of site with regard to the reference genome
   >  3  | `FILTER` | Whether the variant passed the *AF*, *DP* and *DP_ALT* filters defined for the workflow run; variants that do not pass the filters in at least one sample are not reported, but a variant may fail some or all of the filters in *some* samples
   >  4  | `REF` |  Reference base
   >  5  | `ALT` | Alternative base
   >  6  | `DP` | Sequencing depth at the variant site
   >  7  | `AF` | Allele frequency
   >  8  | `AFcaller` | Uncorrected allele frequency emitted by variant caller; for Illumina data, may differ from `AF` due to a [bug](https://github.com/CSB5/lofreq/issues/80) in the lofreq variant caller
   >  8  | `SB` | Strand bias P-value from Fisher's exact test calculated by lofreq
   >  9  |`DP4` | Observed counts of 1. REF-supporting fw-strand reads, 2. REF-supporting rv-strand reads, 3. ALT-supporting fw-strand reads, 4. ALT-supporting rv-strand reads
   >  10 |`IMPACT` | Functional impact (from SnpEff annotation)
   >  11 |`FUNCLASS` | Funclass for change (from SnpEff annotation)
   >  12 |`EFFECT` | Effect of change (from SnpEff annotation)
   >  13 |`GENE` | Name of affected viral gene or ORF (from SnpEff annotation)
   >  14 |`CODON` | Affected codon (from SnpEff annotation)
   >  15 |`AA` | Amino acid change caused by the mutation (from SnpEff annotation)
   >  16 |`TRID` | Name of the affected viral protein (as defined in the feature mapping dataset)
   >  17 |`min(AF)` | Minimum alternate allele frequency across all samples containing this mutation
   >  18 |`max(AF)` | Maximum alternate allele frequency across all samples containing this mutation
   >  19 |`countunique(change)` | Number of distinct types of changes at this site across all samples; if > 1, other samples in the batch have the same site affected by different base changes
   >  20 |`countunique(FUNCLASS)` | Number of distinct FUNCLASS values at this site across all samples; if > 1, other samples in the batch have the same site affected by different base changes that have a different effect on the gene product
   >  21 |`change` | Nucleotide change at this site in this sample
   >
   {: .details}

2. **Combined Variant Report by Variant**

   This table aggregates the information about each mutation *across* samples. It is much more concise than the by-sample report at only a small amount of information loss.

   > <details-title>Structure of the across-samples mutation report</details-title>
   >
   > Column | Field | Meaning
   > --- | --- | ---
   >  1 | `POS` | Position of site with regard to the reference genome
   >  2 | `REF` | Reference base
   >  3 | `ALT` | Alternative base
   >  4 | `IMPACT` | Functional impact (from SnpEff annotation)
   >  5 | `FUNCLASS` | Funclass for change (from SnpEff annotation)
   >  6 | `EFFECT` | Effect of change (from SnpEff annotation)
   >  7 | `GENE` | Name of affected viral gene or ORF (from SnpEff annotation)
   >  8 | `CODON` | Affected codon (from SnpEff annotation)
   >  9 | `AA` | Amino acid change caused by the mutation (from SnpEff annotation)
   >  10 |`TRID` | Name of the affected viral protein (as defined in the feature mapping dataset)
   >  11 |`countunique(Sample)` | Number of distinct samples containing this change
   >  12 |`min(AF)` | Minimum alternate allele frequency across all samples containing this mutation
   >  13 |`max(AF)` | Maximum alternate allele frequency across all samples containing this mutation
   >  14 |`SAMPLES(above-thresholds)` | List of samples, in which this mutation was identified and in which the call passed the *AF*, *DP* and *DP_ALT* filters defined for the workflow run
   >  15 |`SAMPLES(all)` | List of all samples, in which this mutation was identified (including samples in which the call did not pass all filters)
   >  16 |`AFs(all)` | List of allele frequencies at which the mutation was observed in the samples in SAMPLES(all)
   >  17 |`change` |  Nucleotide change at this site
   >
   {: .details}

3. **Variant frequency plot**

   ![Variant frequency plot](../../images/sars-cov-2-variant-discovery/variant-frequency.svg)

   This plot represents the allele-frequencies (AFs, cell color) for the different mutations (columns) in the different samples (rows). Mutations are grouped by viral ORFs (different colors on the 1st row). Information about their impact on translation of viral proteins is color-coded on the second row. Sample clustering is indicated by the tree displayed on the left.

If you are working with your own data, we encourage you to just explore these three outputs on your own. If you are following along with the suggested batch of example data, then we have some guidance and questions prepared for you:

> <question-title>Some results exploration and questions</question-title>
>
> The variant frequency plot shows a large main cluster of samples (in the middle of the plot) flanked by two smaller ones.
> The small cluster at the bottom of the plot consists of only two samples, the cluster at the top of three samples.
>
> Questions
>
> 1. Do you recognize some of the sample identifiers in the two small clusters?
>
>    Use your browser's zoom function to magnify the plot enough for labels to become readable on your screen.
>
> 2. At least for **SRR17054505**, do you have an idea why its row in the plot might look like it does?
>
> 3. What is unique about sample **SRR17051933**?
>
> > <solution-title></solution-title>
> > 1. The two samples forming the smallest cluster, **SRR17054502** and **SRR17054506**, and two of the samples in the three-samples cluster, **SRR17054505** and **SRR17054508**, are exactly the four samples we observed as having particularly low coverage in the **Preprocessing and mapping reports** before.
> > 2. Only a single mutation was identified for **SRR17054505** (i.e. it is *very* different from all other samples).
> >
> >    (By zooming far enough into the plot, or by looking up the sample in the **Combined Variant Report by Sample** you can learn that this single variant is 27,807 C->T.)
> >
> >    From the **Preprocessing and mapping reports** we had learnt before that **SRR17054505** is the sample with critically low coverage. With hardly any sequence information available, even the best variant calling software will probably not be able to identify many mutations for this sample. Chances are that at least for this sample, lots of mutations have simply been missed.
> >
> >    Knowing that the other three samples have a bit higher coverage than **SRR17054505**, one might speculate that for them maybe only *some* mutations went unidentified, and that might explain why they end up outside the main cluster in the variant frequency plot. At the moment that is speculation, but we will revisit that idea later.
> >
> > 3. What is striking about **SRR17051933** are all the red cells it has in the variant frequency plot.
> >
> >    Many of these overlap with black cells of other samples, but some of the darkest cells for **SRR17051933** do not.
> >
> >    Cell color indicates the allele frequency observed for a mutation in a given sample, and pitch-black corresponds to an allele frequency of ~ 1, i.e. to mutations that were found on nearly every sequenced read that overlaps the site of the mutation. The lighter colors for almost all mutations found for **SRR17051933** mean that for this sample some reads at mutation sites confirm the mutation, but typically many do not. We can look at the mutations in this sample also in the **Combined Variant Report by Sample** and confirm that many of these mutations have really low values in the `AF` column.
> >
> >    What about the mutations that seem to be unique to **SRR17051933** from the plot?
> >    We can obtain a nice tabular report of them by filtering the **Combined Variant Report by Variant**.
> >
> >    Run {% tool [Filter data on any column using simple expressions](Filter1) %} with
> >
> >    - {% icon param-file %} *"Filter"*: **Combined Variant Report by Variant**
> >    - *"With following conidition"*: `c15 == "SRR17051933"`
> >    - *"Number of header lines to skip"*: `1`
> >
> >    Column 15 of **Combined Variant Report by Variant** is `SAMPLES(all)`, i.e. lists all samples for which a given mutation in the report has been identified (at any allele frequency). The filtering above will retain those lines from the report that list **SRR17051933** as the *only* sample.
> >
> >    When you inspect the output, you can see that it lists 17 **SRR17051933**-specific variants and you can read off the allele frequencies at which they were observed in the sample from column 13 (`max(AF)`).
> >    It is relatively obvious then that most of these variants have a AF value between 0.68 and 0.8, i.e. they are confirmed by the majority of reads, but not all of them.
> >
> >    We will come back to this observation and look for possible explanations near the end of the tutorial.
> >
> {: .solution}
{: .question}

## From mutations per sample to consensus sequences

We can now run a last workflow which generates reliable consensus sequences according to transparent criteria that capture at least some of the complexity of variant calling:

- Each consensus sequence produced by the workflow is guaranteed to capture all called, filter-passing mutations as defined in the VCF of its sample that reach a user-defined consensus allele frequency threshold.
- Filter-failing mutations and mutations below a second user-defined minimal allele frequency threshold are ignored.
- Genomic positions of filter-passing mutations with an allele frequency in between the two thresholds are hard-masked (with N) in the consensus sequence of their sample.
- Genomic positions with a coverage (calculated from the aligned reads for the sample) below another user-defined threshold are hard-masked, too, unless they are sites of consensus alleles.

> <hands-on-title>Import the consensus construction workflow into Galaxy</hands-on-title>
>
> Just like all workflows before, also the *consensus construction workflow* developed by the Galaxy Covid-19 project can be retrieved from *Dockstore* or *WorkflowHub*:
>
> {% snippet faqs/galaxy/workflows_import_search.md search_query='organization:"iwc" name:"sars-cov-2"' workflow_name="sars-cov-2-consensus-from-variation/COVID-19-CONSENSUS-CONSTRUCTION" disable_image="True" disable_video="True" box_type="none" %}
>
> Again, you can just select the latest version of the workflow, and, once imported, it should appear in your list of workflows under the name: **COVID-19: consensus construction**.
>
{: .hands-on}

The workflow takes the collection of called variants (one VCF dataset per input sample, same collection as used as input for the *reporting* workflow) and a collection of the corresponding aligned reads (for the purpose of calculating genome-wide coverage). Both collections have been generated by the *variation analysis* workflow.

> <hands-on-title>From mutations per sample to consensus sequences</hands-on-title>
>
> 1. Run the **COVID-19: consensus construction** {% icon workflow %} workflow with these parameters:
>
>    <div class="tiled-amplicon-Illumina-paired-end" markdown="1">
>
>    - *"Variant calls"*: `Final (SnpEff-) annotated variants`
>    </div>
>    <div class="tiled-amplicon-ONT" markdown="1">
>
>    - *"Variant calls"*: `Final (SnpEff-) annotated variants`
>    </div>
>    <div class="WGS-Illumina-paired-end" markdown="1">
>
>    - *"Variant calls"*: `Final (SnpEff-) annotated variants with strand-bias soft filter applied`
>    </div>
>    <div class="WGS-Illumina-single-end" markdown="1">
>
>    - *"Variant calls"*: `Final (SnpEff-) annotated variants with strand-bias soft filter applied`
>    </div>
>
>      The collection with variation data in VCF format; output of the first workflow
>
>      > <comment-title>Use the right collection of annotated variants!</comment-title>
>      > The variation analysis workflow should have generated *two* collections of annotated variants - one called `Final (SnpEff-) annotated variants`, the other one called `Final (SnpEff-) annotated variants with strand-bias soft filter applied`.
>      >
>      > <div class="tiled-amplicon-Illumina-paired-end" markdown="1">
>      > For tiled-amplicon data, please consider the strand-bias filter experimental and proceed with the `Final (SnpEff-) annotated variants` collection as input here.
>      > </div>
>      > <div class="tiled-amplicon-ONT" markdown="1">
>      > For tiled-amplicon data, please consider the strand-bias filter experimental and proceed with the `Final (SnpEff-) annotated variants` collection as input here.
>      > </div>
>      > <div class="WGS-Illumina-paired-end" markdown="1">
>      > For WGS (i.e. non-ampliconic) data, use the `Final (SnpEff-) annotated variants with strand-bias soft filter applied` collection as inpu

tools_json:
["Upload", "upload1", "Replace Text in entire line", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.3+galaxy1", "Filter data on any column using simple expressions", "Filter1", "Fasta regular expression finder", "toolshed.g2.bx.psu.edu/repos/mbernt/fasta_regex_finder/fasta_regex_finder/0.1.0", "Pangolin", "toolshed.g2.bx.psu.edu/repos/iuc/pangolin/pangolin/4.3+galaxy2", "Datamash", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.8+galaxy0", "Nextclade", "toolshed.g2.bx.psu.edu/repos/iuc/nextclade/nextclade/2.7.0+galaxy0"]


topic: variant-analysis
tutorial: somatic-variants
title: Identification of somatic and germline variants from tumor and normal sample pairs
difficulty: intermediate
time_estimation: 7h
requirements_json: []
tags_json: []
content_hash: 855018b90e09342881d4b60d84422ce6
last_modified: 2026-05-27T09:01:50.019707
zenodo_link: https://doi.org/10.5281/zenodo.2582555
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/somatic-variants/tutorial.html

content:
When sequencing genomic material from a human tumor, the underlying clinical or
research question typically is **what spectrum of mutations distinguishes this
tumor from healthy tissue**.

This question cannot be answered adequately just by comparing the tumor tissue
to the human reference genome because even in healthy tissue there will be many
thousands of variants compared to the reference genome. This is because every
individual inherits a unique pattern of that many variants from her parents. A
fundamental difference between these variants and the tumor-specific mutations
is that the former are present in the carrier's germline, while the latter have
been acquired somatically and will, thus, not be transmitted to offspring.
Therefor, we talk of germline variants to refer to variants present in healthy
and tumor tissue alike, and of somatic variants to refer to tumor-specific
variants. To be able to distinguish between these two types of variants always
requires a direct comparison of data from tumor and normal tissue samples. 

In addition to acquiring new variants, tumors can also lose or gain chromosomal
copies of variants found heterozygously in an individual's germline. This
phenomenon is termed *loss of heterozygosity* (LOH) because only one of the two
original alleles persists in the tumor (either in a hemizygous state if the
other allele is simply dropped, or in a homozygous state in the case of a
duplication of one allelic copy accompanied by loss of the other).
The detection of LOH events, again, is dependent on a comparison of tumor and
normal tissue data.

In this tutorial we are going to identify somatic and germline variants, as
well as variants affected by LOH, from a tumor and a normal sample of the same
patient. Our goal is to report the variant sites, and the genes affected by
them, annotated with the content of general human genetic and cancer-specific
databases. Ideally, this may provide insight into the genetic events driving
tumor formation and growth in the patient, and might be of prognostic and even
therapeutic value by revealing variants known to affect drug
resistance/sensitivity, tumor aggressiveness, *etc*.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data Preparation

First we need to upload and prepare some input data to analyze. The sequencing
reads we are going to analyze are from real-world data from a cancer patient's
tumor and normal tissue samples.
For the sake of an acceptable speed of the analysis, the original data has been
downsampled though to include only the reads from human chromosomes 5, 12 and
17.

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a meaningful name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import the following four files from
>    [Zenodo](https://zenodo.org/record/2582555):
>
>    ```
>    https://zenodo.org/record/2582555/files/SLGFSK-N_231335_r1_chr5_12_17.fastq.gz
>    https://zenodo.org/record/2582555/files/SLGFSK-N_231335_r2_chr5_12_17.fastq.gz
>    https://zenodo.org/record/2582555/files/SLGFSK-T_231336_r1_chr5_12_17.fastq.gz
>    https://zenodo.org/record/2582555/files/SLGFSK-T_231336_r2_chr5_12_17.fastq.gz
>    ```
>    where the first two files represent the forward and reverse reads sequence
>    data from a patient's **normal** tissue, and the last two represent the
>    data of the same patient's **tumor** tissue.
> 
>    Alternatively, the same files may be available on your Galaxy server
>    through a shared data library (your instructor may tell you so), in which
>    case you may prefer to import the data directly from there.
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="fastqsanger.gz" %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Check that all newly created datasets in your history have their datatypes assigned 
>    correctly, and fix any missing or wrong datatype assignment
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="fastqsanger.gz" %}
>
> 4. Rename the datasets and add appropriate tags to them
>
>    For datasets that you upload via a link, Galaxy will pick the link
>    address as the dataset name, which you will likely want to shorten to
>    just the file names.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
>    Large parts of the analysis in this tutorial will consist of
>    identical steps performed on the normal and on the tumor tissue data
>    in parallel.
>
>    To make it easier to keep track of which dataset represents which
>    branch of an analysis in a linear history, Galaxy supports dataset
>    tags. In particular, if you attach a tag starting with `#` to any
>    dataset, that tag will automatically propagate to any new dataset
>    derived from the tagged dataset.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
>    Before starting our analysis it is, thus, a good idea to tag the two
>    fastq datasets representing the normal tissue with, *e.g.*, `#normal`
>    and the two datasets representing the tumor tissue with, *e.g.*,
>    `#tumor`.
>
> 5. Import the reference genome with the `hg19` version of the sequences of
>    human chromosomes 5, 12 and 17:
>
>    ```
>    https://zenodo.org/record/2582555/files/hg19.chr5_12_17.fa.gz
>    ```
>
>    Make sure you specify the datatype as `fasta` in the import dialog.
>
>    > <comment-title>Shortcut</comment-title>
>    > You can skip this step if the Galaxy server you are working on offers
>    > a `hg19` version of the human reference genome with prebuilt indexes for
>    > *bwa-mem* **and** *samtools* (ask your instructor, or check the tools
>    > **Map with BWA-MEM** {% icon tool %} and **VarScan Somatic**
>    > {% icon tool %} if they list a `hg19` version as an option under
>    > *"(Using) reference genome"*).
>    {: .comment}
>
>    Alternatively, load the dataset from a shared data library.
>
> 6. Rename the reference genome
>
>    The reference genome you have imported above came as a compressed
>    file, but got unpacked by Galaxy to plain `fasta` format according to
>    your datatype selection. You may now wish to remove the `.gz` suffix
>    from the dataset name.
>
{: .hands_on}

# Quality control and mapping of NGS reads

Before starting our analysis, we would like to make sure that the input data
is of good quality, *i.e.*, that there haven't been any major issues during
DNA preparation, exon capture, or during actual sequencing. To avoid spurious
variant calls due to low input quality, we can ensure that all sequencing reads
used in the analysis meet some minimal quality criteria by trimming low-quality
parts off of the ends of reads and/or discarding reads of poor quality
altogether. The resulting set of polished reads then needs to be mapped to the
human reference genome because knowing the genomic positions that the bases of
a read provide evidence for is, of course, a prerequisite for variant calling.

> <comment-title>More on quality control and mapping</comment-title>
> If you would like to explore the topics of quality control and read mapping
> in detail, you should take a look at the separate
> [Quality Control]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %})
> and [Mapping]({% link topics/sequence-analysis/tutorials/mapping/tutorial.md %})
> tutorials.
> Here, we will only illustrate the concrete steps necessary for quality
> control and read mapping of our particular datasets.
{: .comment}

## Quality control

To perform quality control, we will use [Falco](https://falco.readthedocs.io/en/latest/) an efficiency-optimized rewrite of [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/).

> <hands-on-title>Quality control of the input datasets</hands-on-title>
> 1. Run {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} on each of your four fastq datasets
>       - {% icon param-files %} *"Short read data from your current history"*: all 4 FASTQ  datasets selected with **Multiple datasets**
>
>    {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>
>    When you start this job, eight new datasets (one with the calculated raw
>    data, another one with an html report of the findings for each input
>    dataset) will get added to your history.
>
> 2. Use {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0) %} to aggregate the raw **Falco** data of all four input datasets into one report
>      - In *"Results"*
>        - *"Which tool was used generate logs?"*: `FastQC` (no matter whether FastQC or Falco was used)
>        - In *"FastQC output"*
>           - *"Type of FastQC output?"*: `Raw data`
>           - {% icon param-files %} *"FastQC output"*: all four *RawData*
>             outputs of **Falco** {% icon tool %})
>
> 3. Inspect the *Webpage* output produced by the tool
>
>    > <question-title></question-title>
>    >
>    > 1. What do you think of the base qualities of the sequences?
>    > 2. Which aspect of the quality report is most puzzling to you?
>    >    (Hint: Have a look at the GC content of the reads)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The quality of the sequences looks promising. There are no
>    > >    discernible systematic problems with it.
>    > >
>    > >    Even the reverse reads, which are typically of somewhat poorer
>    > >    quality than the corresponding forward reads, look good on average. 
>    > > 2. The GC content plots of the forward and the reverse reads from both
>    > >    samples reveal a very peculiar bimodal distribution.
>    > >
>    > >    Typically, a non-normal distribution of the GC content of the reads
>    > >    from a sample is considered to hint at possible contamination.
>    > >    Here, however, we are dealing with sequencing data from captured
>    > >    exomes, *i.e*, the reads are not representing random sequences from
>    > >    a genome, but rather an arbitrary selection.
>    > >    In fact, the samples at hand were prepared using Agilent's
>    > >    SureSelect V5 technology for exome enrichment, and bimodal GC
>    > >    content distributions can be seen as a hallmark of that capture
>    > >    method in several publications (see, for example, Fig. 4C in
>    > >    [Meienberg et al., 2015](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4477645/)
>    > >    ).
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

## Read trimming and filtering
Although the raw reads used in this tutorial are of relatively good overall
quality already, we will apply read trimming and filtering to see if we can
improve things still a bit, but also to demonstrate the general concept.

> <hands-on-title>Read trimming and filtering of the normal tissue reads</hands-on-title>
> 1. Run {% tool [Trimmomatic](toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.36.5) %} to trim and filter the normal tissue reads
>    - *"Single-end or paired-end reads?"*: `Paired-end (two separate
>      input files)`
>      
>      This makes the tool treat the forward and reverse reads simultaneously.
>
>      - {% icon param-file %} *"Input FASTQ file (R1/first of pair)"*: the
>        forward reads (r1) dataset of the normal tissue sample
>      - {% icon param-file %} *"Input FASTQ file (R2/second of pair)"*: the
>        reverse reads (r2) dataset of the normal tissue sample
>
>    - *"Perform initial ILLUMINACLIP step?"*: `Yes`
>       - *"Select standard adapter sequences or provide custom?"*: `Standard`
>          - *"Adapter sequences to use"*: `TruSeq3 (paired-ended, for MiSeq and
>          HiSeq)`
>       - *"Maximum mismatch count which will still allow a full match to be
>        performed"*: `2`
>       - *"How accurate the match between the two 'adapter ligated' reads must
>        be for PE palindrome read alignment"*: `30`
>       - *"How accurate the match between any adapter etc. sequence must be
>        against a read"*: `10`
>       - *"Minimum length of adapter that needs to be detected (PE specific/
>        palindrome mode)"*: `8`
>       - *"Always keep both reads (PE specific/palindrome mode)?"*: `Yes`
>
>       These parameters are used to cut ILLUMINA-specific adapter sequences
>       from the reads.
>
>    - In *"Trimmomatic Operation"*
>       - In *"1: Trimmomatic Operation"*
>           -  *"Select Trimmomatic operation to perform"*: `Cut the specified number
>       of bases from the start of the read (HEADCROP)`
>              - *"Number of bases to remove from the start of the read"*: `3`
>       - {% icon param-repeat %} "Insert Trimmomatic Operation"*
>       - In *"2: Trimmomatic Operation"*
>           -  *"Select Trimmomatic operation to perform"*: `Cut bases off the end of
>       a read, if below a threshold quality (TRAILING)`
>              - *"Minimum quality required to keep a base"*: `10`
>       - {% icon param-repeat %} "Insert Trimmomatic Operation"*
>       - In *"3: Trimmomatic Operation"*
>           - *"Select Trimmomatic operation to perform"*: `Drop reads below a
>       specified length (MINLEN)`
>              - *"Minimum quality required to keep a base"*: `25`
>
>       These three trimming and filtering operations will be applied to the
>       reads in the given order after adapter trimming.
>    
{: .hands_on}

Running this job will generate four output datasets:
- two for the trimmed forward and reverse reads that still have a proper
  mate in the other dataset
- two more datasets of orphaned forward and reverse reads, for which the
  corresponding mate got dropped because of insufficient length after
  trimming; when you inspect these two files, however, you should find
  that they are empty because none of our relatively high quality reads
  got trimmed that excessively. You can delete the two datasets to keep
  your history more compact.

> <details-title>More on handling unpaired reads in paired-end data</details-title>
> Splitting out potential unpaired reads into separate datasets like this
> is important because read mappers, typically, expect reads in forward
> and reverse input datasets to be arranged in proper pairs, and reads
> in one of the datasets without a counterpart in the other would destroy
> that expected structure. Therefor, when your data is paired-end data
> always make sure you use Trimmomatic in paired-end mode, too!
{: .details}

> <hands-on-title>Read trimming and filtering of the tumor tissue reads</hands-on-title>
> 2. Trim and filter the **tumor tissue** reads following the same steps as above, just change the two input datasets to treat the tumor tissue reads with identical settings.
>
> 3. Check that the two unpaired reads datasets are empty, and delete them.   
{: .hands_on}

> <hands-on-title>Exercise: Quality control of the polished datasets</hands-on-title>
> Use {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} and {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0) %} like before,
> but using the four trimmed datasets produced by Trimmomatic as input.
>
>    > <question-title></question-title>
>    >
>    > How did read trimming affect the quality reports?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > As expected, trimming the relatively high-quality raw reads did not
>    > > have any substantial impact on average. A small fraction of successful
>    > > adapter removal is visible though.
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

## Read Mapping

> <hands-on-title>Read Mapping</hands-on-title>
> 1. Use {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.1) %} to map the reads from the **normal tissue** sample to the reference genome
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>        - *"Using reference genome"*: `Human: hg19` (or a similarly named option)
> 
>      > <comment-title>Using the imported `hg19` sequence</comment-title>
>      > If you have imported the `hg19` sequence as a fasta dataset into your
>      > history instead:
>      >   - *"Will you select a reference genome from your history or use a
>      >     built-in index?"*: `Use a genome from history and build index`
>      >      - {% icon param-file %} *"Use the following dataset as the reference sequence"*: your imported `hg19` fasta dataset.
>      {: .comment}
>
>    - *"Single or Paired-end reads"*: `Paired`
>       - {% icon param-file %} *"Select first set of reads"*: the trimmed 
>         forward reads (r1) dataset of the **normal tissue** sample; output of
>         **Trimmomatic** {% icon tool %}
>       - {% icon param-file %} *"Select second set of reads"*: the trimmed
>         reverse reads (r2) dataset of the **normal tissue** sample; output of
>         **Trimmomatic** {% icon tool %}
>    - *"Set read groups information?"*: `Set read groups (SAM/BAM specification)`
>      - *"Auto-assign"*: `No`
>        - *"Read group identifier (ID)"*: `231335` (this value being taken
>          from the original name of the normal tissue input files)
>      - *"Auto-assign"*: `No`
>        - *"Read group sample name (SM)"*: `Normal`
>
>    > <comment-title>More on read group identifiers and sample names</comment-title>
>    > In general, you are free to choose ID and SM values to your liking,
>    > but the ID should unambiguously identify the sequencing run that
>    > produced the reads, while the SM value should identify the
>    > biological sample.
>    {: .comment}
>
> 2. Use {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.1) %} to map the reads from the **tumor tissue** sample,
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>        - *"Using reference genome"*: `Human: hg19` (or a similarly named option)
>
>      Adjust these settings as before if you are using the imported reference
>      genome.
>    - *"Single or Paired-end reads"*: `Paired`
>       - {% icon param-file %} *"Select first set of reads"*: the trimmed
>         forward reads (r1) dataset of the **tumor tissue** sample; output of
>         **Trimmomatic** {% icon tool %}
>       - {% icon param-file %} *"Select second set of reads"*: the reverse
>         reads (r2) dataset of the **tumor tissue** sample; output of
>         **Trimmomatic** {% icon tool %}
>    - *"Set read groups information?"*: `Set read groups (SAM/BAM specification)`
>      - *"Auto-assign"*: `No`
>        - *"Read group identifier (ID)"*: `231336` (this value, again, being
>          taken from the original name of the tumor tissue input files)
>      - *"Auto-assign"*: `No`
>        - *"Read group sample name (SM)"*: `Tumor`
>
{: .hands_on}

# Mapped reads postprocessing

To ensure that we base our variant analysis only on unambiguous, high-quality
read mappings we will do some postprocessing next.

## Filtering on mapped reads properties

To produce new filtered BAM datasets with only those reads retained that have
been mapped to the reference successfully, have a minimal mapping quality of 1,
and for which the mate read has also been mapped:

> <hands-on-title>Filtering for mapping status and quality</hands-on-title>
>
> 1. Run {% tool [Filter BAM datasets on a variety of attributes](toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.4.1) %} with the following parameters:
>   - {% icon param-files %} *"BAM dataset(s) to filter"*: mapped reads
>     datasets from the normal *and* the tumor tissue data, outputs of
>     **Map with BWA-MEM** {% icon tool %}
>   - In *"Condition"*:
>     - In *"1: Condition"*:
>       - In *"Filter"*:
>         - In *"1: Filter"*:
>           - *"Select BAM property to filter on"*: `mapQuality`
>             - *"Filter on read mapping quality (phred scale)"*: `>=1`
>         - In *"2: Filter"*:
>           - *"Select BAM property to filter on"*: `isMapped`
>             - *"Selected mapped reads"*: `Yes`
>         - Click on *"Insert Filter"*
>         - In *"3: Filter"*:
>           - *"Select BAM property to filter on"*: `isMateMapped`
>             - *"Select reads with mapped mate"*: `Yes`
>
>       When you configure multiple filters within one condition, reads have
>       to pass *all* the filters to be retained in the output. The above
>       settings, thus, retain only read pairs, for which both mates are
>       mapped.
>
>       Note that filtering for a minimal mapping quality is not strictly
>       necessary. Most variant callers (including **VarScan somatic, which we
>       will be using later) have an option for using only reads above a
>       certain mapping quality. In this section, however, we are going to
>       process the retained reads further rather extensively so it pays off
>       in terms of performance to eliminate reads we do not plan to use at an
>       early step.
>
>   - *"Would you like to set rules?"*: `No`
>
{: .hands_on}

This will result in two new datasets, one for each of the normal and tumor data.

> <details-title>Tools for filtering BAM datasets</details-title>
> The related tutorial on variant detection from exome-seq data, demonstrates
> nearly identical filtering of BAM datasets using the tool **Filter BAM
> datasets on a variety of attributes** {% icon tool %}.
>
> These two tools offer very similar functionality and can often be used
> interchangeably. Under the hood, **Filter SAM or BAM, output SAM or BAM**
> {% icon tool %} uses the `samtools view` command line tool, while **Filter
> BAM datasets on a variety of attributes** {% icon tool %} uses `bamtools
> filter`. Whatever the BAM filtering task, you should be able to perform it in
> Galaxy with one of these two tools.
>
{: .details}

## Removing duplicate reads

> <hands-on-title>Remove duplicates</hands-on-title>
>
> 1. Run {% tool [RmDup](toolshed.g2.bx.psu.edu/repos/devteam/samtools_rmdup/samtools_rmdup/2.0.1) %} with the following parameters:
>   - {% icon param-files %} *"BAM file"*: filtered reads datasets from
>     the normal *and* the tumor tissue data; the outputs of
>     **Filter SAM or BAM**
>   - *"Is this paired-end or single end data"*: `BAM is paired-end`
>     - *"Treat as single-end"*: `No`
>
{: .hands_on}

Again, this will produce two new datasets, one for each of the normal and
tumor data.

## Left-align reads around indels

> <hands-on-title>Left-align</hands-on-title>
>
> 1. Run {% tool [BamLeftAlign](toolshed.g2.bx.psu.edu/repos/devteam/freebayes/bamleftalign/1.3.1) %} with the following parameters:
>    - *"Choose the source for the reference genome"*: `Locally cached`
>      - {% icon param-files %} *"BAM dataset to re-align"*: your
>        filtered and deduplicated reads datasets from the normal *and* the tumor
>        tissue data; the outputs of **RmDup**
>      - *"Using reference genome"*: `Human: hg19` (or a similarly
>        named choice)
>    - *"Maximum number of iterations"*: `5`
>
> > <comment-title>Using the imported `hg19` sequence</comment-title>
> > If you have imported the `hg19` sequence as a fasta dataset into your
> > history instead:
> >   - *"Choose the source for the reference genome"*: `History`
> >      - {% icon param-file %} *"Using reference file"*: your imported `hg19` fasta dataset
> {: .comment}
{: .hands_on}

As before, this will generate two new datasets, one for each of the normal and tumor data.

## Recalibrate read mapping qualities

> <hands-on-title>Recalibrate read quality scores</hands-on-title>
>
> 1. Run {% tool [CalMD](toolshed.g2.bx.psu.edu/repos/devteam/samtools_calmd/samtools_calmd/2.0.2) %} with the following parameters:
>    - {% icon param-files %} *"BAM file to recalculate"*: the left-aligned
>      datasets from the normal and the tumor tissue data; the outputs of
>      **BamLeftAlign** {% icon tool %}
>    - *"Choose the source for the reference genome"*: `Use a built-in genome`
>      - *"Using reference genome"*: `Human: hg19` (or a similarly named choice)
>
>      > <comment-title>Using the imported `hg19` sequence</comment-title>
>      > If you have imported the `hg19` sequence as a fasta dataset into your
>      > history instead:
>      >   - *"Choose the source for the reference genome"*: `Use a genome from the history`
>      >      - {% icon param-file %} *"Using reference file"*: your imported `hg19` fasta dataset.
>      {: .comment}
>
>    - *"Do you also want BAQ (Base Alignment Quality) scores to be calculated?"*: `No`
>
>      The *VarScan somatic* tool that we are going to use for calling variants at
>      the next step is typically used in combination with unadjusted base quality
>      scores because the general opinion is that the base quality downgrades
>      performed by *CalMD* and other tools from the *samtools* suite of tools
>      are too severe for *VarScan* to retain good sensitivity. We are sticking
>      with this practice in this tutorial.
>
>      > <comment-title>Using adjusted base quality scores</comment-title>
>      > If, for your own data, you would like to experiment with adjusted base
>      > quality scores, it is important to understand that *VarScan somatic* will
>      > only make use of the adjusted scores if they are incorporated directly into
>      > the read base qualities of a BAM input dataset, but not if they are written
>      > to the dataset separately. 
>      > 
>      > Hence, should you ever decide to use:
>      > - *"Do you also want BAQ (Base Alignment Quality) scores to be calculated?"*: `Yes, run BAQ calculation`
>      >
>      >   and you want this setting to affect downstream variant calling with
>      >   *VarScan somatic*, make sure you also set then:
>      >   - *"Use BAQ to cap read base qualities"*: `Yes`
>      >
>      > Please also note that BAQ scores are quite expensive to calculate so
>      > be prepared to see a substantial (up to 10x!) increase in job run time 
>      > when enabling it.
>      {: .comment}
>
>    - *"Additional options"*: `Advanced options`
>      - *"Change identical bases to '='"*: `No`
>      - *"Coefficient to cap mapping quality of poorly mapped reads"*: `50`
>
>        This last setting is the real reason why we use CalMD at this point.
>        It is an empirical, but well-established finding that the mapping
>        quality of reads mapped with *bwa* should be capped this way before
>        variant calling.
>
{: .hands_on}

This will, once more, produce two new datasets, one for each of the normal
and tumor data.

## Refilter reads based on mapping quality

During recalibration of read mapping qualities **CalMD** may have set some
mapping quality scores to 255. This special value is reserved for *undefined*
mapping qualities and is used by the tool when a recalibrated mapping quality
would drop below zero. In other words, a value of 255 does not indicate a
particularly good mapping score, but a really poor one.
To remove such reads from the data:

> <hands-on-title>Eliminating reads with undefined mapping quality</hands-on-title>
>
> 1. Run {% tool [Filter BAM datasets on a variety of attributes](toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.4.1) %} with the following parameters:
>   - {% icon param-files %} *"BAM dataset(s) to filter"*: the recalibrated
>     datasets from the normal and the tumor tissue data; the outputs of
>     **CalMD** {% icon tool %}
>   - In *"Condition"*:
>     - In *"1: Condition"*:
>       - In *"Filter"*:
>         - In *"1: Filter"*:
>           - *"Select BAM property to filter on"*: `mapQuality`
>             - *"Filter on read mapping quality (phred scale)"*: `<=254`
>
{: .hands_on}


# Variant calling and classification

Having generated a high-quality set of mapped read pairs, we can proceed to
variant calling. The tool **VarScan somatic** is a dedicated solution for
somatic variant calling that:

- detects variant alleles in tumor/normal sample pairs
- calls sample genotypes at variant sites
- classifies variants into germline, somatic and LOH event variants using
  solid classical statistics even in the presence of non-pure samples like
  those obtained from biopsies

> <hands-on-title>Variant calling and classification</hands-on-title>
>
> 1. Run {% tool [VarScan somatic](toolshed.g2.bx.psu.edu/repos/iuc/varscan_somatic/varscan_somatic/2.4.3.6) %} with the following parameters:
>    - *"Will you select a reference genome from your history or use a built-in
>   genome?"*: `Use a built-in genome`
>      - *"reference genome"*: `Human: hg19` (or a similarly named choice)
>
>      > <comment-title>Using the imported `hg19` sequence</comment-title>
>      > If you have imported the `hg19` sequence as a fasta dataset into your
>      > history instead:
>      >   - *"Will you select a reference genome from your history or use a
>      >     built-in genome?"*: `Use a genome from the history`
>      >      - {% icon param-file %} *"reference genome"*: your imported `hg19` fasta dataset.
>      {: .comment}
>
>    - {% icon param-file %} *"aligned reads from normal sample"*: the mapped
>      and fully post-processed normal tissue dataset; one of the two outputs
>      of filtering the **CalMD** {% icon tool %} outputs
>    - {% icon param-file %}*"aligned reads from tumor sample"*: the mapped
>      and fully post-processed tumor tissue dataset; the other output of
>      filtering the **CalMD** {% icon tool %} outputs
>    - *"Estimated purity (non-tumor content) of normal sample"*: `1`
>    - *"Estimated purity (tumor content) of tumor sample"*: `0.5`
>    - *"Generate separate output datasets for SNP and indel calls?"*: `No`
>    - *"Settings for Variant Calling"*: `Customize settings`
>      - *"Minimum base quality"*: `28`
>
>        We have seen, at the quality control step, that our sequencing data is of
>        really good quality, and we have chosen not to downgrade base qualities at
>        the quality scores recalibration step above, so we can increase the base
>        quality required at any given position without throwing away too much of
>        our data. 
>
>      - *"Minimum mapping quality"*: `1`
>
>        During postprocessing, we have filtered our reads for ones with a
>        mapping quality of at least one, but **CalMD** may have lowered some
>        mapping qualities to zero afterwards.
>
>      Leave all other settings in this section at their default values.
>    - *"Settings for Posterior Variant Filtering"*: `Use default values`
{: .hands_on}

# Variant annotation and reporting

For this tutorial we are going to use variant and gene annotations from many
different sources. Most of these are handled for us by the tools we are going
to use in this section, but we need to take care of importing the data from
four sources into Galaxy separately:

- variant annotations from [Cancer Hotspots](https://www.cancerhotspots.org)
- variant and gene information from the
  [Cancer Biomarkers database](https://www.cancergenomeinterpreter.org/biomarkers)
  of the Cancer Genome Interpreter (CGI) project
- variant and gene information from the [CIViC](https://civicdb.org) database
- variant annotations from [dbSNP](https://www.ncbi.nlm.nih.gov/snp)
- lists of genes annotated with the keywords *proto-oncogene* or *tumor
  suppressor* at [UniProt](https://www.uniprot.org/)

Each of these annotation sets has been released either in the public domain or
under a free data license, which allows you to use them as part of this
tutorial, but also for other purposes.

Starting from the data downloaded from each of these sites, we have generated
a set of new data files tailored to the requirements of the workflow of this
tutorial and have made them available through Zenodo, again under a free data
license.

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Import the following **variant annotation files** from
>    [Zenodo](https://zenodo.org/record/7962928):
>
>    ```
>    https://zenodo.org/record/7962928/files/hotspots.bed
>    https://zenodo.org/record/7962928/files/cgi_variant_positions.bed
>    https://zenodo.org/record/7962928/files/01-Feb-2019-CIVic.bed
>    https://zenodo.org/record/2582555/files/dbsnp.b147.chr5_12_17.vcf.gz
>    ```
>
>    Make sure you select `bed` as the datatype for the first three files and
>    `vcf` for the last file.
>
>    Alternatively, add the files from a shared data library on your Galaxy
>    server instance.
>
> 2. Import some **gene-level annotation files** from
>    [Zenodo](https://zenodo.org/record/2581881):
>
>    ```
>    https://zenodo.org/record/2581881/files/Uniprot_Cancer_Genes.13Feb2019.txt
>    https://zenodo.org/record/2581881/files/cgi_genes.txt
>    https://zenodo.org/record/2581881/files/01-Feb-2019-GeneSummaries.tsv
>    ```
>
>    and make sure you select `tabular` as their datatype, or add them from
>    the shared data library.
>
> 3. Download **SnpEff functional genomic annotations**
>
>    > <comment-title>Shortcut</comment-title>
>    > You can skip this step if the Galaxy server you are working on offers
>    > `Homo sapiens: hg19` as a locally installed snpEff database. You can
>    > check the **SnpEff eff** {% icon tool%} tool under **Genome source** to
>    > see if this is the case.
>    {: .comment}
>
>    Use **SnpEff Download** {% icon tool %} to download genome annotation
>    database `hg19`.
>
> 4. Check that the datatypes of all new datasets have been set correctly, and
>    change them if necessary. You may also want to shorten some of the
>    dataset names.
>
{: .hands_on}

## Adding annotations to the called variants

### Adding functional genomic annotations

Certainly, not all variants are equal. Many may just be silent mutations with
no effect at the amino acid level, while a few others may be disrupting the
coding sequence of a protein by introducing a premature stop codon or a
frameshift. Of course, it is also important to know *which* gene is affected
by a variant.
Such functional genomic annotations can be added to a VCF dataset of variants
with *SnpEff*.

> <hands-on-title>Adding annotations with SnpEff</hands-on-title>
> 1. Run {% tool [SnpEff eff](toolshed.g2.bx.psu.edu/repos/iuc/snpeff/snpEff/4.3+T.galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Sequence changes (SNPs, MNPs, InDels)"*: the
>      output of **VarScan somatic** {% icon tool %}
>    - *"Input format"*: `VCF`
>    - *"Output format"*: `VCF (only if input is VCF)`
>    - *"Genome source"*: `Locally installed reference genome`
>       - *"Genome"*: `Homo sapiens: hg19` (or a similarly named option)
>
>      > <comment-title>Using the imported `hg19` SnpEff genome database</comment-title>
>      > If you have imported the `hg19` SnpEff genome database into your
>      > history instead:
>      >   - *"Genome source"*: `Downloaded snpEff database in your history`
>      >      - {% icon param-file %} *"SnpEff4.3 Genome Data"*: your imported `hg19` SnpEff dataset.
>      {: .comment}
>
>    - *"Produce Summary Stats"*: `No`
> 
{: .hands_on}

### Adding genetic and clinical evidence-based annotations

Other interesting pieces of information about a variant include aspects like
whether the variant has been observed before in the human population and,
if so, at which frequency. If a variant is known to be associated with specific
diseases, we would also very much like to know that.
To proceed with this kind of genetic and clinical evidence-based annotations,
we are going to convert our list of variants into a database that can be
handled more efficiently than a VCF dataset. We will use the *GEMINI* tool
suite for this task and for all further work with the variants.

> <hands-on-title>Creating a GEMINI database from a variants dataset</hands-on-title>
> 1. Run {% tool [GEMINI load](toolshed.g2.bx.psu.edu/repos/iuc/gemini_load/gemini_load/0.20.1+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"VCF dataset to be loaded in the GEMINI database"*:
>      the output of **SnpEff eff** {% icon tool %}
>    - *"The variants in this input are"*: `annotated with snpEff`
>    - *"This input comes with genotype calls for its samples"*: `Yes`
>
>      Calling sample genotypes was part of what we used VarScan somatic for.
>
>    - *"Choose a gemini annotation source"*: select the latest available annotations snapshot (most likely, there will be only one)
>    - *"Sample and family information in PED format"*: `Nothing selected`
>    - *"Load the following optional content into the database"* 
>      - {% icon param-check %} *"GERP scores"*
>      - {% icon param-check %} *"CADD scores"*
>      - {% icon param-check %} *"Gene tables"*
>      - {% icon param-check %} *"Sample genotypes"*
>      - {% icon param-check %} *"variant INFO field"*
>
>      Be careful to leave **unchecked**:
>      - *"Genotype likelihoods (sample PLs)"*
>
>         VarScan somatic does not generate these values
>
>      - *"only variants that passed all filters"*
>
>        It is simple and more flexible to filter for this property later
{: .hands_on}

During the creation of the database *GEMINI* already (silently) adds an
impressive amount of annotations it knows about to our variants (including,
*e.g.*, the frequency at which every variant has been observed in large human
genome sequencing projects). We have gotten all of these for free, just by converting the variants to a GEMINI database!

*GEMINI* also extracts a lot of the information stored in the VCF input dataset for us (such as the functional genomic annotations that we added with
*SnpEff*).

However, there are typically additional annotations from other sources (not
incorporated into *GEMINI*) that one would like to add. In addition, *GEMINI*
is not prepared to extract some non-standard information from VCF datasets,
including some important bits added by *VarScan somatic*.

**GEMINI annotate** {% icon tool %} is the tool that is designed to help you
with these rather common issues. It lets you add further annotations to the
variants in an already loaded GEMINI database.

As a first step, we are going to use the tool to add some crucial information
generated by *VarScan somatic*, but not recognized by *GEMINI load*, to the
database. Specifically, we are interested in three values calculated by
*VarScan somatic* for each variant it detected:

- *Somatic status (SS)*

  This is a simple numeric code, in which a value of `1` indicates a germline
  variant, `2` a somatic variant and `3` an LOH event.
  
- *Germline p-value (GPV)*

  For variants with a somatic status of 1, this is the error probability
  associated with that status call.
  
- *Somatic p-value (SPV)*

  This is the error probability associated with status calls of 2 and 3
  (somatic and LOH calls).
  
These values are encoded in the *INFO* column of the VarScan-generated VCF
dataset and we are going to extract them from there and add them to the
*GEMINI* database.

> <details-title>Handling of the VCF INFO field by GEMINI load</details-title>
> If you paid close attention to how we generated the GEMINI database, you
> might remember that, under *"Load the following optional content into the
> database"*, we checked the option {% icon param-check %} *"variant INFO
> field"*. Did the tool not do what it was supposed to do?
>
> The answer is that it did, but not in the way you may expect.
>
> In fact, *GEMINI load* always extracts INFO field elements it knows about and
> stores them into predefined columns of the variants table in the database.
> In addition, it can store the *whole* INFO field of each variant into a
> separate *info* column so that no information gets lost on the way from the
> VCF input to the GEMINI database, and that's the optional behavior we have
> been asking for. To keep things compact, however, this column stores the INFO
> content in compressed form, which is not readily accessible.
>
> That's why it is necessary to extract non-standard INFO field elements
> explicitly if they are supposed to be used for filtering and querying.
{: .details}

> <hands-on-title>Making variant call statistics accessible</hands-on-title>
> 1. Run {% tool [GEMINI annotate](toolshed.g2.bx.psu.edu/repos/iuc/gemini_annotate/gemini_annotate/0.20.1+galaxy2) %}:
>    - {% icon param-file %} *"GEMINI database"*: output of **GEMINI load**
>    - {% icon param-file %} *"Dataset to use as the annotation source"*: output of **VarScan somatic**
>    - *"Strict variant-identity matching of database and annotation records
>   (VCF format only)"*: `Yes`
>
>      This setting does not really matter here since you have built the GEMINI
>      database from the exact same list of variants that we are now retrieving
>      annotations from and because VarScan somatic does not call multiple alleles
>      at single sites. Matching on variant-identity is the behavior we would
>      like to see though, so we may as well be explicit about it.
>
>    - *"Type of information to add to the database variants"*: `Specific values
>      extracted from matching records in the annotation source (extract)`
>
>      - In *"1: Annotation extraction recipe"*:
>        - *"Elements to extract from the annotation source"*: `SS`
>        - *"Database column name to use for recording annotations"*:
>        `somatic_status`
>        - *"What type of data are you trying to extract?"*: `Integer numbers`
>        - *"If multiple annotations are found for the same variant,
>        store ..."*: `the first annotation found`
>
>        This is the recipe for extracting the VarScan-generated *SS* field and
>        adding it as a new column *somatic_status* to the GEMINI database.
>
>      - {% icon param-repeat %} *"Insert Annotation extraction recipe"*
>      - In *"2: Annotation extraction recipe"*:
>        - *"Elements to extract from the annotation source"*: `GPV`
>        - *"Database column name to use for recording annotations"*:
>       `germline_p`
>        - *"What type of data are you trying to extract?"*: `Numbers with
>        decimal precision`
>        - *"If multiple annotations are found for the same variant,
>        store ..."*: `the first annotation found`
>
>        This is the recipe for extracting the VarScan-generated *GPV* field
>        and adding it as a new column *germline_p* to the GEMINI database.
>
>      - {% icon param-repeat %} *"Insert Annotation extraction recipe"*
>      - In *"3: Annotation extraction recipe"*:
>        - *"Elements to extract from the annotation source"*: `SPV`
>        - *"Database column name to use for recording annotations"*:
>       `somatic_p`
>        - *"What type of data are you trying to extract?"*: `Numbers with
>        decimal precision`
>        - *"If multiple annotations are found for the same variant,
>        store ..."*: `the first annotation found`
>
>        This is the recipe for extracting the VarScan-generated *SPV* field
>        and adding it as a new column *somatic_p* to the GEMINI database.
>
{: .hands_on}

Next, we are going to add additional annotations beyond the ones directly
obtainable through *GEMINI* or from the input VCF dataset. Specifically we
want to add:

- more information from **dbSNP**

  As part of the database creation process, GEMINI already checks all
  variants whether they occur in dbSNP and, if so, stores their dbSNP IDs.
  Hence, we only need to extract some additional information of interest.

- information from **Cancer Hotspots**
- links to the **CIViC** database
- information from the **Cancer Genome Interpreter (CGI)**

> <hands-on-title>Adding further annotations</hands-on-title>
> 1. Run {% tool [GEMINI annotate](toolshed.g2.bx.psu.edu/repos/iuc/gemini_annotate/gemini_annotate/0.20.1+galaxy2) %} to add further annotations from **dbSNP**
>    - {% icon param-file %} *"GEMINI database"*: the output of the last
>      **GEMINI annotate** {% icon tool %} run
>    - {% icon param-file %} *"Dataset to use as the annotation source"*: the imported `dbsnp.b147.chr5_12_17.vcf`
>    - *"Strict variant-identity matching of database and annotation records
>      (VCF format only)"*: `Yes`
>   
>      dbSNP stores information about specific SNPs observed in human
>      populations and we would like to record if any exact same SNPs are among
>      our variants.
>
>    - *"Type of information to add to the database variants"*: `Specific
>      values extracted from matching records in the annotation source
>      (extract)`
>      - In *"1: Annotation extraction recipe"*:
>        - *"Elements to extract from the annotation source"*: `SAO`
>        - *"Database column name to use for recording annotations"*:
>        `rs_ss`
>        - *"What type of data are you trying to extract?"*: `Integer numbers`
>        - *"If multiple annotations are found for the same variant,
>        store ..."*: `the first annotation found`
>
>        This recipe extracts the dbSNP *SAO* field and adds it as *rs_ss* to
>        the GEMINI database.
>
> 2. Run {% tool [GEMINI annotate](toolshed.g2.bx.psu.edu/repos/iuc/gemini_annotate/gemini_annotate/0.20.1+galaxy2) %} to add further annotations from **cancerhotspots**
>    - {% icon param-file %} *"GEMINI database"*: the output of the last
>      **GEMINI annotate** {% icon tool %} run
>    - {% icon param-file %} *"Dataset to use as the annotation source"*: the imported `cancerhotspots_v2.bed`
>    - *"Strict variant-identity matching of database and annotation records
>      (VCF format only)"*: `Yes` (with input in BED format this setting will
>      be ignored)
>
>      For the cancerhotspots data, we are simply going to record the best
>      q-value associated with any cancerhotspots variant overlapping one
>      of our variant sites.
>
>    - *"Type of information to add to the database variants"*: `Specific
>      values extracted from matching records in the annotation source
>      (extract)`
>      - In *"1: Annotation extraction recipe"*:
>        - *"Elements to extract from the annotation source"*: `5`
>
>          The q-values are stored in the fifth column of the BED dataset.
>
>        - *"Database column name to use for recording annotations"*:
>        `hs_qvalue`
>        - *"What type of data are you trying to extract?"*: `Numbers with
>         decimal precision`
>        - *"If multiple annotations are found for the same variant,
>        store ..."*: `the smallest of the (numeric) values`
>
>        This is the recipe for extracting the *q-values* of overlapping
>        cancerhotspots sites and adding them as *hs_qvalue* to the GEMINI
>        database.
>
> 3. Run {% tool [GEMINI annotate](toolshed.g2.bx.psu.edu/repos/iuc/gemini_annotate/gemini_annotate/0.20.1+galaxy2) %} to add links to **CIViC**
>    - {% icon param-file %} *"GEMINI database"*: the output of the last
>      **GEMINI annotate** {% icon tool %} run
>    - {% icon param-file %} *"Dataset to use as the annotation source"*: the imported `CIViC.bed`
>    - *"Strict variant-identity matching of database and annotation records
>      (VCF format only)"*: `Yes` (with input in BED format this setting will
>      be ignored)
>
>      For the CIViC data, we are going to record the hyperlink to any variant
>      found in the CIViC database that overlaps one of our variant sites.
>
>    - *"Type of information to add to the database variants"*: `Specific
>      values extracted from matching records in the annotation source
>      (extract)`
>      - In *"1: Annotation extraction recipe"*:
>        - *"Elements to extract from the annotation source"*: `4`
>
>          The hyperlinks are stored in the fourth column of the BED dataset.
>
>        - *"Database column name to use for recording annotations"*:
>        `overlapping_civic_url`
>        - *"What type of data are you trying to extract?"*: `Text (text)`
>        - *"If multiple annotations are found for the same variant,
>        store ..."*: `a comma-separated list of non-redundant (text) values`
>
>        This is the recipe for extracting the hyperlinks of overlapping CIViC
>        sites and adding them as a list of *overlapping_civic_urls* to the
>        GEMINI database.
>
> 3. Run {% tool [GEMINI annotate](toolshed.g2.bx.psu.edu/repos/iuc/gemini_annotate/gemini_annotate/0.20.1+galaxy2) %} to add information from the **Cancer Genome Interpreter (CGI)**
>    - {% icon param-file %} *"GEMINI database"*: the output of the last
>      **GEMINI annotate** {% icon tool %} run
>    - {% icon param-file %} *"Dataset to use as the annotation source"*: the imported `cgi_variant_positions.bed`
>    - *"Strict variant-identity matching of database and annotation records
>      (VCF format only)"*: `Yes` (with input in BED format this setting will
>      be ignored)
>
>      For the CGI data, we are going to record if any of our variant sites
>      is overlapped by a variant in the CGI biomarkers database.
>
>    - *"Type of information to add to the database variants"*: `Binary
>      indicator (1=found, 0=not found) of whether the variant had any match in
>      the annotation source (boolean)`
>      - *"Database column name to use for recording annotations"*: `in_cgidb`
>
{: .hands_on}

## Reporting selected subsets of variants

Now that we have built our GEMINI database and enriched it with additional
annotations, it is time that we explore the wealth of information stored in it.
The goal is to produce filtered variant reports that list specific classes
(somatic, germline, LOH) of high-quality variants together with their most
relevant annotations. The way to achieve this is through **GEMINI queries**
that specify:

1. filters we want to apply to the variant list stored in the database
2. the pieces of 

tools_json:
["Falco", "toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0", "Trimmomatic", "toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.36.5", "Map with BWA-MEM", "toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.1", "Filter BAM datasets on a variety of attributes", "toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.4.1", "RmDup", "toolshed.g2.bx.psu.edu/repos/devteam/samtools_rmdup/samtools_rmdup/2.0.1", "BamLeftAlign", "toolshed.g2.bx.psu.edu/repos/devteam/freebayes/bamleftalign/1.3.1", "CalMD", "toolshed.g2.bx.psu.edu/repos/devteam/samtools_calmd/samtools_calmd/2.0.2", "VarScan somatic", "toolshed.g2.bx.psu.edu/repos/iuc/varscan_somatic/varscan_somatic/2.4.3.6", "SnpEff eff", "toolshed.g2.bx.psu.edu/repos/iuc/snpeff/snpEff/4.3+T.galaxy1", "GEMINI load", "toolshed.g2.bx.psu.edu/repos/iuc/gemini_load/gemini_load/0.20.1+galaxy2", "GEMINI annotate", "toolshed.g2.bx.psu.edu/repos/iuc/gemini_annotate/gemini_annotate/0.20.1+galaxy2", "GEMINI query", "toolshed.g2.bx.psu.edu/repos/iuc/gemini_query/gemini_query/0.20.1+galaxy1", "Join two files", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_easyjoin_tool/1.1.1", "Column arrange by header name", "toolshed.g2.bx.psu.edu/repos/bgruening/column_arrange_by_header/bg_column_arrange_by_header/0.2"]


topic: variant-analysis
tutorial: beaconise_1000hg
difficulty: advanced
time_estimation: 2H
requirements_json: ["type: external", "type: external", "type: internal"]
tags_json: []
content_hash: a67c1d246e56ed631e433c7af5e5d1c1
last_modified: 2026-05-27T09:01:50.014716
zenodo_link: https://zenodo.org/records/10658688
workflows_json: []
title:
Working with Beacon V2: A Comprehensive Guide to Creating, Uploading, and Searching for Variants with Beacons

url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/beaconise_1000hg/tutorial.html

content:
Beacon v2 is a data query protocol and API that allows the researcher to seek information about specific genomic variants of biomedical research and clinical applications from the data providers (Beacon provider) without 
accessing the original or the whole dataset {% cite Rambla2022 %}. The protocol was developed by the [Global Alliance for Genomics and Health (GA4GH)](https://www.ga4gh.org/) in 2021 as an update for the former Beacon v1.
The second version of Beacon comes with additional features that provide users with more detailed information about the queried variants. Unlike the previous version, 
which only returned a Yes or No response indicating the presence of variants, Beacon v2 presents comprehensive results about the variants being searched. {% cite Rueda2022 %}.

The Beacon v2 comprises two main parts, the framework and the models. The framework specifies how the requests and responses should be formatted, while the models determine the organization of the biological data response 
{% cite Rueda2022 %}.

![Beacon Framework and Model](../../images/Beacon_framework_and_model.png "Schematic representation of Beacon v2 API specification from B2RI Documentation")

MongoDB is a versatile, document-oriented NoSQL database that provides flexibility and scalability for modern applications, allowing developers to efficiently manage and query unstructured data. 

MongoDB databases are created with different security levels. This controls accessing the Beacon through the import and query processes. There are three security levels for Beacon. Public, registered and controlled, in our import and query tools, we have classified them into public and authenticated.


| Security Level | Description                                         |
|----------------|-----------------------------------------------------|
| Public         | Beacon can be accessed by any request               |
| Registered     | Only known users can access the data                |
| Controlled     | Only specifically granted users can access the data |

The Beacon data is divided into two parts: the Metadata and Genomic variants. The metadata is initially saved in different formats like Excel, CSV, etc., while the Genomic variations are kept in VCF files.
So, before creating a Beacon database for them, the Beacon providers must prepare the data in JSON, also known as Beacon-friendly format (BFF). This can be done by extracting the required information from those files following the Beacon schemas. 
The Beacon providers can use already developed tools such as the [B2RI tools](https://github.com/EGA-archive/Beacon2-ri-tools) or develop their own tools. 

For this tutorial, we will show the steps to create a Beacon and query it for the [HG00096](https://s3.console.aws.amazon.com/s3/buckets/1000genomes-dragen?region=us-west-2&bucketType=general&prefix=data/dragen-3.5.7b/hg38_altaware_nohla-cnv-anchored/HG00096/&showversions=false) 
biosample structural variants VCF file obtained from the [1000 Genome project](https://s3.console.aws.amazon.com/s3/buckets/1000genomes-dragen?region=us-west-2&bucketType=general&tab=objects) using scripts created by The University of 
Bradford Computational and Data-Driven Science research team.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Create Beacon protocol using MongoDB

The institution that provides the Beacon creates a customized Beacon according to its specific requirements. The Beacon can be made open access or private. For this training, we will create a private access Beacon. 
This Beacon will have administrative user rights to modify it, and an additional account dedicated solely to querying the database.

We will use docker and docker-compose for this step. If you don't have it installed, please follow this [documentation](https://docs.docker.com/engine/install/) to install it


> <hands-on-title>Create Beacon Database on MongoDB</hands-on-title>
> 1. Create a directory in your local environment and name it as `beacon`
>    ```bash
>    mkdir beacon
>    ```
> 2. Change your location to the created directory
>    ```bash
>    cd beacon
>    ```
> 3. Use any text editor you are comfortable with to create a new YAML file and name it `docker-compose.yaml`
>    ```bash
>    nano docker-compose.yaml
>    ```
> 4. Copy the text below into the `docker-compose.yaml` file
>    ```yaml
>    version: '3.6'
>    services:
>    
>      mongo-client:
>        image: mongo:3.6
>        restart: unless-stopped
>        volumes:
>          - ./mongo/db:/data/db
>          - ./mongo-init:/docker-entrypoint-initdb.d
>        ports:
>          - "27017:27017"
>        environment:
>          MONGO_INITDB_ROOT_USERNAME: root
>          MONGO_INITDB_ROOT_PASSWORD: example
>     
>      mongo-express:
>        image: mongo-express
>        restart: unless-stopped
>        environment:
>          - ME_CONFIG_MONGODB_SERVER=mongo-client
>          - ME_CONFIG_MONGODB_PORT=27017
>          - ME_CONFIG_BASICAUTH_USERNAME=root
>          - ME_CONFIG_BASICAUTH_PASSWORD=example
>        ports:
>          - "8081:8081"
>     
>      mongo-init:
>        image: mongo:3.6
>        restart: "no"
>        depends_on:
>          - mongo-client
>        environment:
>          - MONGO_INITDB_DATABASE=admin
>          - MONGO_INITDB_ROOT_USERNAME=root
>          - MONGO_INITDB_ROOT_PASSWORD=example
>        volumes:
>          - ./mongo-init:/docker-entrypoint-initdb.d
>    ```
> 5. Create the path `mongo/db` in your directory using `$mkdir` tool
>    ```bash
>    mkdir mongo
>    mkdir mongo/db
>    ```
> You can change the name of that bath, but you have to change that also from the docker-compose.yaml file. 
> We have everything ready for creating the MongoDB server hosted in the docker container. This will create a Beacon database with admin access. Next, we will add users. 
> 6. Create another directory in your mongo-init directory and name it as `mongo-init`
>    ```bash
>    mkdir mongo-init
>    ```
> 7. Change your location to the created directory
>    ```bash
>    cd mongo-init
>    ```
> 8. Use any text editor you are comfortable with to create a new JS file and name it `create-user.js`
>    ```bash
>    nano create-user.js
>    ```
> 9. Copy the text below into the `create-user.js` file
>    ```js
>    // create_user.js
>    
>    // Connect to the admin database
>    var adminDB = db.getSiblingDB("admin");
>    
>    // Create a new user with read-only access to all databases
>    adminDB.createUser({
>      user: "query_user",
>      pwd: "querypassword",
>      roles: [
>        { role: "read", db: "admin" },
>        { role: "read", db: "Beacon" }, // Adjust this for your needs
>        // Add additional read roles as needed
>      ]
>    });
>    ```
>    This will add a user (user name: `query_user` and password:`querypassword`) account with read-only permission to the Beacon database. This is important to avoid unwanted modifications to the Beacon database. 
>    To know more about MongoDB, please read the [MongoDB documentation](https://www.mongodb.com/docs/).
> 10. Run the command `$docker-compose` in the directory containing the `docker-compose.yaml` file with the specified parameters.
>    ```bash
>    docker-compose up -d
>    ```
> This will create an empty MongoDB server where we can add the Beacon database or any additional databases. 
>
> 11. Check the created docker containers and test if the created docker container is running
>    ```bash
>    docker ps
>    ```
> This will generate a message similar to the text below
>    ```
>    CONTAINER ID   IMAGE           COMMAND                  CREATED       STATUS                         PORTS                                           NAMES
>    96d7886f9cdb   mongo:3.6       "docker-entrypoint.s…"   5 weeks ago   Up 5 weeks                     27017/tcp                                       mongo-init_mongo-init_1
>    1bb520888fbf   mongo:3.6       "docker-entrypoint.s…"   5 weeks ago   Up 5 weeks                     0.0.0.0:27017->27017/tcp, :::27017->27017/tcp   mongo-init_mongo-client_1
>    a148fb3db385   mongo-express   "/sbin/tini -- /dock…"   5 weeks ago   Restarting (1) 9 seconds ago                                                   mongo-init_mongo-express_1
>    ```
> 12. Test docker to see if it is running by using `$docker exec` command
>    ```bash
>    docker exec -it <mongo-client> bash #If the docker image for the Mongo was installed with a different name, Change the name in <mongo-client>
>    ```
> This will take lunsh the docker container. Use `ctrl + d` to quit.
>
{: .hands_on}


> <comment-title>Create B2RI V2 Beacon Protocole</comment-title>
> Clone to the [beacon2-ri-tools-v2](https://github.com/EGA-archive/beacon2-ri-tools-v2) GitHub repository and follow the instructions to build the beacon protocols, and use additional tools needed for preprocessing the data and metadata
{: .comment}


# Data Preparation
Upload the data into Galaxy and prepare it to be imported into the Beacon database. The datasets were obtained from 
[1000genomes-dragen](https://us-west-2.console.aws.amazon.com/s3/buckets/1000genomes-dragen?region=us-west-2&bucketType=general&tab=objects).

| Name | Format | Data size (MB) |
| ---- | ----- | ------------ |
| HG00096.cnv.vcf | vcf | HG00096.cnv.vcf KB |
| igsr-1000-genomes-30x-on-grch38.tsv | tsv | 405.7 KB |

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. For this tutorial, make a new history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import the data files from
>    [Zenodo](https://zenodo.org/records/10658688):
>
>    ```
>    https://zenodo.org/records/10658688/files/HG00096.cnv.vcf
>    https://zenodo.org/records/10658688/files/igsr-1000-genomes-30x-on-grch38.tsv
>    ```
>   This will download the structural variant and the metadata file.
> 
>    In some cases, the same datasets are available in the Galaxy shared data library. Please consult the instructor for further details.
>
>    Alternatively, the datasets can be downloaded from local storage.
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="json/tabular" %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 4. Provide meaningful names for the data (HG00096 for VCF file and Metadata to igsr-1000-genomes-30x-on-grch38.tsv) and tags to facilitate analysis.
>     
>
>    When uploading data from a link, Galaxy names the files after the link address.
>    It might be useful to change or modify the name to something more meaningful.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
>
> 5. To track the data in the history, it is recommended to tag the datasets by attaching a meaningful tag '#'
>    to them. The tagging will automatically be attached to any file generated
>    from the original tagged dataset.
>   *e.g.*, `#genomicVariations` for structural variants VCF file and
>   *e.g.*, `#individuals` metadata tsv file.
>
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
>
{: .hands_on}


## Convert the Genomic Variants VCF file into JSON
This step is to preprocess the genomic variant data and convert it from VCF to JSON.

> <hands-on-title>Convert the Genomic Variants VCF file into JSON</hands-on-title>
> 1. Run {% tool [ CNV VCF2JSON](toolshed.g2.bx.psu.edu/repos/iuc/cnv_vcf2json/cnv_vcf2json/1.1.0+galaxy0.1) %} on the structural variants VCF file
>       - {% icon param-files %} *"CNV VCF file"*: `HG00096` file
>
>
>
> 2. Inspect the *JSON* output produced by the tool
>
>    > <question-title></question-title>
>    >
>    > In the inspected JSON file, what are the key variables and why are they important for genomic analysis?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. **biosampleId**: This variable represents the unique identifier for the biological sample being analyzed.
>    > >    Understanding the biosample allows researchers to trace the origins of the genetic data and contextualize its
>    > >    relevance within a specific study or experiment.
>    > >
>    > > 2. **assemblyId**: The assemblyId denotes the reference genome assembly against which the genetic variants are aligned. 
>    > >    This is crucial for ensuring consistency and accuracy in genomic analyses, as different assemblies may produce varying results.
>    > >
>    > > 3. **variantInternalId**: This identifier specifies the genomic location and type of the variant, providing a standardized format for 
>    > >    referencing genetic variations. It enables researchers to precisely locate and identify specific genetic alterations within the genome.
>    > >
>    > > 4. **variantType**: The variantType indicates the type of genetic variant being described, such as deletion (DEL), insertion (INS), or substitution (SNP). 
>    > >    Understanding the variant type is essential for interpreting its potential impact on gene function and phenotype.
>    > >
>    > > 5. **variantId**: This variable serves as a unique identifier for the variant, often referencing external databases or ontologies. 
>    > >    It facilitates data integration and interoperability across different genomic datasets and analysis platforms.
>    > >
>    > >
>    > > 6. **start** and **end**: These values specify the genomic coordinates of the variant, delineating its precise location within the reference genome. 
>    > >    Knowing the start and end positions is crucial for accurately defining the boundaries of the variant and assessing its potential functional consequences.
>    > >
>    > > 7. **referenceName**: The referenceName indicates the chromosome or genomic contig on which the variant is located. It provides crucial 
>    > >    contextual information for interpreting the genomic coordinates and understanding the genetic context of the variant.
>    > >
>    > >
>    > > 8. **info**: This section contains additional information related to the variant, such as legacy identifiers, copy number counts (cnCount), 
>    > >    and copy number values (cnValue). These supplementary details can provide insights into the structural and functional 
>    > >    implications of the variant within the genome.
>    > >
>    > >
>    > >    Understanding these key variables is essential for conducting effective genomic analyses, as they provide critical information 
>    > >    about the genetic variants under investigation and enable researchers to interpret their biological significance accurately.
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}


## Phenopacket Schema

The Phenopacket Schema is an open standard for exchanging disease and phenotypic data. It helps us understand, diagnose, and treat both common and unusual diseases. By using Phenopackets, physicians, 
biologists, and researchers can build more comprehensive disease models, connecting comprehensive phenotype descriptions with patient, disease, and genetic data. This enables better study of disease and drugs.

We are using the Biosamples phenopacket. A biological material unit from which the substrate molecules (genomic DNA, RNA, proteins, etc.) are extracted for molecular analyses 
(mass spectrometry, array hybridization, sequencing, etc.). Tissue biopsies, single cells from cultures used for single-cell genome sequencing, and protein fractions from gradient centrifugations are a 
few examples. The same Biosample may be referred to by many instances (e.g., technical replicates) or types of research (e.g., both RNA-seq and genomic array experiments).

![v2.0 phenopacket schema](../../images/phenopacket-schema-v2-overview.png "Overview of v2.0 of the schema optained from Phenopacket Schema Documentation")


> <hands-on-title>Create Phenopacket Schema JSON file from Metadata</hands-on-title>
> 1. Run {% tool [ CNV Phenopacket](toolshed.g2.bx.psu.edu/repos/iuc/cnv_phenopacket/cnv_phenopacket/1.0.2+galaxy0) %} on the structural variants VCF file
>       - {% icon param-files %} *"Metadata file"*: `Metadata` file
>
>
>
> 2. Inspect the *JSON* output produced by the tool
>
>    > <question-title></question-title>
>    >
>    > In the inspected JSON file, what are the key variables and why are they important for genomic analysis?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. **id**: The id field serves as a unique identifier for the individual described in the Phenopacket. This 
>    > >    Identifier allows researchers to track and reference specific individuals across different datasets 
>    > >    and analyses.
>    > >
>    > > 2. **individualId**: This variable represents the individual's unique identifier within the dataset or 
>    > >    study. It helps maintain data integrity and consistency by ensuring accurate identification of the 
>    > >    individual across various analyses and datasets.
>    > >
>    > > 3. **sex**: The sex field specifies the biological sex of the individual, providing important demographic
>    > >    information for genomic analyses. Understanding the individual's sex is crucial for interpreting 
>    > >    genetic variations and assessing their potential relevance to sex-specific traits or diseases.
>    > >
>    > > 4. **description**: This field provides additional information about the individual, such as their 
>    > >    participation in specific genomic projects or studies. It offers context for the individual's 
>    > >    genomic data and helps researchers understand the sources and potential biases of the data.
>    > >
>    > > 5. **procedure**: The procedure field contains information about the procedures or protocols used to 
>    > >    generate the genomic data associated with the individual. This includes codes or identifiers that 
>    > >    reference specific methodologies or experimental protocols, ensuring transparency and 
>    > >    reproducibility in genomic analyses.
>    > >
>    > >
>    > > 6. **files**: This section contains details about the genomic data files associated with the individual. It 
>    > >    includes information such as file identifiers and attributes like the data format (htsFormat) and 
>    > >    genome assembly (genomeAssembly). Understanding these file attributes is essential for 
>    > >    accessing and interpreting the individual's genomic data effectively.
>    > >
>    > > 7. **htsFormat**: The htsFormat field specifies the format of the genomic data files, such as Variant 
>    > >    Call Format (VCF). Knowing the data format is crucial for selecting appropriate analysis tools 
>    > >    and pipelines and ensuring compatibility with downstream analysis workflows.
>    > >
>    > > 8. **genomeAssembly**: This variable denotes the reference genome assembly used for aligning and
>    > >    interpreting the individual's genomic data. It ensures consistency and accuracy in genomic 
>    > >    analyses by specifying the genomic reference against which genetic variants are annotated and interpreted.
>    > >
>    > >
>    > >    By understanding these key variables in the Phenopacket schema, researchers can effectively interpret and analyze the genomic profile 
>    > >    of the individual described, facilitating insights into genetic variations and their potential implications for health and disease.
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}


# Import data into Beacon MongoDB

Now that the data are in the Beacon proper format and with creating the Beacon MongoDB server, we are ready to import the data we have into Beacon.

We will use the Beacon2 import tool to import the data into the created Beacon database. 

> <comment-title>Use Credentials to Access Specific Beacon</comment-title>
> 1. Make sure you are logged in to Galaxy.
> 2. Go to **User > Preferences** in the top menu bar.
> 3. To add beacon database credentials, click on **Manage Information** and fill in the Beacon2 Account empty fields `db_auth_source`, `db_user` and `db_password`.
> 4. Make the changes and click the **Save** button at the bottom.
{: .comment}

> <hands-on-title>Import data into Beacon MongoDB</hands-on-title>
> 1. Use {% tool [Beacon2 Import](toolshed.g2.bx.psu.edu/repos/iuc/beacon2_import/beacon2_import/2.1.1+galaxy0) %} to import genomic variant file Beacon database
>   - {% icon param-file %} *"INPUT JSON FILE"*: `Output dataset 'HG00096'` from {% tool [ CNV VCF2JSON](toolshed.g2.bx.psu.edu/repos/iuc/cnv_vcf2json/cnv_vcf2json/1.1.o+galaxy0.1) %}
>   - *"DATABASE HOST"*: `The Hostname/IP of the Beacon database` for example `20.108.51.167`
>   - *"DATABASE PORT"*: `27017`
>   - *"DATABASE"*: `beacon`
>   - *"COLLECTION"*: `genomicVariations`
>
>
>
>   - When you run the tool for the first time, it will create a new database named Beacon and a collection named genomicVariations on MongoDB. It will then upload the data from the HG00096 JSON file into the genomicVariations collection.
>   - If you use the same database name and collection name again, it will append the data to the collection. There is an option to clear all or specific collections before uploading the data, but be careful, as deleting is permanent.
>   - Always remember to back up important data before making any deletions.
>
> 2. Use {% tool [Beacon2 Import](toolshed.g2.bx.psu.edu/repos/iuc/beacon2_import/beacon2_import/2.1.1+galaxy0) %} to import phenopacket file into the created Beacon database
>   - {% icon param-file %} *"INPUT JSON FILE"*: `Output dataset 'Phenopacket'` from {% tool [ CNV Phenopacket](toolshed.g2.bx.psu.edu/repos/iuc/cnv_phenopacket/cnv_phenopacket/1.0.2+galaxy0) %}
>   - *"DATABASE HOST"*: `The Hostname/IP of the Beacon database` for example `20.108.51.167`
>   - *"DATABASE PORT"*: `27017`
>   - *"DATABASE"*: `beacon`
>   - *"COLLECTION"*: `individuals`
>
{: .hands_on}


# Search Beacon MongoDB for CNVs

In the last step, we imported the data into the Beacon. Now, we will query the database to look for the samples that match our query. 

We are looking to see if there is a deletion mutation in the gene **located** in `chromosome 1`, which **starts** at `58278107` and **ends** at `58279217`.


> <hands-on-title>Query the Beacon MongoDB</hands-on-title>
>
> 2. Use {% tool [Beacon2 CNV](toolshed.g2.bx.psu.edu/repos/iuc/beacon2_cnv/beacon2_cnv/2.1.1+galaxy0) %} to perform a range query on Beacon **genomicVariations** collection
>   - *"DATABASE HOST"*: `The Hostname/IP of the Beacon database` for example `20.108.51.167`
>   - *"DATABASE PORT"*: `27017`
>   - *"DATABASE"*: `beacon`
>   - *"COLLECTION"*: `genomicVariations`
>   - *"CHROMOSOME"*: `1`
>   - *"START"*: `243618689`
>   - *"END"*: `243620819`
>   - *"VARIANT STATE ID"*: `EFO:0030068`
> The srarch function will queiry the Beacon database and print out the resutls that matches our quiery specifications. In this case it will print something like this. 
> ```json
>    {'_id': ObjectId('6690160a3a936e8e0a7828e2'),
> 'assemblyId': 'GRCh38',
> 'biosampleId': 'HG00096',
> 'definitions': {'Location': {'chromosome': '1',
>                              'end': 243620819,
>                              'start': 243618689}},
> 'id': 'refvar-6690160a3a936e8e0a7828e2',
> 'info': {'cnCount': 1,
>          'cnValue': 0.422353,
>          'legacyId': 'DRAGEN:LOSS:chr1:243618690-243620819'},
> 'updated': '2024-07-11T17:26:27.265115',
> 'variantInternalId': 'chr1:243618689-243620819:EFO:0030068',
> 'variantState': {'id': 'EFO:0030068', 'label': 'low-level loss'}}
> ```
>
> When sharing a Beacon protocol, it is important to provide users with read-only access to query the Beacon database. Creating read-only users for Beacon-providing institutions helps prevent unwanted data overwrites that can occur by mistake.
> 
>
> > <question-title></question-title>
>    >
>    > What dose variantId "EFO:0030068" means? 
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > EFO:0030069 is a term used to describe the low-level loss. The term was set by the CNV community.
>    > > For more information go to the [CNV annotation formats](https://cnvar.org/resources/CNV-annotation-standards/#cnv-term-use-comparison-in-computational-fileschema-formats)
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

# Conclusion

Now, you have a general knowledge of Beacon and MongoDB and how to create your own Beacon and upload your data. 

You can apply what you learned in this tutorial to create a Beacon query for your institution's genomic variant data.

For more information about how to query Beacon databases, please look into [pymongo documentation](https://pymongo.readthedocs.io/en/stable/tutorial.html).


We hope you find this tutorial helpful!

tools_json:
["CNV VCF2JSON", "toolshed.g2.bx.psu.edu/repos/iuc/cnv_vcf2json/cnv_vcf2json/1.1.0+galaxy0.1", "CNV Phenopacket", "toolshed.g2.bx.psu.edu/repos/iuc/cnv_phenopacket/cnv_phenopacket/1.0.2+galaxy0", "Beacon2 Import", "toolshed.g2.bx.psu.edu/repos/iuc/beacon2_import/beacon2_import/2.1.1+galaxy0", "toolshed.g2.bx.psu.edu/repos/iuc/cnv_vcf2json/cnv_vcf2json/1.1.o+galaxy0.1", "Beacon2 CNV", "toolshed.g2.bx.psu.edu/repos/iuc/beacon2_cnv/beacon2_cnv/2.1.1+galaxy0"]


topic: variant-analysis
tutorial: mapping-by-sequencing
title: Mapping and molecular identification of phenotype-causing mutations
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 858a6515a3121892e289e1b8fd22ee87
last_modified: 2026-05-15T22:28:17.723022
zenodo_link: https://doi.org/10.5281/zenodo.1098033
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/mapping-by-sequencing/tutorial.html

content:
In order to map and identify phenotype-causing mutations efficiently from a
single experiment, modern genetic research aims to combine classical genetic
mapping concepts with the power of next-generation-sequencing.

For example, after having obtained a mutant strain of an organism with a particular phenotype from a forward genetic screen, the classical approach towards
identification of the underlying causative mutation would be to:

1. perform mapping crosses to probe for linkage between the unknown mutation
   and selected markers with known location on the genome, in order to determine
   the approximate genomic region that the mutation resides in, then
2. sequence candidate DNA stretches in this region to identify the precise
   nature of the mutation.

Modern approaches, in contrast, use just one set of genome-wide sequencing data
obtained from mapping-cross progeny and suitable parent strains to
simultaneously:

* discover available marker mutations for linkage analysis,
* map the causative mutation using these markers, and
* identify candidate mutations.

Since it uses sequencing data already at the mapping step, this process is
called **mapping-by-sequencing**.

Since genome-wide sequencing will often reveal a few hundreds to many thousands
of marker variants that should be considered together in the linkage analysis,
mapping-by-sequencing data is typically analyzed with dedicated computational
tools. For Galaxy, the **MiModD** suite of tools offers efficient and flexible
analysis workflows compatible with a variety of mapping-by-sequencing
approaches.

> <details-title>Further reading</details-title>
> The MiModD documentation has its own chapter on supported
> [mapping-by-sequencing schemes](http://mimodd.readthedocs.io/en/latest/nacreousmap.html)
> not covered here.
{: .details}

In this tutorial we are going to use several of these tools to map and identify
a point mutation in an *Arabidopsis thaliana* strain from whole-genome
sequencing data. This will demonstrate one particular kind of linkage analysis
supported by **MiModD** called *Variant Allele Frequency Mapping*.

### Variant Allele Frequency Mapping

![Principal of variant allele frequency mapping](../../images/mbs_scheme.png "Principle of variant allele frequency mapping")

As illustrated in Figure 1, variant allele frequency mapping analyzes marker
variant inheritance patterns in pooled F2 recombinants selected for the
mutation to be mapped.

In the figure we assume a diploid
organism with just two pairs (dark and light grey) of chromosomes.

A mutant line established for any organism will harbor
two types of variants (diamonds):
- variants induced during mutagenesis (orange), among them the causative
  variant responsible for the mutant phenotype (yellow)
- variants (with respect to the species' reference genome) that were
  present in the premutagenesis strain (green open diamonds)

For mapping the causative mutation, additional variants (blue open diamonds)
are introduced through a cross to a mapping strain and F2 progeny is selected
for the mutant phenotype. While each individual in the F2 generation carries
its uniquely recombined variant pattern (3 possible outcomes for the diploid
genome are shown in the figure), this phenotypic selection will work against
mapping strain variants in the vicinity (yellow boxes) of the causative
variant. Hence, when the recombined DNA from phenotypic F2 progeny gets pooled
and sequenced, the causative variant can be found by looking for a region for
which all sequenced reads support variant alleles found in the original mutant
line, rather than mapping strain variant alleles.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Data Preparation

In this tutorial we are going to (re-)identify the recessive causative mutation
in a late-flowering mutant line of *Arabidopsis thaliana*.

> <comment-title></comment-title>
> **MiModD** was developed as a successor of the older and now unmaintained
> *CloudMap* suite of tools, but also uses some key concepts first implemented
> in the command line package *SHOREmap*.
> The data used in this tutorial represent a subsample of the
> [SHOREmap proof-of-principle data](http://bioinfo.mpipz.mpg.de/shoremap/examples.html)
> (specifically of the `OCF2` and `Ler` datasets).
{: .comment}

The mutant line was obtained from a mutagenesis screen on the **Col**umbia
reference background of *A. thaliana*. For mapping-by-sequencing, the line was
outcrossed to the highly polymorphic **L**andsberg ***er**ecta* background as a
source of marker variants. From this outcross, 119 F2 siblings showing the
original late-flowering phenotype were recovered and their genomic DNA was
pooled and subjected to whole-genome sequencing.
Together with whole-genome sequencing data of a pure L*er* background strain,
this outcrossed pool data forms the starting point of our analysis.
The L*er* strain sample, thus, can inform us about the markers that the
outcrossed F2 pool *could* have inherited and exclusion of L*er* markers from
a particular genomic region from the F2 pool provides evidence that the
phenotype causing variant lies close to this region. With respect to
[Figure 1](#figure-1), this means that we will only use "blue diamond" marker
variants introduced through the mapping cross for the linkage analysis and
ignore potentially informative (green and orange) variants found in the
original mutant line.

For your convenience, the sequenced reads from both datasets have already been
mapped (using **Bowtie2**) to the *A. thaliana* Col reference genome (version
TAIR10) so the two datasets take the form of BAM files with the mapping results
(see the tutorial on
[NGS data mapping](../../../sequence-analysis/tutorials/mapping/tutorial.html)
if you do not know what this means).

> <hands-on-title>Data upload and preprocessing</hands-on-title>
>
> > <tip-title></tip-title>
> > This section assumes that you already know
> >
> > - how to upload data to Galaxy via links
> > - edit dataset attributes
> > - do some basic text manipulation on datasets
> >
> > An excellent introduction into these topics can be found in the
> > [From peaks to genes](../../../introduction/tutorials/galaxy-intro-peaks2genes/tutorial.html)
> > tutorial (specifically, in the sections on
> > [Data upload](../../../introduction/tutorials/galaxy-intro-peaks2genes/tutorial.html#data-upload)
> > and
> > [File preparation](../../../introduction/tutorials/galaxy-intro-peaks2genes/tutorial.html#file-preparation)).
> {: .tip}
>
> 1. Import the two BAM datasets representing the:
>    - [L*er* mapping strain](https://zenodo.org/record/1098034/files/Ler_mapping_strain.bam)
>    - [outcrossed F2 pool](https://zenodo.org/record/1098034/files/outcrossed_F2.bam)
> 2. Specify the genome that was used at the reads mapping step
>    1. Click on **Edit attributes** (the pencil icon displayed with the dataset)
>    2. As the **Database/Build** select `Arabidopsis thaliana TAIR10`
>    3. **Save** the edited attributes
> 3. Import the reference genome
>
>    > <comment-title>TAIR10 Version</comment-title>
>    > This step is only necessary if the TAIR10 version of the *A. thaliana*
>    > genome is not available as a built-in genome on your Galaxy server.
>    {: .comment}
>
>    1. Upload the reference genome using the URL:
>       `https://www.arabidopsis.org/download_files/Genes/TAIR10_genome_release/TAIR10_chromosome_files/TAIR10_chr_all.fas`
>    2. Use **Replace parts of text** {% icon tool %} to truncate the title
>       lines in the fasta dataset to just the bare chromosome names. Set:
>       - the **Find pattern** to `^>(\w+).*`
>
>         This will find entire fasta title lines, or more specifically, any
>         lines starting with `>` followed by at least one letter, digit or
>         underscore.
>       - the **Replace with** pattern to `>chr$1`
>
>         The `$1` in it refers to the parenthesized part of the find pattern,
>         *i.e.*, just the substring of letters/digits/underscores directly
>         following the `>`.
>       - **Find-Pattern is a regular expression** to `Yes`
>    3. Make sure the datatype of the resulting dataset is set to `fasta` and
>       give the dataset a nice name (*e.g.*, TAIR10 reference).
>    4. At this point you can delete the originally downloaded dataset.
{: .hands_on}


# Joint Variant Calling and Extraction

A hallmark of all mapping-by-sequencing analyses is that one would like to
follow the fate of variants through genetic crosses. In this tutorial, for
example, we would like to determine, for every marker variant provided by the
L*er* background, at what frequency it got inherited by the outcrossed F2 pool.
For such analyses to work reliably, it is best to call variants jointly for all
relevant samples because this allows the variant caller to give us per-sample
variant call statistics for all sites at which **any** sample is found to have
a non-wt genotype. In our case, this means that the variant caller will report
any variants it finds for the L*er* background sample along with the call
statistics for the outcrossed F2 pool at that same site - even if the variant
allele contributes very little to the pool and would, thus, have been
disregarded in a standard single-sample analysis.

Here, we are using the **MiModD Variant Calling** tool for joint variant
calling for our two samples. This tool outputs variant call statistics across
all sites and all samples in the compact binary BCF format. We can then use
the **MiModD Extract Variant Sites** tool to extract sites, for which at least
one sample appears to have a non-wt genotype, and report them in VCF format.

> <hands-on-title>Variant Calling</hands-on-title>
>
> 1. Use **MiModD Variant Calling** {% icon tool %} to generate combined
>    variant call statistics:
>    - Select the **reference genome** to call variants against.
>
>      If your Galaxy server offers version TAIR10 of the *A. thaliana*
>      reference as a **built-in genome**, you can go ahead and use it.
>      Otherwise, download and preprocess the reference as described under
>      Data Preparation and use it as a **genome from my history**.
>    - As the **Aligned reads input datasets** for the analysis select the two
>      BAM datasets representing the two samples (use the `Control` key on the
>      keyboard to select multiple datasets from the list.
>    - **Execute** the job
> 2. Run **MiModD Extract Variant Sites** {% icon tool %} with
>    - **BCF input file** set to the BCF dataset generated in the previous
>      step.
>
> > <question-title></question-title>
> >
> > 1. Approximately how many variants were extracted?
> > 2. Does this number surprise you?
> >
> > > <solution-title></solution-title>
> > >  1. You can click on the extracted variants dataset in your history to
> > >     get more details about it. You should see that the file has an
> > >     estimated 130,000 lines. Since one variant gets reported per line in VCF format (apart from a number
> > >     of header lines), this roughly
> > >     equals the number of variants in the dataset.
> > >  2. The huge number of variants reflects the fact that the L*er*
> > >     mapping strain is really quite diverged from the Col reference
> > >     background that the mutant was generated in and, thus, offers many
> > >     marker variants for mapping.
> > >
> > >     In total, the L*er* genome harbors >400,000 single nucleotide
> > >     variants compared to the Col reference, but to keep the tutorial
> > >     datasets small we only provided reads mapping to 2 out of 5
> > >     chromosomes. In addition, we provided really low-coverage data
> > >     even for those two chromosomes so our analysis will miss many true
> > >     variants and contain many questionable variant calls that may represent
> > >     false-positives. Regardless, an analysis based on such a large overall number
> > >     of variants is robust enough to provide useful results.
> > {: .solution }
> >
> {: .question}
{: .hands_on}


# Linkage Analysis

The extracted variants dataset obtained in the previous step contains
information about thousands of sites that can be used as markers to map the
causative mutation in the F2 pool. We will now analyze the data with the
**MiModD NacreousMap** tool in **Variant Allele Frequency** mapping mode to
identify these markers and look for linkage between them and the causative
mutation.

> <hands-on-title>Linkage Analysis with one set of parentally contributed markers</hands-on-title>
>
> In the **MiModD NacreousMap** {% icon tool %} interface, set
> - **type of mapping analysis to perform** to `Variant Allele Frequency
>   Mapping` and
> - **data source to use** to `VCF file of variants`.
> - As the **input file with variants to analyze** select your extracted
>   variants VCF dataset obtained in the previous step.
> - As the **mapping sample name** use `outcrossed F2`.
> - Leave the **name of the related parent sample** input field empty because
>   in this two-sample there is no such sample.
> - As the **name of the unrelated parent sample** specify `Ler`.
>    > <comment-title>Sample roles</comment-title>
>    > In `Variant Allele Frequency Mapping` mode the tool requires:
>    > - a **mapping sample**, which corresponds to the recombinant pool, from
>    >   which allele frequencies should be calculated
>    > - either a **related** or an **unrelated parent sample** (or both),
>    >   from which the marker variant sites to use in the linkage analysis
>    >   will be determined. A related parent sample is related (or identical)
>    >   to the original mutant line by descent and provides information about
>    >   (some of) the variants present in that line, while an unrelated parent
>    >   sample provides information about the mapping strain variants
>    >   introduced in the mapping cross. If both a related and an unrelated
>    >   parent sample are available, the tool can also use the marker sets
>    >   from both sides for the linkage analysis.
>    >
>    {: .comment}
> With these settings, we base the linkage analysis on crossed-in variants
> only, *i.e.*, the tool will look for variants, for which the L*er* sample
> appears homozygous mutant, and determine the contribution of each such
> variant allele to the recombinant pool highlighting regions of low
> contribution, *i.e.*, with suspected linkage to the causative variant, in its
> output.
>
> Running this job will generate two datasets, one with linkage tables for each
> chromosome in the *A. thaliana* genome, one with graphical representations
> of the data.
>
> > <question-title></question-title>
> >
> > 1. Have a look at the linkage plots dataset generated by the tool run. In
> >    which interval on which chromosome would you expect the causative
> >    mutation?
> > 2. Are the per-marker scatter plots and the histogram plots in agreement?
> > 3. What about those `No markers found!` messages (you saw them, right?)
> >    that the tool printed?
> >
> > > <solution-title></solution-title>
> > > 1. The plots suggest that the causative variant is likely to be found
> > >    in the region from 18,000,000 to 19,000,000 on chr2.
> > > 2. Look at the y axis labels! The scatter plots show the rates at
> > >    which marker sites appear to recombine with the causative mutation
> > >    in the pool of F2 recombinants, while the histograms show linkage
> > >    evidence. Hence, a dip of the scatter plot regression line towards zero
> > >    corresponds to a peak in the histograms and the two kinds of plots are,
> > >    thus, in very good agreement for this analysis.
> > >
> > >    In general, the histogram plots are easier to digest and often
> > >    provide more precise mapping intervals, and are often all you need for
> > >    mapping a causative variant. The per-marker scatter plots, on the other
> > >    hand, are closer to the raw data and can sometimes reveal additional
> > >    irregularities in the data that may indicate problems with the mapping
> > >    cross or unexpected factors influencing the inheritance pattern.
> > > 3. Since we included only reads mapping to chromosome 1 or 2 in
> > >    the original BAM files, the tool will not be able to detect any markers
> > >    for the other chromosomes.
> > {: .solution }
> >
> {: .question}
{: .hands_on}


# Identifying Candidate Mutations

With the mapping results obtained above we can now try to get a list of
candidates for the causative mutation. Here is what we know about this mutation
so far:

* The F2 recombinant pool should appear homozygous mutant for it.

  This is because the F2 recombinants got selected based on the phenotype of
  the recessive mutation so we can assume that all individuals in the pool were
  homozygous for it.

* The L*er* strain should not carry it.

* It should reside in the genomic region identified in the previous step.


> <hands-on-title>Variant Filtering</hands-on-title>
>
> With the above information, we can run **MiModD VCF Filter** {% icon tool %}
> to obtain a new VCF dataset with only those variants from the originally
> extracted ones that fulfill all three criteria. In the tool interface,
>
> - select as **VCF input file** the extracted variants dataset from your
>   history
> - add a **Sample-specific Filter** and set its
>   - **sample** to `outcrossed F2`
>   - **genotype pattern(s) for the inclusion of variants** to `1/1`
>
>   These settings will make the filter retain only variants for which the F2
>   pool has a `1/1`, *i.e.*, homozygous mutant genotype.
> - add another **Sample-specific Filter** and set its
>   - **sample** to `Ler`
>   - **genotype pattern(s) for the inclusion of variants** to `0/0`
>
>   This filter will retain only variants for which the mapping strain sample
>   has a homozygous wt genotype.
> - add a **Region Filter** and set its
>   - **Chromosome** to `chr2`
>   - **Region Start** to 18000000
>   - **Region End** to 19000000
>
>   to retain only variants falling into the previously determined mapping
>   interval.
>
>    > <comment-title>Sample and chromosome names</comment-title>
>    > This is the second time in this tutorial that we ask you to type sample
>    > or chromosome names exactly as they appear in a dataset, and so you
>    > may have started wondering how you are supposed to keep track of which
>    > names are used where in your own analyses.
>    >
>    > Because this is a valid concern, the MiModD tool suite features the
>    > **MiModD File Information** {% icon tool %} tool that can report the
>    > sample and sequence names stored in almost any dataset that you may
>    > encounter in a mapping-by-sequencing analysis (specifically, the
>    > tool can handle datasets in vcf, bcf, sam, bam and fasta format).
>    >
>    {: .comment}
>
> Running the job yields a new VCF dataset with variants that passed all
> filters.
>
> > <question-title></question-title>
> >
> > 1. How many variants passed the filters?
> > 2. How many variants would you have to consider as candidates if you did
> >    not have any mapping information?
> >
> > > <solution-title></solution-title>
> > > 1. If you used the suggested mapping interval of 1Mb on chr2, you
> > >    should have retained 5 variants. Filtering is a really powerful way to
> > >    reduce the number of variants to consider!
> > > 2. Re-running the job with the region filter removed shows that across
> > >    the two chromosomes, for which the input datasets provided sequencing
> > >    reads, there are 50 variants for which the pool appears homozygous
> > >    mutant and for which the L*er* strain is homozygous wt.
> > {: .solution }
> {: .question}
{: .hands_on}

Not all of the retained variants are equally likely candidates for being the
causative mutation. Since the input datasets consist of reads from whole-genome
sequencing, as opposed to exome sequencing, some of the variants may fall in
intergenic regions or introns where they are unlikely to cause a phenotype.
Other variants may fall in coding regions, but may be silent on the protein
level. Hence, we can prioritize our candidate variants after annotating them
with predicted functional effects.

> <hands-on-title>Variant Annotation</hands-on-title>
>
> To annotate our variants we are going to use **SnpEff** with its
> `Arabidopsis_thaliana` genome database as the source of functional genomic
> annotations, then generate a human-friendly variant report with the
> **MiModD Report Variants** {% icon tool %} tool.
>
> 1. Use **SnpEff Download** {% icon tool %} to download genome database
>    `Arabidopsis_thaliana`.
>
>    > <comment-title>SnpEff versions</comment-title>
>    > SnpEff genome databases can only be used with the specific `major.minor`
>    > version of SnpEff they were built for.
>    > Make sure you are using matching versions (e.g., `4.3` - differences
>    > past the second digit are fine and denote subversions with compatible
>    > database schemes) of **SnpEff Download** {% icon tool %} in this and of
>    > **SnpEff Variant effect and annotation** {% icon tool %} in the next
>    > step.
>    > 
>    > Please also note that before SnpEff version 4.3 the TAIR10 version of
>    > the *A. thaliana* genome database was named `athalianaTair10`.
>    > If your Galaxy server offers only older versions of the SnpEff tools,
>    > you will have to use that name instead of `Arabidopsis_thaliana`.
>    {: .comment}
> 2. In the **SnpEff Variant effect and annotation** {% icon tool %} interface,
>    set:
>    - **Sequence changes (SNPs, MNPs, InDels)** to your filtered VCF dataset
>    - **Genome source** to `Downloaded snpEff database in your history`
>    - **SnpEff4.3 Genome Data** to your just downloaded genome dataset
>    - under **Annotation options** check `Use 'EFF' field compatible with
>      older versions (instead of 'ANN')`
>
>      This last setting is required for compatibility of the resulting dataset
>      with the current version of **MiModD Report Variants** {% icon tool %}.
>
>    Running this job will produce another VCF dataset with the functional
>    annotation data added to its INFO column. Inspecting this dataset is not
>    particularly convenient, but you can turn it into a more user-friendly
>    report by
> 3. using **MiModD Report Variants** {% icon tool %} and setting:
>    - **The VCF input with the variants to be reported** to the annotated
>      variants dataset generated by **SnpEff**
>    - the **Format to use for the report** to `HTML`
>    - **Species** to `A. thaliana`.
>
>      The species information can be used by the tool to enrich the output
>      with species-specific database and genome browser links.
>
>    > <comment-title></comment-title>
>    > Variant reports in html format are meant to be used on relatively small
>    > lists of candidate variants like in this example. If the input VCF
>    > contains > 1000 variants, the relatively large size of the reports and
>    > slow loading in the browser may become annoying. In this case, or if you
>    > experience formatting problems with the generated html in your browser,
>    > it may be better to choose `Tab-separated plain text` as the report
>    > format although the output will then lack those convenient hyperlinks.
>    {: .comment}
>
> > <question-title></question-title>
> >
> > The published confirmed mutations in the mutant line are *chr2:18,774,111
> > C→T* and *chr2:18,808,927 C→T*. The latter was shown through follow-up
> > experiments to be the causative variant.
> >
> > 1. What are the functional effects of these two mutations according to the
> >    report?
> > 2. What biological functions are described for the two genes? How do these
> >    relate to the phenotype of our mutant line?
> >
> > > <solution-title></solution-title>
> > > 1. The C→T transition at 18,774,111 results in a premature
> > >    translational stop for the transcript AT2G45550.1. The C→T transition
> > >    at 18,808,927 affects a splice donor site and, thus, is likely to
> > >    interfere with correct splicing of the transcript AT2G45660.1.
> > > 2. From the linked gene pages at *TAIR* you can learn that the
> > >    gene AT2G45550 encodes a cytochrome P450, while AT2G45660 encodes a
> > >    protein called AGL20/SOC1, which is known to control flowering. Thus, there
> > >    is a direct connection between the late-flowering phenotype of the
> > >    mutant line studied here and the known role of the gene affected by
> > >    chr2:18,808,927 C→T.
> > {: .solution }
> {: .question}
{: .hands_on}

# Conclusion


Mapping-by-sequencing can greatly speed up and facilitate the molecular
identification of mutations recovered from mutagenesis screens. The method
makes extensive use of cross-sample comparison of variants and benefits strongly
from joint variant calling to generate reliable and manageable variant
information. A combination of variant linkage analysis and filtering often
results in very small lists of candidate variants that can then be confirmed
through further experimental work. Importantly, the selection of biological
samples for sequencing determines the meaningful variant comparisons that can
be made in the bioinformatics analysis so it is important to understand the
essence of the analysis method **before** the preparation of any biological
samples.


topic: variant-analysis
tutorial: pox-tiled-amplicon
title: Pox virus genome analysis from tiled-amplicon sequencing data
difficulty: advanced
time_estimation: 4H
requirements_json: []
tags_json: ["virology", "one-health"]
content_hash: 25d02ffdce0a927c6d6d4c0b41aebeaf
last_modified: 2026-05-15T22:28:17.726004
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/pox-tiled-amplicon/tutorial.html

content:
Pox viruses (*Poxviridae*) are a large family of viruses, and members of it have various vertebrate and arthropod species as their natural hosts. The most widely known species in the family are the now extinct variola virus from the genus *orthopoxvirus* as the cause of smallpox, and vaccinia virus, a related, likely horsepox virus, which served as the source for the smallpox vaccine that allowed eradication of that disease.

However, various other pox viruses infect mammalian hosts and cause often severe diseases, and some of them have the ability to jump to humans. Mpox, caused by monkeypox virus, is the most recent and prominent example demonstrating the zoonotic potential of *orthopoxviruses*. The sister genus *capripoxvirus* is a major concern in livestock farming with all its three species - sheeppox virus (SPPV), goatpox virus (GTPV), and lumpy skin disease virus (LSDV) causing severe disease [in sheep and goats](https://www.woah.org/en/disease/sheep-pox-and-goat-pox/) (SPPV, GTPV) and [in cattle](https://www.woah.org/en/disease/lumpy-skin-disease/) (LSDV).

All pox viruses are characterized by a large double-stranded DNA genome (genome sizes of currently sequenced species range from 135 to 360kb) that gets replicated exclusively in the cytoplasm of host cells.

The ends of pox virus genomes consist of inverted terminal repeats (ITRs), i.e. regions of exact reverse-complementarity at the 5'- and 3'-end of the genome. At the very tips of the genome, these regions contain tandem repeats and terminal bases that are cross-linked to the opposite genome strand as first described for vaccinia virus and reviewed in {% cite Wittek1982 %}.

There is considerable length variation in the ITR regions of different species. For example, Capripox virus genomes have ITR lengths of 2.2 - 2.4 kb, monkeypox virus has 6.4 kb long ITRs, and variola virus had the shortest known
ITRs with a length of just 700 bases (reviewed in {% cite Haller2014 %}).

Independent of their size, ITRs pose a problem for the analysis of high-throughput sequencing data of pox viruses. Their repetitive nature can confuse de-novo assembly algorithms, but is, to some extent, also a challenge during reference-based mapping, particularly at the boundary between ITRs and the central region of the genome and with samples less related to the reference.
  
Recently, a tiled-amplicon approach with separate sequencing of half-genomes has been developed for *capripoxviruses* ({% cite Mathijs2022 %}) that makes it possible to assign ITR-derived reads unambiguously to the 5'- or 3'-end of a pox virus genome when combined with a suitable bioinformatics pipeline.
  
This tutorial demonstrates how such data can be analyzed correctly with Galaxy.

> <comment-title>Advanced tutorial</comment-title>
> This tutorial demonstrates a very close to real-world analysis of viral sequencing data with Galaxy[^prior_work].
> As such, it avoids many simplifications that are often made in training material to ease understanding.
> On the technical side, in particular, it makes use of many different Galaxy features and often assumes more than just basic familiarity with them.
> Like in other tutorials, you will find precise instructions for running every step in the analysis, but explanations focus very much on the biology of pox viruses and will often not offer much details on *why* a certain component of Galaxy is used the way it is described.
>
> To get the most out of this tutorial, you should have worked through several other parts of the training material, which are listed under **Requirements** above, in particular:
> - If you are not yet familiar with
>   - some typical ways of getting data into Galaxy,
>   - dataset collections and ways to manipulate them
>   - importing and running workflows,
>
>   you may benefit more from this tutorial if you first follow some of the introductory tutorials on these topics.
>
>   The tutorial also uses the rule-based uploader, which is an advanced feature, and treats it very much like a black box, which somehow lets you structure data exactly the way it is needed for the analysis. If you want to learn how to use this powerful feature for your own purposes, please follow the dedicated tutorials [Rule Based Uploader]({% link topics/galaxy-interface/tutorials/upload-rules/tutorial.md %}) and [Rule Based Uploader Advanced]({% link topics/galaxy-interface/tutorials/upload-rules-advanced/tutorial.md %}).
>
> - If this is your first exposure to sequencing data analysis, our more basic tutorials on quality control and mapping of such data are highly recommended to get you familiar with general concepts, data types, etc.
>
{: .comment}  

The following figure provides an overview of the wet lab data generation process described in {% cite Mathijs2022 %} and of the flow of the analysis that we will perform in this tutorial.

![A schematic drawing of a pox virus genome with a central coding region flanked by inverted terminal repeat regions. Below the genome 23 tiled amplicons are drawn as numbered lines to show their approximate distribution across the genome. Below the amplicons, the main steps in their wet lab processing and of the bioinformatics analysis of their sequences is shown as a flowchart](../../images/pox-virus-tiled-amplicon/pox-tiled-amplicon-wf.png "Tiled-amplicon approach with mapping to half-genomes")

On the wet lab side, the genome of a viral sample gets amplified in the form of 23 overlapping amplicons using a set of pan-genus primers suitable for all species of *capripoxvirus*.

Amplification happens in four separate multiplex PCR reactions: the primers for the odd-numbered amplicons (1-11) from the 5' amplicon pool are used in reaction 1a, the primers for the even-numbered amplicons (2-12) in reaction 1b, the odd-numbered amplicons (13-23) from the 3' amplicon pool in reaction 2a and the even numbered amplicons (12-22) from that pool in reaction 2b.

Note that the 5'-ITR gets amplified as part of amplicon 1, while the 3'-ITR is part of amplicon 23. Amplification of the two regions is specific because the right primer of amplicon 1 and the left primer of amplicon 23 both bind in the "central coding region", i.e. outside the reverse-complementary ITRs.

After amplification, reactions 1a and 1b, and reactions 2a and 2b are combined to form the two sequencing pools, "pool 1" and "pool 2". Note that amplicon 12 becomes part of both tools in the published protocol because overlap between the two sequencing pools is required for assembly-based analysis. For the mapping-based approach shown in this tutorial the overlap is not essential.

The DNA from the two pools is subjected separately to Illumina sequencing. This way, reads obtained from sequencing of "pool 1" are known to be derived from the 5'-half of the genome, while reads obtained from sequencing of "pool 2" are known to originate from the 3'-half.

In the tutorial analysis, we will exploit this information and map the "pool 1" data to a modified reference genome, in which the 3'-half of the sequence has been replaced with Ns. This masking prevents the read mapper from placing ambiguous reads (in particular reads derived from the 5'-ITR region) onto the 3'-half of the genome.
Likewise, we will use a 5'-masked genome for the mapping of the "pool 2" reads, again preventing erroneous mapping of reads that we know must be derived from the 3'-half of the genome.

Once the correct alignments of the reads from the two pools have been recorded, we can safely merge the two results, and use that merged data to construct a consensus sequence for the sample.

Compared to a standard tiled-amplicon approach involving just one sequencing pool, the half-genome sequencing approach is more accurate, especially at the boundaries between the "central coding region" and the ITRs, but the analysis is complicated by the doubled number of inputs, by the need to generate masked reference sequences and by the extra step of merging two read mapping results.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare analysis history and data

Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Prepare the Galaxy history</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md name="pox-tiled-amplicon tutorial" %}
>
{: .hands_on}

## Get reference and sequenced samples data

In this tutorial you are going to work on the sequencing data from two samples of lumpy skin disease virus that have been sequenced as amplified half-genomes in Illumina paired-end mode. This means that we will be dealing with a forward and a reverse read dataset per sample and amplified half-genome, and so will have to download a total of eight sequenced reads datasets for our analysis.

The samples are the first two (`SAMN20238786` and `SAMN20238785`) from BioProject `PRJNA746718` as found through the [European Nucleotide Archive browser](https://www.ebi.ac.uk/ena/browser/view/PRJNA746718).

If you follow the link, you should see that
- for each of the two samples there is data from two sequencing runs (representing the two sequenced half-genome pools of amplicons).

  If it isn't shown by default, enable display of the *Library Name* column by clicking on *Show Column Selection* and selecting the checkbox for *library_name*. The pool identity (`1` or `2`) is provided as the last part of each sequencing run's library name.

- for each pool the data consists of two downloadable `fastq.gz` results files, which represent the forward and reverse reads for that sequencing run.

> <comment-title>Choice of samples</comment-title>
> An in-depth analysis ({% cite Vandenbussche2022 %}) of the samples from BioProject PRJNA746718 has revealed that their genomes are the result of recombination between at least two different parent strains.
> So not only are these samples sequenced following our wet lab protocol of choice, but there is also some prior knowledge about their (rather interesting) biology.
> This gives us the opportunity to evaluate the correctness of our analysis by comparing the results we will generate in this tutorial to the published findings.
>
{: .comment}

In addition, we will require:
- a **reference genome** to map the sequenced reads to

  We will be using the public lumpy skin disease virus reference sequence [NC_003027.1](https://www.ncbi.nlm.nih.gov/nuccore/NC_003027.1/) from the [NCBI RefSeq](https://www.ncbi.nlm.nih.gov/refseq/) database here.

- a **primer scheme file** with information about
  - which amplicons were present in which half-genome sequencing pool
  - where each primer binds with respect to the reference genome
  
  This is a custom file, which you may have to modify, when analysing your own data, depending on the actual primer scheme used during sample preparation, and which we will just copy/paste into Galaxy. The version we are using describes the binding sites of the 46 primers published by ({% cite Mathijs2022 %}) with respect to the NC_003027.1 reference sequence.

> <hands-on-title>Get the data</hands-on-title>
>
> 1. Import all **pool 1** sequenced reads files of the two samples to analyze into your history.
>
>    You could collect the download links and additional metadata from the ENA page for `PRJNA746718`, but for simplicity lets just use this precompiled combined information:
>
>    ```
>    20L81    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/074/SRR15145274/SRR15145274_1.fastq.gz
>    20L81    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/074/SRR15145274/SRR15145274_2.fastq.gz
>    20L70    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/076/SRR15145276/SRR15145276_1.fastq.gz
>    20L70    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/076/SRR15145276/SRR15145276_2.fastq.gz
>    ```
>
>    and upload them as a *List of Pairs* collection.
>
>    1. Copy the links above
>    2. Open the {% tool [Upload](upload1) %} Manager
>    3. In the top row of tabs select **Rule-based**
>    4. Set **Upload data as** to `Collection(s)`
>    5. Paste the copied links into the text field on the right
>    6. Click **Build** to bring up the *Rule Builder* dialog
>    7. Click on the wrench icon {% icon tool %} next to **Rules** in the left panel of the window
>    8. In the text field, erase the prefilled content and copy/paste the following rule definition into it instead:
>
>       ```
>       {
>         "rules": [
>           {
>             "type": "add_column_regex",
>             "target_column": 1,
>             "expression": ".+_(\\d)\\.fastq\\.gz",
>             "group_count": 1,
>             "replacement": null
>           }
>         ],
>         "mapping": [
>           {
>             "type": "url",
>             "columns": [1]
>           },
>           {
>             "type": "list_identifiers",
>             "columns": [0],
>             "editing": false
>           },
>           {
>             "type": "paired_identifier",
>             "columns": [2]
>           }
>         ]
>       }
>       ```
>    9. Click on **Apply**
>    10. In the bottom row of options, change **Type** from "Auto-detect" to `fastqsanger.gz`
>    11. Set the name of the new collection to `pool1 reads`
>    12. Click on **Upload**
>
>    It is ok to continue with the next step even while the upload of the pool 1 reads is still ongoing.
>    Just click on the **Close** button of the information pop-up window if it is still open when you are ready to proceed.
> 2. {% tool [Upload](upload1) %} the **pool2** sequenced reads files of the same samples using the corresponding metadata:
>
>    ```
>    20L81    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/073/SRR15145273/SRR15145273_1.fastq.gz
>    20L81    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/073/SRR15145273/SRR15145273_2.fastq.gz
>    20L70    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/075/SRR15145275/SRR15145275_1.fastq.gz
>    20L70    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/075/SRR15145275/SRR15145275_2.fastq.gz
>    ```
>
>    - Follow the same steps as for the pool 1 reads to get the data into your history.
>    - The rule definition used for the first pool will work unchanged for the second.
>    - Just make sure to choose `pool2 reads` as the name of the new collection.
>
>    As before, it is ok to proceed to the next step while the data upload is still ongoing.
>
> 3. Tag the uploaded collections of **pool1** and **pool2** reads
>
>    For each of the two collections
>
>    1. Click on the collection in the history panel to step into it
>    2. Click on the {% icon galaxy-pencil %} icon next to the collection name
>    3. Click on *"Add Tags"*
>    4. Use `#pool1` and `#pool2` as the tag for the pool1 and the pool2 collection of reads, respectively
>    5. Click on {% icon galaxy-save %} **Save**
>    6. Check that the tag appears below the collection name
>
>    Parts of the analysis in this tutorial will consist of identical steps performed on the data of each pool.
>    To make it easier to distinguish which analysis step has been conducted with which pool, we will be using name tags.
>    These are tags starting with `#`, and they will automatically propagate to new datasets and collections derived from the tagged collections.
>
> 4. Get the `NC_003027.1` LSDV reference.
>
>    A convenient public download link for this sequence is best obtained from the ENA again, where the sequence is known under its [INSDC](https://www.insdc.org/) alias [AF325528.1](https://www.ebi.ac.uk/ena/browser/view/AF325528.1):
>
>    ```
>    https://www.ebi.ac.uk/ena/browser/api/fasta/AF325528.1?download=true
>    ```
>
>    1. {% tool [Upload](upload1) %} the reference to your history via the link above and make sure the dataset format is set to `fasta`.
>
>       {% snippet faqs/galaxy/datasets_import_via_link.md format="fasta" %}
>    2. {% tool [Replace Text in entire line](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2) in entire line %} to simplify the reference sequence name
>       - {% icon param-file %} *"File to process"*: the uploaded reference sequence from the ENA
>       - In {% icon param-repeat %} *"1. Replacement"*:
>         - *"Find pattern"*: `^>.+`
>         - *"Replace with"*: `>AF325528.1`
>    3. When the Replace Text tool run is finished, **rename** the output dataset
>
>       {% snippet faqs/galaxy/datasets_rename.md name="LSDV reference" format="fasta" %}
>
> 5. {% tool [Upload](upload1) %} the primer scheme information in [BED](http://www.genome.ucsc.edu/FAQ/FAQformat.html#format1) format
>
>    ```
>    NC_003027.1    235       264       CaPV-V1_1_LEFT      pool1a    +
>    NC_003027.1    6789      6819      CaPV-V1_2_LEFT      pool1b    +
>    NC_003027.1    7771      7801      CaPV-V1_1_RIGHT     pool1a    -
>    NC_003027.1    13515     13543     CaPV-V1_3_LEFT      pool1a    +
>    NC_003027.1    14451     14482     CaPV-V1_2_RIGHT     pool1b    -
>    NC_003027.1    20192     20219     CaPV-V1_4_LEFT      pool1b    +
>    NC_003027.1    21201     21229     CaPV-V1_3_RIGHT     pool1a    -
>    NC_003027.1    26787     26814     CaPV-V1_5_LEFT      pool1a    +
>    NC_003027.1    27786     27814     CaPV-V1_4_RIGHT     pool1b    -
>    NC_003027.1    33277     33305     CaPV-V1_6_LEFT      pool1b    +
>    NC_003027.1    34363     34395     CaPV-V1_5_RIGHT     pool1a    -
>    NC_003027.1    39894     39921     CaPV-V1_7_LEFT      pool1a    +
>    NC_003027.1    40867     40897     CaPV-V1_6_RIGHT     pool1b    -
>    NC_003027.1    46393     46420     CaPV-V1_8_LEFT      pool1b    +
>    NC_003027.1    47419     47453     CaPV-V1_7_RIGHT     pool1a    -
>    NC_003027.1    52864     52890     CaPV-V1_9_LEFT      pool1a    +
>    NC_003027.1    54039     54069     CaPV-V1_8_RIGHT     pool1b    -
>    NC_003027.1    59439     59472     CaPV-V1_10_LEFT     pool1b    +
>    NC_003027.1    60530     60559     CaPV-V1_9_RIGHT     pool1a    -
>    NC_003027.1    66008     66041     CaPV-V1_11_LEFT     pool1a    +
>    NC_003027.1    66960     66989     CaPV-V1_10_RIGHT    pool1b    -
>    NC_003027.1    72534     72560     CaPV-V1_12_LEFT     pool1b    +
>    NC_003027.1    72534     72560     CaPV-V1_12_LEFT     pool2b    +
>    NC_003027.1    73568     73601     CaPV-V1_11_RIGHT    pool1a    -
>    NC_003027.1    79081     79111     CaPV-V1_13_LEFT     pool2a    +
>    NC_003027.1    80176     80202     CaPV-V1_12_RIGHT    pool1b    -
>    NC_003027.1    80176     80202     CaPV-V1_12_RIGHT    pool2b    -
>    NC_003027.1    85743     85777     CaPV-V1_14_LEFT     pool2b    +
>    NC_003027.1    86753     86781     CaPV-V1_13_RIGHT    pool2a    -
>    NC_003027.1    92208     92232     CaPV-V1_15_LEFT     pool2a    +
>    NC_003027.1    93323     93351     CaPV-V1_14_RIGHT    pool2b    -
>    NC_003027.1    98974     99002     CaPV-V1_16_LEFT     pool2b    +
>    NC_003027.1    99970     100000    CaPV-V1_15_RIGHT    pool2a    -
>    NC_003027.1    105634    105662    CaPV-V1_17_LEFT     pool2a    +
>    NC_003027.1    106584    106612    CaPV-V1_16_RIGHT    pool2b    -
>    NC_003027.1    112082    112108    CaPV-V1_18_LEFT     pool2b    +
>    NC_003027.1    113111    113144    CaPV-V1_17_RIGHT    pool2a    -
>    NC_003027.1    118617    118649    CaPV-V1a_19_LEFT    pool2a    +
>    NC_003027.1    119649    119675    CaPV-V1_18_RIGHT    pool2b    -
>    NC_003027.1    125361    125390    CaPV-V1_20_LEFT     pool2b    +
>    NC_003027.1    126297    126328    CaPV-V1_19_RIGHT    pool2a    -
>    NC_003027.1    131713    131742    CaPV-V1_21_LEFT     pool2a    +
>    NC_003027.1    132961    132988    CaPV-V1_20_RIGHT    pool2b    -
>    NC_003027.1    138437    138467    CaPV-V1_22_LEFT     pool2b    +
>    NC_003027.1    139333    139365    CaPV-V1_21_RIGHT    pool2a    -
>    NC_003027.1    143117    143145    CaPV-V1_23_LEFT     pool2a    +
>    NC_003027.1    146188    146219    CaPV-V1_22_RIGHT    pool2b    -
>    NC_003027.1    150505    150538    CaPV-V1_23_RIGHT    pool2a    -
>    ```
>
>    Since BED is a tabular format, but the above display uses spaces to separate columns, please make sure, you have **Convert spaces to tabs** checked when creating the dataset from the copied content!
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md name="LSDV primer scheme" format="bed" convertspaces="true" %}
>
{: .hands_on}

At this point you should have these four items in your history:

1. The collection of pool 1 reads (representing the 5'-half of the genome)
   - This should be a list of two elements (representing the two samples we are going to analyze).
   - Each of these elements should itself be a paired collection of a *forward* and a *reverse* reads dataset in `fastqsanger.gz` format.
   - A `#pool1` name tag should be attached to the collection in the history.

2. The collection of pool 2 reads (representing the 3'-half of the genome)
   - This should be structured exactly as the first collection.
   - Also the names of the elements in both collections should be identical.
   - A `#pool2` name tag should be attached to the collection in the history.

3. The LSDV reference as a single *fasta* dataset
4. The primer scheme as a single *bed* dataset

If this is the case for you, you are ready to start the actual data analysis.

# Generating masked reference genome versions for mapping

As illustrated in Figure 1, we would like to map the "pool 1" reads of each sample to a version of the LSDV reference, in which the 3'-half of the genome (starting after the right end of amplicon 12, the 3'-most amplicon still contained in pool 1) is masked (i.e., bases are replaced by Ns). This ensures that the read mapper will only consider the 5'-half of the genome when placing reads from pool 1.
Likewise, for the "pool2" read mapping, we would like to use a reference version, in which the 5'-half that is not covered by the second amplicon pool is masked out.

To build these two masked reference versions we need to extract the information about amplicon pool boundaries from the primer scheme BED dataset and use it twice on the LSDV reference to replace the appropriate ranges of its bases with Ns.

As this is a complex task that would take a considerable amount of time to perform step-by-step via manual tool runs, we are providing a pre-built workflow for you that you only need to import and run on the input data in your history.

> <hands-on-title>Import and run the reference masking workflow</hands-on-title>
>
> 1. Go to the training material [page for this workflow](workflows/) and use one of the options listed there to import the workflow into your account on your Galaxy server.
> 2. Back on the list of workflows page, click the *Run workflow* icon {% icon workflow-run %} to the right of the newly imported workflow.
> 3. On the next page, select
>    - *"Primer Scheme (BED file with Pool identifiers)"*: the uploaded ``LSDV primer scheme`` BED dataset
>    - *"Reference FASTA"*: the ``LSDV reference`` fasta dataset generated with **Replace Text**
>
>    Running the workflow will produce two key output datasets:
>    - a *Masked reference for mapping of pool1*
>    - a *Masked reference for mapping of pool2*
>
> 4. Check whether the workflow produced correctly masked half-genomes
>
>    You can {% icon galaxy-eye %} inspect the two masked reference datasets by eye, which quickly reveals that one sequence has its beginning N-masked, while for the other it is the end, but it is hard to find out whether the regions are exactly right this way. Luckily, we can use a tool to determine the exact extent of Ns for each dataset:
>    1. {% tool [Fasta regular expression finder](toolshed.g2.bx.psu.edu/repos/mbernt/fasta_regex_finder/fasta_regex_finder/0.1.0) %}
>       - {% icon param-files %} *"Input"*: select the two masked reference datasets; output of the masking workflow run
>       - *"Regular expression"*: `N+`
>
>         This looks for stretches of one or more Ns.
>       - In *"Specify advanced parameters"*:
>         - *"Do not search the reverse complement"*: `Yes`
>
>           We are only interested in the forward strand (and an N is an N on both strands anyway) so we can save some compute by turning on this option.
>         - *"Maximum length of the match to report"*: `1`
>
>           It would be boring to have the matched stretches of Ns written out again.
>
>    > <question-title></question-title>
>    >
>    > Inspect the two result datasets created by the tool and compare the regions of Ns reported in them to the primer scheme.
>    > Did the correct regions get masked?
>    >
>    > Tip: Galaxy's {% icon galaxy-scratchbook %} Window Manager, which you can enable (and disable again) from the menu bar can be very helpful for comparing multiple datasets.
>    >
>    > > <solution-title>Answer</solution-title>
>    > >
>    > > The masked reference for mapping of pool1 has Ns from 80,202 - 150,773.
>    > > The 3'-most primer contained in pool1, CaPV-V1_12_RIGHT, has its binding site listed as 80,176 - 80,202.
>    > > Bed format specifies zero-based half-open intervals, i.e. the first base of a sequence has position 0, and the end position of any interval is the first base *not* included in it!
>    > > So the 3'-most base covered by the primer is base number 80,201 (when counting from 0) and the reference is masked correctly from 80,202 onwards.
>    > >
>    > > The masked reference for mapping of pool2 has Ns from 0 - 72,534.
>    > > The 5'-most primer of pool2 is CaPV-V1_12_LEFT and the primer scheme lists the start of its binding site as 72,534.
>    > > Again, the way bed format is specified means that position 72,534 is the first base *not* masked in the reference for pool2 mapping, and that is correct.
>    > > Also note that amplicon 12 is contained in both pools and the masking takes this fact into account. 
>    > >
>    > > Feel free to open the workflow in the workflow editor and inspect the steps it took to work out the masking, but it's also ok to treat this part as a black box and move on. There is still a lot of analysis that is waiting to be done.
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# Quality control of the sequencing data of all samples

The next step is to check the quality of the sequencing data of both samples and to eliminate, if necessary, poor-quality reads or parts of them.

The tool **fastp** lets us perform these tasks and obtain a nice quality report for our reads before and after processing in one go, but many other options exist to perform sequenced reads quality control and trimming/filtering in Galaxy, and the dedicated tutorial on [quality control]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}) introduces more of them.

> <hands-on-title>QC and read trimming/filtering with fastp</hands-on-title>
>
> 1. {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.23.2+galaxy0) %} on the collection of **pool1** reads
>    - *"Single-end or paired reads"*: `Paired Collection`
>    - *"Select paired collection(s)"*: the uploaded collection of **pool1** reads
>    - In *"Filter Options"*
>      - In *"Length filtering options"*
>        - *"Length required"*: `30`
>
>    > <details-title>fastp default settings</details-title>
>    >
>    > fastp comes with basic default settings for some of its many parameters.
>    > In particular, with no explicit parameter specifications, the tool will still remove reads that
>    > - have base qualities < 15 for more than 40% of their bases
>    > - have more than 5 Ns in their sequence
>    > - have a length < 15 bases (after removing poly-G tails from Illumina reads)
>    >
>    >   Because our input data has far longer MiSeq-generated reads, we can increase this length threshold to 30 bases as done above.
>    >
>    > These are very relaxed criteria that will only remove the least usable reads, but still assure minimal
>    > quality standards that are good enough for many (including training) purposes.
>    {: .details}
>
>    Running the tool will result in two outputs. One is a new paired collection with the processed reads, the other one is a report of initial quality, the processing actions performed and their effect on key quality metrics.
>
> 2. {% icon galaxy-eye %} **Inspect** the reports for both samples and try to answer the following questions:
>
>    > <question-title>Questions</question-title>
>    >
>    > 1. After filtering, which of the two samples has more reads retained?
>    > 2. Which sample had more reads filtered out?
>    > 
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. Sample 20L81 has more reads retained (947,000) than 20L70 (856,000).
>    > > 2. Sample 20L81 also had more reads filtered out.
>    > >    "Only" 93.2% of its original reads got retained compared to 99.1% for 20L70.
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. Now repeat the steps above for the collection of pool 2 reads.
>
>    {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.23.2+galaxy0) %} on the collection of **pool2** reads
>    - *"Single-end or paired reads"*: `Paired Collection`
>    - *"Select paired collection(s)"*: the uploaded collection of **pool2** reads
>    - In *"Filter Options"*
>      - In *"Length filtering options"*
>        - *"Length required"*: `30`
>
>    > <question-title>Questions</question-title>
>    > 1. Are the results for the pool 2 sequences similar to those of pool 1?
>    > 2. What's your overall conclusion regarding the quality of the sequences and the effect of filtering?
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. Results are rather similar although for pool2 the metrics for the two samples are even more similar than for poo1, and it is now 20L70, which has a bit more reads retained after filtering.
>    > > 2. The input sequences are of rather good quality and the relaxed filtering criteria we have configured had little impact. This is not a bad thing though and we can proceed with our analysis.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# Mapping to the masked reference genomes

We are now ready to map each pool of each sample to the corresponding masked reference. We will use the tool BWA-MEM with essentially default settings for this purpose. As shown in figure 1, however, we want to merge the mapping results afterwards, but still be able to tell mapped reads from pool 1 and pool 2 apart. To this end, we need to tag reads with `pool1` and `pool2` read group identifiers, respectively, at the mapping step. Just because a sample has pool1 and pool2 reads it is still just one sample though, and we are going to express that fact through a common sample name (which we let BWA-MEM autodetect as the name of the collection elements).

> <hands-on-title>Read mapping and quality control</hands-on-title>
>
> 1. {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2) %} for **pool1** data
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>      - {% icon param-file %} *"Use the following dataset as the reference sequence"*: Masked reference for mapping of **pool1**
>    - *"Single or Paired-end reads"*: `Paired Collection`
>    - *"Select a paired collection"*: the `#pool1` collection of trimmed and filtered reads; output of **fastp** run on the pool1 data
>    - *"Set read groups information?"*: `Set read groups (SAM/BAM specification)`
>
>      for *Read group identifier*:
>      - *"Auto-assign"*: `No`
>      - *"Read group identifier"*: `pool1`
>
>      for *Read group sample name*:
>      - *"Auto-assign"*: `Yes`
>
>        This interprets names of collection elements as sample names.
>
> 2. {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2) %} for **pool2** data
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>      - {% icon param-file %} *"Use the following dataset as the reference sequence"*: Masked reference for mapping of **pool2**
>    - *"Single or Paired-end reads"*: `Paired Collection`
>    - *"Select a paired collection"*: the `#pool2` collection of trimmed and filtered reads; output of **fastp** run on the pool2 data
>    - *"Set read groups information?"*: `Set read groups (SAM/BAM specification)`
>
>      for *Read group identifier*:
>      - *"Auto-assign"*: `No`
>      - *"Read group identifier"*: `pool2`
>
>      for *Read group sample name*:
>      - *"Auto-assign"*: `Yes`
>
{: .hands_on}

# Merging the mapped reads of both pools

We have mapped both pool 1 and pool 2 reads to masked versions of the same reference sequence. This means that any mapping in any of the separate BWA-MEM outputs also describes a valid mapping to the complete, unmasked reference so all we need to do is merge the pool 1 and pool 2 mapped reads datasets of each sample into one to obtain complete mapping results for the entire genome of the sample.

A tool that can perform this task for us is **samtools merge**. However, if we gave this tool our collection of pool1 reads with one BAM dataset per sample and the collection of pool2 reads with again one BAM dataset per sample, you would be very disappointed to see that it would merge all datasets into one, i.e. it would not only combine pools, but also sample data, which is not what we want.

Why does **samtools merge** in Galaxy behave like this? Well, the tool is written so that it knows how to merge multiple datasets *and* it knows how to merge the contents of simple collections - and it's just too eager to apply that knowledge. What we can do, however, is try and pass it something it does *not* know how to handle in a single run, and that's a nested collection. So instead of providing the input as two separate collections (one for pool1 and one for pool2 data) that each hold two elements (representing data from one sample), we want to provide a collection of two samples, each one consisting of a collection of pool 1 and pool 2 reads, in turn.
When we do this, Galaxy will recognize that the tool cannot handle this, but that it would know how to handle each outer element (sample) separately (as a simple collection). In this situation, Galaxy will automatically schedule two parallel excutions of the tool for us, each one receiving one element of the outer collection as input.

We can do the necessary restructuring in two steps, then pass the result to **samtools merge**.

> <hands-on-title>Reorganize the data and merge across pools within samples</hands-on-title>
>
> 1. {% tool [Zip collections](__ZIP_COLLECTION__) %} with the following parameters
>    - {% icon param-collection %} *"Input 1"*: mapped reads `#pool1`; output of **BWA-MEM** run on the pool1 data
>    - {% icon param-collection %} *"Input 2"*: mapped reads `#pool2`; output of **BWA-MEM** run on the pool2 data
>
>    This combines the two separate collections into one nested structure, but, unfortunately, that structure is not organized the way we want it because **Zip collections** creates lists of pairs, i.e. list collections that hold paired collections inside. Now we still need to turn the list of pairs into a list of lists.
>
> 2. {% tool [Apply rules](__APPLY_RULES__) %} with the following parameters
>    - *"Input Collection"*: zipped collection of mapped reads from both pools
>    - Under *"Rules"*
>      - {% icon galaxy-wf-edit %} **Edit**
>    - In the *"Build Rules for Applying to Existing Collection"* window
>      1. Click {% icon tool %} next to **Rules**
>      2. Erase the prefilled rules description
>      3. Copy the rules below and paste them into the empty rules description area
>
>         ```
>         {
>           "rules": [
>             {
>               "type": "add_column_metadata",
>               "value": "identifier0"
>             },
>             {
>               "type": "add_column_metadata",
>               "value": "identifier1"
>             }
>           ],
>           "mapping": [
>             {
>               "type": "list_identifiers",
>               "columns": [
>                 0,
>                 1
>               ],
>               "editing": false
>             }
>           ]
>         }
>         ```
>      4. Click **Apply**
>      5. Click **Save**, then run the tool
>
>    This will reorganize the data once more, this time into the desired structure, which will give us the correct merge result.
>
> 3. {% tool [Samtools merge](toolshed.g2.bx.psu.edu/repos/iuc/samtools_merge/samtools_merge/1.15.1+galaxy0) %}
>    - {% icon param-collection %} *"Alignments in BAM format"*: the reorganized collection; output of **Apply rules**
>    - *"Make @RG headers unique"*: `No`
>    - *"Make @PG headers unique"*: `Yes`
>
> 4. {% icon galaxy-eye %} Now lets confirm that things have worked as expected:
>
>    > <question-title>Questions</question-title>
>    >
>    > 1. What are the datasets inside the **Samtools merge** collection?
>    > 2. {% icon galaxy-eye %} Display each of the datasets. What information about read groups and samples is stored in each dataset?
>    > 
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. The collection consists of two BAM datasets, one for sample 20L81 and one for 20L70.
>    > > 2. The header of the 20L81 dataset lists
>    > >
>    > >    ```
>    > >    @RG ID:pool1 SM:20L81 PL:ILLUMINA
>    > >    @RG ID:pool2 SM:20L81 PL:ILLUMINA
>    > >    ```
>    > >
>    > >    , which declares that the mapped reads in this file come from two read groups with the IDs pool1 and pool2 that belong to the same sample (SM), 20L81, sequenced on the Illumina platform.
>    > >
>    > >    The 20L70 dataset contains analogous lines in its header declaring the two read groups of the other sample.
>    > >
>    > >    When scrolling to the very right in the body of the datasets, you can see that a RG tag of the form RG:ID (where ID is one of the alternatives declared in the header) is attached to each mapped read line (because the dataset is sorted by mapped coordinates and pool2 reads should have been mapped only to the 3'-half of the genome, you will have to scroll down rather far in the dataset to find a RG tag with a pool2 ID value).
>    > >
>    > >    All this read group information in the dataset headers and on the mapped reads has been put there by BWA-MEM and Samtools merge has, apparently, merged the reads and the read group information correctly!
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


# Mapped reads filtering, trimming, and QC

So now that we have one dataset of mapped reads per sample, we will do some cleaning and quality control on the mapped data to be sure we pass only reliable mapping data into consensus calling.

We will first use **samtools view** to discard any alignments that we are not fully confident in.

Next, we will run **ivar trim** on the retained reads. This tool can detect primer sequences at the ends of the mapped reads that represent the tiled amplicon primers used during preparation of the samples for sequencing, and will mask them so that they will not be considered during generation of the consensus sequence.

> <details-title>Primer trimming</details-title>
>
> The primers used for the tiled-amplicon generation will get incorporated physically at the start and end of each amplicon. These ends, therefor, do *not* represent amplified sequence from the sample genome, but merely reflect the primer sequences. Consequently, the amplicon ends will never capture any mutations in the genome region they cover, and we cannot meaningfully use them for building a consensus sequence for the sample.
>
> Luckily, since we are dealing with *tiled* (i.e., overlapping) amplicons, each amplicon end should be overlapped by the non-primer part of another amplicon, and we can use these sequences to obtain information about the sequenced sample at primer binding sites. By masking parts of reads that are derived from incorporated primers, we make sure these "pseudo"-sequences are not interfering with mutation detection based on truly informative reads, and that we can faithfully reconstruct the genome of the sample.
>
{: .details}

Finally, we will use **Qualimap BamQC** to generate a quality report on the trimmed and filtered mapped reads, which we'll inspect to judge whether read mapping worked well enough for consensus generation purposes.

> <hands-on-title>Mapped reads filtering and trimming and quality control for mappings</hands-on-title>
>
> 1. {% tool [Samtools view](toolshed.g2.bx.psu.edu/repos/iuc/samtools_view/samtools_view/1.15.1+galaxy0) %} for filtering the mapped reads
>    - {% icon param-collection %} *"SAM/BAM/CRAM data set"*: collection of merged mapped reads; output of **Samtools merge**
>    - *"What would you like to look at?"*: `A filtered/subsampled selection of reads`
>    - In *"Configure filters"*:
>      - *"Filter by quality"*: `20`
>      - *"Require that these flags are set"*: `Read is paired`
>      - *"Exclude reads with any of the following flags set"*: `Read is unmapped` and `Mate is unmapped` and `Alignment of the read is not primary` and `Alignment is supplementary`
>
> 2. {% tool [ivar trim](toolshed.g2.bx.psu.edu/repos/iuc/ivar_trim/ivar_trim/1.4.2+galaxy0) %} for trimming of amplicon primers from mapped reads
>    - {% icon param-collection %} *"BAM file"*: collection of filtered mapped reads BAM dataset; output of **Samtools view**
>    - *"Source of primer information"*: `History`
>      - *"BED file with primer sequences and positions"*: the uploaded `LSDV primer scheme` BED dataset
>    - *"Filter reads based on amplicon info"*: `Yes, drop reads that extend beyond amplicon boundaries`
>    - *"Include reads not ending in any primer binding sites?"*: `Yes`
>    - *"Require a minimum length for reads to retain them after any trimming?"*: `Yes, and provide a custom threshold`
>      - *"Minimum trimmed length threshold"*: `30
>
> 3. {% tool [QualiMap BamQC](toolshed.g2.bx.psu.edu/repos/iuc/qualimap_bamqc/qualimap_bamqc/2.2.2d+galaxy3) %} for mapped reads quality control
>    - {% icon param-collection %} *"Mapped reads input dataset"*: filtered and trimmed mapped reads BAM dataset; output of **ivar trim**
>    - *"Skip duplicate reads"*: `Deselect all`
>    - In *"Settings affecting specific plots"*:
>      - *"Number of bins to use in across-reference plots"*: `200`
>
> 4. {% icon galaxy-eye %} Study the report generated with QualiMap
>
>    > <question-title>Questions</question-title>
>    >
>    > 1. What is the mean coverage for each of the samples across the entire genome?
>    > 2. Which sample might be a bit more problematic coverage-wise?
>    > 3. What can you observe for the "Mapping Quality Across Reference" plots for both samples?
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. Mean coverage for sample 20L81 is 2,073, for 20L70 it is 2,155.
>    > > 2. Despite its slightly higher mean coverage, sample 20L70 has the slightly worse coverage *distribution*.
>    > >
>    > >    The "Coverage across reference" plot shows higher variability of coverage for this sample.
>    > >    While it peaks at higher values, it also drops below a coverage of 500 more clearly and in more regions across the genome than 20L81, and even drops to potentially problematically low values in a short stretch of sequence just after the middle of the genome.
>    > >    The "Coverage Histogram" plot also reveals that the coverage distribution for 20L70 is a bit skewed towards lower values than that for 20L81.
>    > >
>    > >    All these effects are not very dramatic and should not prevent us from continuing with the analysis of both samples. We should watch out for a potentially incomplete consensus sequence in that low-coverage stretch of the 20L70 genome though.
>    > > 3. Mapping quality is almost constant at or very near 60 (which happens to be the maximum mapping quality value emitted by BWA-MEM) across the entire genome for both samples. Interestingly, there is a tiny dip in mapping quality around ~ 117,000 - 118,000 on the genome, which seems reproducible across the two samples.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# Consensus sequence construction

And with that we are ready for consensus sequence generation!

To accept any base suggested by the mapped sequenced reads as the consensus base for the corresponding genome position, we ask for the following requirements to be fulfilled:

- at least 20 sequenced reads have to provide information about the base in question
- at a minimum, 70% of these reads have to agree on the base at this position (80% for indels)

To avoid getting misled too much by sequencing errors, we are also going to ignore bases with a base calling quality less than 20 in the above counts (i.e., we are going to base our decisions only on bases in sequenced reads that the basecaller of the sequencer was reasonably sure about.

Now what if we cannot obtain a consensus base for a position with the above criteria? In such cases of uncertainty we want to insert an N (i.e. an unknown base) to express that we either did not have enough information about the position or that this information was ambiguous.

{% icon galaxy-info %} All of the above limits for consensus base calling are arbitrary to some degree, and depend somewhat on the quality of the sequencing data. With very high overall coverage, for example, it is possible to increase the coverage threshold, but if you increase that threshold too much, you may end up with a consensus sequence consisting mostly of Ns.

> <hands-on-title>Consensus building</hands-on-title>
>
> 1. {% tool [ivar consensus](toolshed.g2.bx.psu.edu/repos/iuc/ivar_consensus/ivar_consensus/1.4.2+galaxy0) %}
>    - {% icon param-collection %} *"BAM file"*: collection of trimmed mapped reads BAM dataset; output of **ivar trim**
>    - *"Minimum frequency threshold"*: `0.7`
>    - *"Minimum indel frequency threshold"*: `0.8`
>    - *"Minimum depth to call consensus"*: `20`
>
> 2. {% icon galaxy-eye %} Inspect the consensus sequences generated for the two samples
>
>    > <question-title></question-title>
>    >
>    > Does everything look ok? What about those leading and trailing Ns in the sequences?
>    >
>    > > <solution-title>Answer</solution-title>
>    > >
>    > > Certainly both sequences start and end with stretches of Ns.
>    > >
>    > > Since the sequencing data is ampliconic in nature, and the primer scheme lists the first and the last primer as binding close to but not exactly at the end of the genome, it is expected that we cannot resolve the first and last couple of hundreds bases.
>    > >
>    > > However, to determine whether those terminal stretches of Ns are exactly as long as we expect them to be would be a tedious task to perform manually. Likewise, it is not trivial to assert that there are no additional Ns somewhere in these long sequences.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# Quality check on results and interpretation

## Composition of the consensus sequences

So we have consensus sequences for our samples. Are there better ways for checking their quality than just looking through them by eye?

Luckily, there is a tool to help us, and we have even used it before to check the masking of the reference genome!

> <hands-on-title>Check consensus sequences for unresolved bases</hands-on-title>
>
> 1. {% tool [Fasta regular expression finder](toolshed.g2.bx.psu.edu/repos/mbernt/fasta_regex_finder/fasta_regex_finder/0.1.0) %}
>    - {% icon param-collection %} *"BAM file"*: collection of consensus sequences; output of **ivar consensus**
>    - *"Regular expression"*: `[^ATGC]+`
>
>      This expression matches stretches of sequences that do *not* (`^`) contain any of the standard bases (any of the characters in `[]`). The expression is case-sensitive, but ivar consensus only uses uppercase in sequences.
>    - In *"Specify advanced parameters"*:
>      - *"Do not search the reverse complement"*: `Yes`
>      - *"Maximum length of the match to report"*: `1`
>
> 2. {% icon galaxy-eye %} Inspect the output of the tool for both samples and compare it to the primer scheme
>
>    > <question-title></question-title>
>    >
>    > Has consensus building worked well?
>    >
>    > > <solution-title>Answer</solution-title>
>    > >
>    > > The tool reports only two stretches consisting of Ns at both ends of the genome, and their positions are exactly identical between the samples. There are no interior Ns, nor other ambiguity characters.
>    > >
>    > > The 5'-end stretch of Ns extends up to position 264 (in bed format, so zero-based and the position itself not included). The binding site of the first primer in the primer scheme is from (bed format again) 235 up to 264.
>    > >
>    > > Remember that we used ivar trim to mask primer sequences? This seems to have worked well, and we got a consensus sequence from the first position for which we could reasonably expect information!
>    > >
>    > > The 3'-end stretch of Ns starts at position 150,453 and continues (as we have seen by eye before) until the end of the genome, which seems to be at 150,721.
>    > > Hmm, these values are surrounding the binding site of the last primer in the primer scheme, but do not match perfectly. But wait:
>    > >
>    > > It is possible that the consensus sequences incorporate deletions, i.e. it could be that they are shorter than the reference we have used for mapping. So obviously the **consensus sequences have a length of 

tools_json:
["Upload", "upload1", "Replace Text in entire line", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2", "Fasta regular expression finder", "toolshed.g2.bx.psu.edu/repos/mbernt/fasta_regex_finder/fasta_regex_finder/0.1.0", "fastp", "toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.23.2+galaxy0", "Map with BWA-MEM", "toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2", "Zip collections", "__ZIP_COLLECTION__", "Apply rules", "__APPLY_RULES__", "Samtools merge", "toolshed.g2.bx.psu.edu/repos/iuc/samtools_merge/samtools_merge/1.15.1+galaxy0", "Samtools view", "toolshed.g2.bx.psu.edu/repos/iuc/samtools_view/samtools_view/1.15.1+galaxy0", "ivar trim", "toolshed.g2.bx.psu.edu/repos/iuc/ivar_trim/ivar_trim/1.4.2+galaxy0", "QualiMap BamQC", "toolshed.g2.bx.psu.edu/repos/iuc/qualimap_bamqc/qualimap_bamqc/2.2.2d+galaxy3", "ivar consensus", "toolshed.g2.bx.psu.edu/repos/iuc/ivar_consensus/ivar_consensus/1.4.2+galaxy0", "Compute sequence length", "toolshed.g2.bx.psu.edu/repos/devteam/fasta_compute_length/fasta_compute_length/1.0.3", "MAFFT", "toolshed.g2.bx.psu.edu/repos/rnateam/mafft/rbc_mafft/7.520+galaxy0", "faToVcf", "toolshed.g2.bx.psu.edu/repos/iuc/ucsc_fatovcf/fatovcf/426+galaxy0", "MiModD File Information", "toolshed.g2.bx.psu.edu/repos/wolma/mimodd_main/mimodd_info/0.1.8_1"]


topic: variant-analysis
tutorial: somatic-variant-discovery
title: Somatic Variant Discovery from WES Data Using Control-FREEC
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: []
content_hash: c5a4bec885ea40f968edaf8dc55c3deb
last_modified: 2026-05-27T09:01:50.018228
zenodo_link: https://doi.org/10.5281/zenodo.2582555
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/somatic-variant-discovery/tutorial.html

content:
Human Copy Number Variations (hCNVs) are the result of structural genomic rearrangements that result in the duplication or deletion of DNA segments. These changes contribute significantly to human genetic variability, diseases, and somatic genome variations in cancer and other diseases {% cite Nam2015 %}. hCNVs can be routinely investigated by genomic hybridisation and sequencing technologies
. There is a range of software tools that can be used to identify and quantify hCNVs. Unfortunately, locating hCNVs is still a challenge in standardising formats for data representation and exchange. Furthermore, the sensitivity, specificity, reproducibility, and reusability of hCNV detection and analysis research software varies. As a result, there is a need for the adoption of community-developed standards for data discovery and exchange. To address that ELIXIR developed {% cite Beacon %} protocol and {% cite GA4GH %} standards, as well as mechanisms for annotating, benchmarking, creating reproducible and sharable tools and workflows, such as {% cite WorkflowHub %}, Galaxy, and {% cite OpenEBench %}, and, most importantly, accessible training resources and infrastructure.

 This tutorial is a modification of a Galaxy Training Network tutorial [Somatic variant calling]({% link topics/variant-analysis/tutorials/somatic-variants/tutorial.md %}) tutorial to provide training on how to preprocess, identify and visualise hCNV regions using Control-FreeC tool using tumor/normal samples pairs.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data Preparation
First, start with uploading and preparing the input data to analyze. The sequencing reads used in this analysis are from real-world data from a cancer patient’s tumor and normal tissue samples. For the sake of an acceptable speed of the analysis, the original data has been downsampled though to include only the reads from human chromosomes 5, 12 and 17.

| Name | Format | Origin | Encoding | Sequence length | Total Sequences | Chromosome  | Data size (MB) |
| ---- | -----| -------| ------  | -------- | --------------- | -------------- | ------------ |
| SLGFSK-N_231335_r1_chr5_12_17 | fastq | Normal tissue | Sanger / Illumina 1.9 | 101 |10602766 | 5, 12 and 7 | 530.4 MB |
| SLGFSK-N_231335_r2_chr5_12_17 | fastq | Normal tissue | Sanger / Illumina 1.9 | 101 | 10602766 | 5, 12 and 7 | 582.0 MB |
| SLGFSK-T_231336_r1_chr5_12_17 | fastq | Cancer tissue | Sanger / Illumina 1.9 | 101 |16293448 | 5, 12 and 7 | 811.3 MB |
| SLGFSK-T_231336_r2_chr5_12_17 | fastq  | Cancer tissue | Sanger / Illumina 1.9 | 101 |16293448 | 5, 12 and 7 | 868.7 MB |

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. For this tutorial, make a new history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import the data files from
>    [Zenodo](https://zenodo.org/record/2582555):
>
>    ```
>    https://zenodo.org/record/2582555/files/SLGFSK-N_231335_r1_chr5_12_17.fastq.gz
>    https://zenodo.org/record/2582555/files/SLGFSK-N_231335_r2_chr5_12_17.fastq.gz
>    https://zenodo.org/record/2582555/files/SLGFSK-T_231336_r1_chr5_12_17.fastq.gz
>    https://zenodo.org/record/2582555/files/SLGFSK-T_231336_r2_chr5_12_17.fastq.gz
>    ```
>   This will download four sequenced files ordered as: 
>   The first two files are for the forward and the reverse reads for the sample **normal**
>   tissue sequence.
>   The other two belong to the **tumor** reads.
> 
>    In some cases the same dataset can be found in the Galaxy shared data library. 
>    Ask the instructor for more details about this.
>
>    The dat aset can also be downloaded a local storage.  
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="fastqsanger.gz" %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Make sure to upload the sequences in fastaq format. Look at the history and
>    check if the created datasets have their data types assigned correctly with two reads for
>    the tumor tissues and two reads for the normal tissues. If not, fix any
>    missing or wrong data type assignments.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="fastqsanger.gz" %}
>
> 4. Give the data meaningful names and tags to facilitate analysis.
>     
>
>    When uploading data from a link, Galaxy names the files after the link address.
>    It might be useful to change or modify the name to something more meaningful.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
>
> 5. This tutorial has a set of shared steps performed on the data. To track the
>    data in the history, it is recommended to tag the datasets by attaching a meaningful tag '#'
>    to them. The tagging will automatically be attached to any file generated
>    from the original tagged dataset.
>   *e.g.*, `#normal` for normal tissue datasets (with `-N_` in the name) and
>   *e.g.*, `#tumor` for tumor dataset (with `-T_` in the name).
>
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
>
{: .hands_on}

# Quality control and mapping of NGS reads
The data was obtained following a series of laboratory procedures, including DNA preparation, extraction, and sequencing, which means there is a possibility of errors occurring during those steps, which could affect data quality. To address that, it is necessary to test the quality of the fastq reads. The data quality needs to be within an acceptable range before looking for hCNVs. The low-quality data can lead us to false results. To detect low-quality data,  preprocessing step is required to trim or discard the low-quality reads before proceeding with the mapping and hCNV detection steps.

> <comment-title>More on quality control and mapping</comment-title>
>  To read more about quality control this is tutorial on Galaxy training network 
> [ Quality Control]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}) 
> For mapping [Mapping]({% link topics/sequence-analysis/tutorials/mapping/tutorial.md %})
{: .comment}


## Quality Control

To perform quality control, we will use [Falco](https://falco.readthedocs.io/en/latest/) an efficiency-optimized rewrite of [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/).

> <hands-on-title>Quality control of the input datasets</hands-on-title>
> 1. Run {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} on the fastq datasets
>       - {% icon param-files %} *"Raw read data from the current history"*: all 4 FASTQ  datasets selected with **Multiple datasets**
>
>    {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>
>    This job will generate eight new datasets to the history. To
>    parse the quality results view the html report of each dataset.
>
>    For the next step use the raw data findings from Falco.
>
> 2. Use {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0) %} to aggregate the raw **Falco** data of all four input datasets into one report
>      - In *"Results"*
>        - *"Which tool was used generate logs?"*: `FastQC` (no matter whether FastQC or Falco was used)
>        - In *"FastQC output"*
>           - *"Type of FastQC output?"*: `Raw data`
>           - {% icon param-files %} *"FastQC output"*: all four *RawData*
>             outputs of **Falco** {% icon tool %})
>
> 3. Inspect the *Webpage* output produced by the tool
>
>    > <question-title></question-title>
>    >
>    > What do you feel about the sequence's overall quality? 
>    > ![fastqbefore](../../images/Control_Freec_fastqc_per_base_sequence_quality_plot_before.png "Sequence quality per base generated by Falco before end trimming.")
>    > ![GCbefore](../../images/Control_Freec_fastqc_per_sequence_gc_content_plot_before.png "Sequence quality per Sequence GC content generated by Falco before end trimming.")
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The forwards and reversed reads show good 
>    > >    quality, , with no major issues discovered
>    > >    discovered during the preparation process..
>    > >
>    > > 2. TThe GC content plots for the samples' forward and reverse reads 
>    > >    show an unusual bimodal distribution.
>    > >
>    > >    The unnormal distribution of the GC content of reads from a sample is 
>    > >    usually interpreted as a sign of possible contamination.
>    > >    However, we are dealing with sequencing data from captured exomes,
>    > >    which means that the reads do not represent random sequences from a  genome.
>    > >    They rather represent an arbitrary selection. Indeed, the samples were 
>    > >    prepared using Agilent's SureSelect V5 technology for exome enrichment, 
>    > >    and bimodal GC content distributions have been identified as a hallmark 
>    > >    of that capture method
>    > >    for example, see Fig. 4C in
>    > >   {% cite Meienberg2015 %}.
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

## Read trimming and filtering
As previously demonstrated, The data have relatively high-quality sequenced reads.
However, the aim is to detect clear reads for hCNVs and will use a trimming step to see if the analysis can be improved.


> <hands-on-title>Read trimming and filtering of the normal tissue reads</hands-on-title>
> 1. Run {% tool [Trimmomatic](toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.36.5) %} to trim and filter the normal tissue reads
>    - *"Single-end or paired-end reads?"*: `Paired-end (two separate
>      input files)`
>      
>      This makes the tool treat the forward and reverse reads simultaneously.
>
>      - {% icon param-file %} *"Input FASTQ file (R1/first of pair)"*: the forward reads (r1) dataset of the normal tissue sample
>      - {% icon param-file %} *"Input FASTQ file (R2/second of pair)"*: the reverse reads (r2) dataset of the normal tissue sample
>
>    - *"Perform initial ILLUMINACLIP step?"*: `Yes`
>       - *"Select standard adapter sequences or provide custom?"*: `Standard`
>       - *"Adapter sequences to use"*: `TruSeq3 (paired-ended, for MiSeq and HiSeq)`
>       - *"Maximum mismatch count which will still allow a full match to be performed"*: `2`
>       - *"How accurate the match between the two 'adapter ligated' reads must be for PE palindrome read alignment"*: `30`
>       - *"How accurate the match between any adapter etc. sequence must be against a read"*: `10`
>       - *"Minimum length of adapter that needs to be detected (PE specific/ palindrome mode)"*: `8`
>       - *"Always keep both reads (PE specific/palindrome mode)?"*: `Yes`
>
>       These parameters are used to cut ILLUMINA-specific adapter sequences
>       from the reads.
>
>    - *"Trimmomatic Operation"*
>      - *"Select Trimmomatic operation to perform"*: `Cut the specified number of bases from the start of the read (HEADCROP)`
>      - *"Number of bases to remove from the start of the read"*: `3`
>      - {% icon param-repeat %} "Insert Trimmomatic Operation"*
>        - *"Select Trimmomatic operation to perform"*: `Cut bases off the end of a read, if below a threshold quality (TRAILING)`
>        - *"Minimum quality required to keep a base"*: `10`
>      - {% icon param-repeat %} "Insert Trimmomatic Operation"*
>        - *"Select Trimmomatic operation to perform"*: `Drop reads below a
>       specified length (MINLEN)`
>        - *"Minimum quality required to keep a base"*: `25`
>
>    
{: .hands_on}

This step will creates four files in the history. The sizes of those two files vary depending on the original data quality and trimming intensity. The first two files are for mated forward and reverse reads, respectively.
The other two are for unmated reads as a result of excessive trimming.
However, because of the high average data quality, there was no need to perform excessive trimming by selecting the previous three trimming conditions, so those files should be empty. Those files can be heden to keep the history cleaner.

> <details-title>Keeping attention on the data type</details-title>
> Track whether the reads are paired or unpaired, and remember to include them in any tool to be used.
> The reason is that there are some tools, such as read mappers,
> that expect reads to be in a specific order and having unmapped reads can result in significant.
{: .details}

> <hands-on-title>Read trimming and filtering of the tumor tissue reads</hands-on-title>
> 2. repeat the previous step for **tumor tissue** reads following the same steps as above.
{: .hands_on}

> <hands-on-title>Exercise: Quality control of the polished datasets</hands-on-title>
> Use {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} and {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0) %} like before,
> but using the four trimmed datasets produced by Trimmomatic as input.
>
>    > <question-title></question-title>
>    >
>    > Is there any difference between the reads before and after the trimming?
>    > ![fastqafter](../../images/Control_Freec_fastqc_per_base_sequence_quality_plot_after.png "Sequence quality per base generated by Falco after end trimming. The x-axis on the grapgh is for the read length and the y-axis shows the quality scores. The higher the score the better the base call. The background of the graph divides the y axis into very good quality calls (green), calls of reasonable quality (orange), and calls of poor quality (red).")
>    > ![GCafter](../../images/Control_Freec_fastqc_per_sequence_gc_content_plot_afrer.png "Sequence quality per Sequence GC content generated by Falco after end trimming the plot shows an unusually shaped distribution that could indicate a contaminated library or some other kinds of biased subset")
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The quality of the data is good, so 
>    > > trimming them didn't lead to dramatic changes. However,
>    > > we can point out that some of the adapters were removed.  
>    > {: .solution}
>   {: .question}
{: .hands_on}

## Read Mapping

> <hands-on-title>Read Mapping</hands-on-title>
> 1. Use {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.1) %} to map the reads from the **normal tissue** sample to the reference genome
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>        - *"Using reference genome"*: `Human: hg19` (or a similarly named option)
>
>      > <comment-title>Using the imported `hg19` sequence</comment-title>
>      > If you have imported the `hg19` sequence as a fasta dataset into your
>      > history instead:
>      >   - *"Will you select a reference genome from your history or use a
>      >     built-in index?"*: `Use a genome from history and build index`
>      >      - {% icon param-file %} *"Use the following dataset as the reference sequence"*: your imported `hg19` fasta dataset.
>     {: .comment}
>
>    - *"Single or Paired-end reads"*: `Paired`
>       - {% icon param-file %} *"Select first set of reads"*: the trimmed
>         forward reads (r1) dataset of the **normal tissue** sample; output of
>        **Trimmomatic** {% icon tool %}
>       - {% icon param-file %} *"Select second set of reads"*: the trimmed
>         reverse reads (r2) dataset of the **normal tissue** sample; output of
>        **Trimmomatic** {% icon tool %}
>    - *"Set read groups information?"*: `Set read groups (SAM/BAM specification)`
>      - *"Auto-assign"*: `No`
>        - *"Read group identifier (ID)"*: `Not available.`
>      - *"Auto-assign"*: `No`
>        - *"Read group sample name (SM)"*: `Not available.`
>      - *"Platform/technology used to produce the reads (PL)"*: `ILLUNINA`
>      - *"Select analysis mode"*: `Simple illumina mode`
>
>    > <comment-title>More on read group identifiers and sample names</comment-title>
>    > In general, we can choose our own ID and SM values, but the ID should
>    > unambiguously identify the sequencing run that produced the reads,
>    > while the SM value should identify the biological sample.
>   {: .comment}
>
> 2. Use {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.1) %} to map the reads from the **tumor tissue** sample
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>        - *"Using reference genome"*: `Human: hg19` (or a similarly named option)
>
>      Adjust these settings as before in case of using imported reference
>      genome.
>    - *"Single or Paired-end reads"*: `Paired`
>       - {% icon param-file %} *"Select first set of reads"*: the trimmed
>         forward reads (r1) dataset of the **tumor tissue** sample; output of
>        **Trimmomatic** {% icon tool %}
>       - {% icon param-file %} *"Select second set of reads"*: the reverse
>         reads (r2) dataset of the **tumor tissue** sample; output of
>        **Trimmomatic** {% icon tool %}
>    - *"Set read groups information?"*: `Set read groups (SAM/BAM specification)`
>      - *"Auto-assign"*: `No`
>        - *"Read group identifier (ID)"*: `Not available.`
>      - *"Auto-assign"*: `No`
>        - *"Read group sample name (SM)"*: `Not available.`
>      - *"Platform/technology used to produce the reads (PL)"*: `ILLUNINA`
>      - *"Select analysis mode"*: `Simple illumina mode`
{: .hands_on}

Name the created list as **Mapping-lsit**




# Copy Number Variation detection (hCNV).

After the mapping step, the data are ready to the hCNV detection step. This tutorial focuses on the **Control-FreeC** tool as a method for hCNV identification:

- Identifies variant alleles in tumor/normal pair samples.
- Visualize the hCNV using the Circos tool.

## Mapped reads filtering

The remaining data preprocessing until the Control-FreeC step is the same for Normal and Tumor reads.  Create a data collection to include those two files.

> <hands-on-title>Filtrate the mapped reads</hands-on-title>
> 1. Use {% tool [Build list](__BUILD_LIST__) %} to creat a list from the maped reads of the **normal tissue** and **tumor tissue**
>    - {% icon param-file %} *"Dataset"*: `Insert dataset`
>       - *"Input dataset"*: `The output of map with BWA-MEM for normal tissue`
>    - {% icon param-file %} *"Dataset"*: `Insert dataset`
>       - *"Input dataset"*: `The output of map with BWA-MEM for cancer tissue`
> 2. Run {% tool [Create text file](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_text_file_with_recurring_lines/1.1.0) %}  with the following parameters
>   - *"Characters to insert"*: `normal reads`
>   - *"Specify the number of iterations by"*: `User defined number`
>   - *"How many times?"*: `1`
>   - {% icon param-repeat %} "Insert selection"*
>      - *"Characters to insert"*: `tumor reads`
>      - *"Specify the number of iterations by"*: `User defined number`
>      - *"How many times?"*: `1`
>   - This will create a text file with only two lines normal reads and tumor reads
>
> 3. Run {% tool [Relabel identifiers](__RELABEL_FROM_FILE__) %}  with the following parameters
>   - {% icon param-collection %} *"Input Collection"*: the creat a list from `Build lsit tool`.
>   - *"How should the new labels be specified?"*: `Using lines in a simple text file.`
>   - *"New Identifiers"*: The outcom from `Create txt file tool`
>   - *"Ensure strict mapping"*: `False`
>
>   - Make sure to use the same files order for the both of **Build list** tool and **Creat text file**.   
{: .hands_on}

It is essential to filtrate the reads before the mapping step.
It is also required to filtrate the reads after. The preprocessing step
works by filtrating the mapped reads by removing the low-quality regions
and the duplicated reads. This step is needed to reduce the running time
and in results interpretation.

> <hands-on-title>Data filtration and Remove duplicates</hands-on-title>
> 1. Run {% tool [Samtools view](toolshed.g2.bx.psu.edu/repos/iuc/samtools_view/samtools_view/1.9+galaxy3) %}  with the following parameters
>   - {% icon param-collection %} *"SAM/BAM/CRAM dataset"*: The outpot of  `Relabel identifiers` dataset cpllection.
>   - *"What would you like to look at?"*: `A filtered/subsampled selection of reads`
>   - *"Configure filters:"*:
>     - *"Filter by quality"*: `1`
>     - *"Exclude reads with any of the following flags set"*: `Read is unmapped` `Mate  is unmapped`
>   - *Produce extra dataset with dropped reads?"*: `False`
>   - *"Output format"*: `BAM (-b)`
>   - *"Reference data"*: `No`
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
> 2. Run {% tool [RmDup](toolshed.g2.bx.psu.edu/repos/devteam/samtools_rmdup/samtools_rmdup/2.0.1) %}  with the following parameters
>   - {% icon param-collection %} *"BAM File"*:  The outpot of  `Samtools view`
>   - *"Is this paired-end or single end data"*: `BAM is paired-end`
>   - *"Treat as single-end"*: `False`
{: .hands_on}




## Homogenize mapped reads

To detect hCNVs expression accurately. The reads must go through the lift alignment process. Lift Alignment works by shifting reads that contain indels to the left of a reference genome until they can not be shifted anymore. As a result, it will only extract reads with indels, with no false reads (reads that mismatch with the reference genome other than the indels).


> <hands-on-title>Homogenize the positional distributed indels</hands-on-title>
> 1. Run {% tool [BamLeftAlign](toolshed.g2.bx.psu.edu/repos/devteam/freebayes/bamleftalign/1.3.1) %}  with the following parameters
>   - *"Choose the source for the reference genome"*: `Locally cached `
>   - {% icon param-collection %} *"Select alignment file in BAM format"*: The outpot of  `tool RmDup`.
>   - *"Using reference genome"*: `hg19`
>   - *"Maximum number of iterations"*: `5`
>
> 2. Run {% tool [Samtools calmd](toolshed.g2.bx.psu.edu/repos/devteam/samtools_calmd/samtools_calmd/2.0.2) %}  with the following parameters
>   - {% icon param-collection %} *"BAM file to recalculate"*: The outpot of  `BamLeftAlign`.
>   - *"Choose the source for the reference genome"*: `Use a built-in genome`
>   - *"Using reference genome"*: `hg19`
>   - *"Do you also want BAQ (Base Alignment Quality) scores to be calculated?"*: `No`
>   - *"Additional options"*: `Advanced options`
>   - *"Change identical bases to '='"*: `False`
>   - *"Coefficient to cap mapping quality of poorly mapped reads"*: `50`
{: .hands_on}


After the Homogenizing step, it is now to extract the reads which hold indels from the mapped reads.

## Filtrate indels

> <hands-on-title>Filtrate the indels reads</hands-on-title>
> 1. Run {% tool [Samtools view](toolshed.g2.bx.psu.edu/repos/iuc/samtools_view/samtools_view/1.9+galaxy3) %}  with the following parameters
>   - {% icon param-collection %} *"BAM file to recalculate"*: The outpot of  `Samtools CalMD`.
>   - *"What would you like to look at?"*: `Just the input header (-H)`
>   - *"What would you like to have reported?"*: `The header in ...`
>   - *"Output format"*: `SAM`
>   - *"Reference data"*: `No, see help (-output-fmt-option no_ref)`  
> 2. Run {% tool [Select](Grep1) %}  with the following parameters
>   - {% icon param-collection %} *"Select lines from"*: The outpot of  `Samtools view`.
>   - *"that"*: `Matching`
>   - *"the pattern"*: `^@SQ\tSN:(chr[0-9]+|chrM|chrX|chrY)\tLN:[0-9]+`
>   - *"Keep header line"*: `False`
>
>
> 3. Run {% tool [Replace Text in entire line](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2) %} with the following parameters:
>   - {% icon param-collection %} *"Select lines from"*: The output of `Select`
>   - *"pattern"*: `^@SQ\tSN:(chr[0-9]+|chrM|chrX|chrY)\tLN:([0-9]+)`
>   - *"Replace with"*: `\1\t0\t\2`
>
>    This step will generate a data collection folder with two files inside. Change the datatype for the files inside it to **BED format**.
>
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
>
> 4. Run {% tool [Samtools view](toolshed.g2.bx.psu.edu/repos/iuc/samtools_view/samtools_view/1.9+galaxy3) %}  with the following parameters
>   - {% icon param-collection %} *"SAM/BAM/CRAM data set"*: The outpot of  `CalMD`.
>   - *"What would you like to look at?"*: `A filtered/subsampled selection of reads`
>   - *Configure filters*
>   - *"Filter by quality"*: `255`
>   - *What would you like to have reported?"*: `Reads dropped during filtering and subsampling`
>   - *Produce extra dataset with dropped reads?"*: `False`
>   - *"Output format"*: `BAM (-b)`
>   - *"Reference data"*: `No`
> 5. Run {% tool [Samtools view](toolshed.g2.bx.psu.edu/repos/iuc/samtools_view/samtools_view/1.9+galaxy3) %}  with the following parameters
>   - {% icon param-collection %} *"SAM/BAM/CRAM data set"*: The outpot of  `Samtools view`.
>   - *"What would you like to look at?"*: `A filtered/subsampled selection of reads`
>   - *"Configure filters"*
>   - *"Filter by regions:"*: `Regions from BED file`
>   - {% icon param-collection %} *"Filter by intervals in a bed file:"*: The outpot of  ` Replace Text` BED format.
>   - *"Filter by readgroup"*: `NO`
>   - *"Filter by quality"*: `1`
>   - *Produce extra dataset with dropped reads?"*: `False`
>   - *"Output format"*: `BAM (-b)`
>   - *"Reference data"*:`No`
{: .hands_on}

> <hands-on-title>Extract files form list</hands-on-title>
> extract the files from the list to handel them separitly
> 1. Run {% tool [Extract Dataset](__EXTRACT_DATASET__) %}  with the following parameters:
>   - {% icon param-collection %} *"Input List"*: `The outpot of Samtools view`.
>   - *"How should a dataset be selected?"*: `Select by element identifier`
>   - *"Element identifier:"*: `tumor reads`
>
> 2. Run {% tool [Extract Dataset](__EXTRACT_DATASET__) %}  with the following parameters:
>   - {% icon param-collection %} *"Input List"*: The outpot of  `Samtools view`.
>   - *"How should a dataset be selected?"*: `Select by element identifier`
>   - *"Element identifier:"*: `normal reads`
{: .hands_on}







## Control_FREEC for hCNV detection  
The data are now ready to detect hCNV. Control-FREEC detects copy-number alterations and allelic imbalances (including loss of heterozygosity; LOH) by automatically computing, normalising, and segmenting copy number profile and beta allele frequency (BAF) profile, and then calling copy number alterations and LOH. Control-FREEC differentiates between somatic and germline variants. Based on those profiles. The control reads display the gene status for each segment.
![Controlfreec](../../images/Control_Freec_output.png "1. Control-FREEC calculates copy number and BAF profiles and detects regions of copy number gain/loss and LOH regions in chromosomes 5,12 17. Gains, losses and LOH are shown in red, blue and light blue, respectively.")

> <comment-title>More on control_FREEC and hCNVs detection</comment-title>
> Control-freec works by: 
> 1. Annotating genomic changes and heterozygosity loss in the sample dataset.
> 2. Distinguishes between germline and somatic variants by creating copy number profile and BAF profile.
> 3. Employs the information in those profiles to detect copy number changes in sample reads.
>
{: .comment}


> <question-title></question-title>
> Can you expect the essential factors that afflict hCNVs detection?
>
> > <solution-title></solution-title>
> >
> > 1. Coverage bias in reads 
> > Changes in reading mobility and GC content may favour the duplication of specific reads over others.
> > 2. Bias in reading alignment
> > Because normal cells have higher read coverage than allelic reads during alignment, they may be classified as noise findings.
> > 3. Normal cell contamination 
> > The presence of normal cells within tumour cells can have an impact on the construction of a tumour genome's copy number profile.
> {: .solution}
{: .question}





> <hands-on-title>Detection of copy-number changes</hands-on-title>
> 1. Import the BED file for the captured reagions from [Zenodo](https://doi.org/10.5281/zenodo.5697358):
>
>    ```
>    https://zenodo.org/record/5697358/files/capture_targets_chr5_12_17.bed
>    ```
>
 > 2. Run {% tool [Control-FREEC](toolshed.g2.bx.psu.edu/repos/iuc/control_freec/control_freec/11.6+galaxy1) %}  with the following parameters
>   - *"Select the sequencing method of the input file(s)?"*: `whole-exome sequencing (WES)`
>   - {% icon param-file %} *"Sample file"*: The outpot of  `Extract Dataset` (**cancer reads**)
>   - {% icon param-file %} *"Control file"*: The outpot of  `Extract Dataset` (**normal reads**)
>   - {% icon param-file %} *"BED file with capture regions"*: `chr 5, 7, 12 capture reagions **Input file**` 
>   - *"Format of reads"*: `Illumina paired-end (FR)`
>
>   - *"Advanced WES settings:"*:
>     - *"Degree of polynomial:"*: `control-read-count-based normalization, WES (1)`
>     - *"Read Count (RC) correction for GC-content bias and low mappability:"*: `normalize the sample and the control RC using GC-content and then calculate the ratio "Sample RC/contol RC" (1)`
>     - *"Minimal number of consecutive windows to call a CNA"*: `WES (3)`
>     - *"Segmentation of normalized profiles (break point)"*: `1.2`
>     - *"Desired behavior in the ambiguous regions"-1" "*: `make a separate fragment of this "unknown" region and do not assign any copy number to this region at all (4)`
>     - *"Adjust sample contamination?"*: `True`
>     - *"Sample contamination by normal cells"*: `0.30000000000000004`
>     - *"Intercept of polynomial"*: `with GC-content (1)`
>     - *"Sample sex"*: `XX`
>
>   - *"Choose the source for the reference genome"*: `Locally cached`
>   - *"Reference genome"*: `hg19`
>   - *"Outputs:"*:
>     - *"BedGraph Output for UCSC genome browser"*: `False`
>     - *"Visualize normalized copy number profile with predicted CNAs"*: `True`
>     - *"2D data track file for Circos"*: `True`
>
> > <question-title></question-title>
> >
> > In your opinion, what are the challenges in hCNVs detection?
> > > <solution-title></solution-title>
> > >
> > > 1. The bias in Reads coverage changes in reads mobility and GC content can lead to 
> > > favouring the duplication of specific reads over others.
> > > 2. The bias in reads alignment. 
> > > Normal cells' reads coverage is higher than the allelic reads, for 
> > > that the allelic reads some times expressed as noise findings
> > > 3. Contamination with normal cells 
> > > The availability of normal cells within tumor cells can
> > > effect on the construction of the copy number profile of a 
> > > tumor genome
> > {: .solution}
> {: .question}
{: .hands_on}


## Visualise detected hCNVs

Cicros demonstrates the relationship and the positions of different objects with an appealing,
high quality and illustrative multilayers circular plot. Circos gives the user flexibility to
present the link between their data at a high rate by providing the ability to control the features 
and elements in creating the plot. Circos visualise the genomic alterations in genome structure and 
the relationships between the genomic intervals [Krzywinski, Schein et al. 2009](https://genome.cshlp.org/content/19/9/1639.short).

> <hands-on-title>Visualise the hCNV findings</hands-on-title>
> 1. Run {% tool [Circos](toolshed.g2.bx.psu.edu/repos/iuc/circos/circos/0.69.8+galaxy7) %}  with the following parameters
>   - *"Reference Genome Source"*: `Custom Karyotype`
>   - {% icon param-file %} *"Sample file"*: The outpot of  `Output dataset out_chr_sorted_circos from control freec`
>   - *"Ideogram:"*:
>     - *Spacing Between Ideograms (in chromosome units)"*: `3.0`
>     - *"Radius"*: `0.8`
>     - *"Thickness"*: `45.0`
>     - *"Labels:"*:
>       - *"Radius"*: `0.01`
>       - *"Label Font Size"*: `40`
>     - *"Cytogenic Bands:"*:
>       - *"Band Stroke Color"*: `Black`
>    - *"2D Data Tracks:"*:
>      - {% icon galaxy-wf-new %} "Insert 2D Data Plot"*
>        - *"Outside Radius"*: `0.95`
>        - *"Plot Type"*: `Scatter`
>        - {% icon param-file %} *"Scatter Plot Data Source"*: `Output dataset 'out_ratio_log2_circos' from Control-FreeC`
>        - *"Plot Format Specific Options:"*:
>          - *"Glyph"*: `Circle`
>          - *"Glyph Size"*: `4`
>          - *"Fill Color"*: `Gray`
>          - *"Stroke Color"*: `Black`
>          - *"Stroke Thickness"*: `0`  
>         - *"Rules:"*:
>           - {% icon galaxy-wf-new %} *"Insert Rule"*:
>             - {% icon galaxy-wf-new %} *"Insert Conditions to Apply"*
>               - *"Condition"*: `Based on value (ONLY for scatter/histogram/heatmap/line)`
>               - *"Points above this value"*: `0.0`
>             - {% icon galaxy-wf-new %} *"Insert Actions to Apply"*
>               - *"Action"*: `Change Fill Color for all points`
>               - *"Fill Color"*: `Red`
>             - *"Continue flow"*: `False`
>           - {% icon param-repeat %} *"Insert Rule"*:
>               - {% icon galaxy-wf-new %} *"Insert Conditions to Apply"*:
>                 - *"Condition"*: `Based on value (ONLY for scatter/histogram/heatmap/line)`
>                 - *"Points below this value"*: `0.0`
>               - {% icon galaxy-wf-new %} *"Insert Actions to ApplyInsert Actions to Apply"*:
>                 - *"Action"*: `Change Fill Color for all points`
>                 - *"Fill Color"*: `Blue`
>               - *"Continue flow"*: `False`
>         -  In*"Axes:"*:
>           - {% icon galaxy-wf-new %} *"Insert Conditions to Apply"*
>             - *"Radial Position"*: `Absolute position (values match data values)`
>             - *"Spacing"*: `1.0`
>             - *"y0"*: `-4.0`
>             - *"y1"*: `4.0`
>             - *"Color"*: `Gray`
>             - *"Color Transparency"*: `1.0`
>             - *"Thickness"*: `2`
>           - *"When to show"*: `Always`
>   - *"Limits:"*:
>     - *"Maximum number of links to draw"*: `2500000`
>     - *"Maximum number of points per track"*: `2500000`
{: .hands_on}

> <question-title></question-title>
> Can you interpret generated plot from the Circos tool?
> ![Cicros](../../images/Control_Freec-%5BCircos_Plot%5D.png "	This circos plot shows of genetic alterations in chromosomes 5,12 and 17 tutmer tissues. The outermost circle represents the targeted chromosomes and inner circle for deletion related and amplificated related variations ")
> ![Cicros](../../images/Control_Freec_Cicros_zoomed.png "Zoomed hCNV regions detected by Control_FREEC using Cicros")
> > <solution-title></solution-title>
> > The outermost circle represents the targeted chromosomes. 
> > The inner circle
> > shows the copy number changes in those regions.
> >
> > The inner circle has two parts. The deletion-related variations are the red dots directed to the centre 
> > (below the centre line), while the amplificated variations are the green dots pointed out of the center (toward chromosomes).
> {: .solution}
{: .question}

# Conclusion

In this tutorial, we introduced Contol-FreeC as an alternative tool for detecting hCNVs and highlighted the steps for preparing reads and analysis.

tools_json:
["Falco", "toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0", "Trimmomatic", "toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.36.5", "Map with BWA-MEM", "toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.1", "Build list", "__BUILD_LIST__", "Create text file", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_text_file_with_recurring_lines/1.1.0", "Relabel identifiers", "__RELABEL_FROM_FILE__", "Samtools view", "toolshed.g2.bx.psu.edu/repos/iuc/samtools_view/samtools_view/1.9+galaxy3", "RmDup", "toolshed.g2.bx.psu.edu/repos/devteam/samtools_rmdup/samtools_rmdup/2.0.1", "BamLeftAlign", "toolshed.g2.bx.psu.edu/repos/devteam/freebayes/bamleftalign/1.3.1", "Samtools calmd", "toolshed.g2.bx.psu.edu/repos/devteam/samtools_calmd/samtools_calmd/2.0.2", "Select", "Grep1", "Replace Text in entire line", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2", "Extract Dataset", "__EXTRACT_DATASET__", "Control-FREEC", "toolshed.g2.bx.psu.edu/repos/iuc/control_freec/control_freec/11.6+galaxy1", "Circos", "toolshed.g2.bx.psu.edu/repos/iuc/circos/circos/0.69.8+galaxy7"]


