# tutorials

topic: variant-analysis
tutorial: mapping-by-sequencing
title: Mapping and molecular identification of phenotype-causing mutations
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 858a6515a3121892e289e1b8fd22ee87
last_modified: 2026-02-26T21:12:18.621133
zenodo_link: https://doi.org/10.5281/zenodo.1098033
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/mapping-by-sequencing/tutorial.html

content:
In order to map and identify phenotype-causing mutations efficiently from a
single experiment, modern genetic research aims to combine classical genetic
mapping concepts with the power of next-generation-sequencing.

For example, after having obtained a mutant strain of an organism with a particular phenotype from a forward genetic screen, the classical approach towards
identification of the underlying causative mutation would be to:

1. perform mapping crosses to probe for linkage between the unknown mutation
   and selected markers with known location on the genome, in order to determine
   the approximate genomic region that the mutation resides in, then
2. sequence candidate DNA stretches in this region to identify the precise
   nature of the mutation.

Modern approaches, in contrast, use just one set of genome-wide sequencing data
obtained from mapping-cross progeny and suitable parent strains to
simultaneously:

* discover available marker mutations for linkage analysis,
* map the causative mutation using these markers, and
* identify candidate mutations.

Since it uses sequencing data already at the mapping step, this process is
called **mapping-by-sequencing**.

Since genome-wide sequencing will often reveal a few hundreds to many thousands
of marker variants that should be considered together in the linkage analysis,
mapping-by-sequencing data is typically analyzed with dedicated computational
tools. For Galaxy, the **MiModD** suite of tools offers efficient and flexible
analysis workflows compatible with a variety of mapping-by-sequencing
approaches.

> <details-title>Further reading</details-title>
> The MiModD documentation has its own chapter on supported
> [mapping-by-sequencing schemes](http://mimodd.readthedocs.io/en/latest/nacreousmap.html)
> not covered here.
{: .details}

In this tutorial we are going to use several of these tools to map and identify
a point mutation in an *Arabidopsis thaliana* strain from whole-genome
sequencing data. This will demonstrate one particular kind of linkage analysis
supported by **MiModD** called *Variant Allele Frequency Mapping*.

### Variant Allele Frequency Mapping

![Principal of variant allele frequency mapping](../../images/mbs_scheme.png "Principle of variant allele frequency mapping")

As illustrated in Figure 1, variant allele frequency mapping analyzes marker
variant inheritance patterns in pooled F2 recombinants selected for the
mutation to be mapped.

In the figure we assume a diploid
organism with just two pairs (dark and light grey) of chromosomes.

A mutant line established for any organism will harbor
two types of variants (diamonds):
- variants induced during mutagenesis (orange), among them the causative
  variant responsible for the mutant phenotype (yellow)
- variants (with respect to the species' reference genome) that were
  present in the premutagenesis strain (green open diamonds)

For mapping the causative mutation, additional variants (blue open diamonds)
are introduced through a cross to a mapping strain and F2 progeny is selected
for the mutant phenotype. While each individual in the F2 generation carries
its uniquely recombined variant pattern (3 possible outcomes for the diploid
genome are shown in the figure), this phenotypic selection will work against
mapping strain variants in the vicinity (yellow boxes) of the causative
variant. Hence, when the recombined DNA from phenotypic F2 progeny gets pooled
and sequenced, the causative variant can be found by looking for a region for
which all sequenced reads support variant alleles found in the original mutant
line, rather than mapping strain variant alleles.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Data Preparation

In this tutorial we are going to (re-)identify the recessive causative mutation
in a late-flowering mutant line of *Arabidopsis thaliana*.

> <comment-title></comment-title>
> **MiModD** was developed as a successor of the older and now unmaintained
> *CloudMap* suite of tools, but also uses some key concepts first implemented
> in the command line package *SHOREmap*.
> The data used in this tutorial represent a subsample of the
> [SHOREmap proof-of-principle data](http://bioinfo.mpipz.mpg.de/shoremap/examples.html)
> (specifically of the `OCF2` and `Ler` datasets).
{: .comment}

The mutant line was obtained from a mutagenesis screen on the **Col**umbia
reference background of *A. thaliana*. For mapping-by-sequencing, the line was
outcrossed to the highly polymorphic **L**andsberg ***er**ecta* background as a
source of marker variants. From this outcross, 119 F2 siblings showing the
original late-flowering phenotype were recovered and their genomic DNA was
pooled and subjected to whole-genome sequencing.
Together with whole-genome sequencing data of a pure L*er* background strain,
this outcrossed pool data forms the starting point of our analysis.
The L*er* strain sample, thus, can inform us about the markers that the
outcrossed F2 pool *could* have inherited and exclusion of L*er* markers from
a particular genomic region from the F2 pool provides evidence that the
phenotype causing variant lies close to this region. With respect to
[Figure 1](#figure-1), this means that we will only use "blue diamond" marker
variants introduced through the mapping cross for the linkage analysis and
ignore potentially informative (green and orange) variants found in the
original mutant line.

For your convenience, the sequenced reads from both datasets have already been
mapped (using **Bowtie2**) to the *A. thaliana* Col reference genome (version
TAIR10) so the two datasets take the form of BAM files with the mapping results
(see the tutorial on
[NGS data mapping](../../../sequence-analysis/tutorials/mapping/tutorial.html)
if you do not know what this means).

> <hands-on-title>Data upload and preprocessing</hands-on-title>
>
> > <tip-title></tip-title>
> > This section assumes that you already know
> >
> > - how to upload data to Galaxy via links
> > - edit dataset attributes
> > - do some basic text manipulation on datasets
> >
> > An excellent introduction into these topics can be found in the
> > [From peaks to genes](../../../introduction/tutorials/galaxy-intro-peaks2genes/tutorial.html)
> > tutorial (specifically, in the sections on
> > [Data upload](../../../introduction/tutorials/galaxy-intro-peaks2genes/tutorial.html#data-upload)
> > and
> > [File preparation](../../../introduction/tutorials/galaxy-intro-peaks2genes/tutorial.html#file-preparation)).
> {: .tip}
>
> 1. Import the two BAM datasets representing the:
>    - [L*er* mapping strain](https://zenodo.org/record/1098034/files/Ler_mapping_strain.bam)
>    - [outcrossed F2 pool](https://zenodo.org/record/1098034/files/outcrossed_F2.bam)
> 2. Specify the genome that was used at the reads mapping step
>    1. Click on **Edit attributes** (the pencil icon displayed with the dataset)
>    2. As the **Database/Build** select `Arabidopsis thaliana TAIR10`
>    3. **Save** the edited attributes
> 3. Import the reference genome
>
>    > <comment-title>TAIR10 Version</comment-title>
>    > This step is only necessary if the TAIR10 version of the *A. thaliana*
>    > genome is not available as a built-in genome on your Galaxy server.
>    {: .comment}
>
>    1. Upload the reference genome using the URL:
>       `https://www.arabidopsis.org/download_files/Genes/TAIR10_genome_release/TAIR10_chromosome_files/TAIR10_chr_all.fas`
>    2. Use **Replace parts of text** {% icon tool %} to truncate the title
>       lines in the fasta dataset to just the bare chromosome names. Set:
>       - the **Find pattern** to `^>(\w+).*`
>
>         This will find entire fasta title lines, or more specifically, any
>         lines starting with `>` followed by at least one letter, digit or
>         underscore.
>       - the **Replace with** pattern to `>chr$1`
>
>         The `$1` in it refers to the parenthesized part of the find pattern,
>         *i.e.*, just the substring of letters/digits/underscores directly
>         following the `>`.
>       - **Find-Pattern is a regular expression** to `Yes`
>    3. Make sure the datatype of the resulting dataset is set to `fasta` and
>       give the dataset a nice name (*e.g.*, TAIR10 reference).
>    4. At this point you can delete the originally downloaded dataset.
{: .hands_on}


# Joint Variant Calling and Extraction

A hallmark of all mapping-by-sequencing analyses is that one would like to
follow the fate of variants through genetic crosses. In this tutorial, for
example, we would like to determine, for every marker variant provided by the
L*er* background, at what frequency it got inherited by the outcrossed F2 pool.
For such analyses to work reliably, it is best to call variants jointly for all
relevant samples because this allows the variant caller to give us per-sample
variant call statistics for all sites at which **any** sample is found to have
a non-wt genotype. In our case, this means that the variant caller will report
any variants it finds for the L*er* background sample along with the call
statistics for the outcrossed F2 pool at that same site - even if the variant
allele contributes very little to the pool and would, thus, have been
disregarded in a standard single-sample analysis.

Here, we are using the **MiModD Variant Calling** tool for joint variant
calling for our two samples. This tool outputs variant call statistics across
all sites and all samples in the compact binary BCF format. We can then use
the **MiModD Extract Variant Sites** tool to extract sites, for which at least
one sample appears to have a non-wt genotype, and report them in VCF format.

> <hands-on-title>Variant Calling</hands-on-title>
>
> 1. Use **MiModD Variant Calling** {% icon tool %} to generate combined
>    variant call statistics:
>    - Select the **reference genome** to call variants against.
>
>      If your Galaxy server offers version TAIR10 of the *A. thaliana*
>      reference as a **built-in genome**, you can go ahead and use it.
>      Otherwise, download and preprocess the reference as described under
>      Data Preparation and use it as a **genome from my history**.
>    - As the **Aligned reads input datasets** for the analysis select the two
>      BAM datasets representing the two samples (use the `Control` key on the
>      keyboard to select multiple datasets from the list.
>    - **Execute** the job
> 2. Run **MiModD Extract Variant Sites** {% icon tool %} with
>    - **BCF input file** set to the BCF dataset generated in the previous
>      step.
>
> > <question-title></question-title>
> >
> > 1. Approximately how many variants were extracted?
> > 2. Does this number surprise you?
> >
> > > <solution-title></solution-title>
> > >  1. You can click on the extracted variants dataset in your history to
> > >     get more details about it. You should see that the file has an
> > >     estimated 130,000 lines. Since one variant gets reported per line in VCF format (apart from a number
> > >     of header lines), this roughly
> > >     equals the number of variants in the dataset.
> > >  2. The huge number of variants reflects the fact that the L*er*
> > >     mapping strain is really quite diverged from the Col reference
> > >     background that the mutant was generated in and, thus, offers many
> > >     marker variants for mapping.
> > >
> > >     In total, the L*er* genome harbors >400,000 single nucleotide
> > >     variants compared to the Col reference, but to keep the tutorial
> > >     datasets small we only provided reads mapping to 2 out of 5
> > >     chromosomes. In addition, we provided really low-coverage data
> > >     even for those two chromosomes so our analysis will miss many true
> > >     variants and contain many questionable variant calls that may represent
> > >     false-positives. Regardless, an analysis based on such a large overall number
> > >     of variants is robust enough to provide useful results.
> > {: .solution }
> >
> {: .question}
{: .hands_on}


# Linkage Analysis

The extracted variants dataset obtained in the previous step contains
information about thousands of sites that can be used as markers to map the
causative mutation in the F2 pool. We will now analyze the data with the
**MiModD NacreousMap** tool in **Variant Allele Frequency** mapping mode to
identify these markers and look for linkage between them and the causative
mutation.

> <hands-on-title>Linkage Analysis with one set of parentally contributed markers</hands-on-title>
>
> In the **MiModD NacreousMap** {% icon tool %} interface, set
> - **type of mapping analysis to perform** to `Variant Allele Frequency
>   Mapping` and
> - **data source to use** to `VCF file of variants`.
> - As the **input file with variants to analyze** select your extracted
>   variants VCF dataset obtained in the previous step.
> - As the **mapping sample name** use `outcrossed F2`.
> - Leave the **name of the related parent sample** input field empty because
>   in this two-sample there is no such sample.
> - As the **name of the unrelated parent sample** specify `Ler`.
>    > <comment-title>Sample roles</comment-title>
>    > In `Variant Allele Frequency Mapping` mode the tool requires:
>    > - a **mapping sample**, which corresponds to the recombinant pool, from
>    >   which allele frequencies should be calculated
>    > - either a **related** or an **unrelated parent sample** (or both),
>    >   from which the marker variant sites to use in the linkage analysis
>    >   will be determined. A related parent sample is related (or identical)
>    >   to the original mutant line by descent and provides information about
>    >   (some of) the variants present in that line, while an unrelated parent
>    >   sample provides information about the mapping strain variants
>    >   introduced in the mapping cross. If both a related and an unrelated
>    >   parent sample are available, the tool can also use the marker sets
>    >   from both sides for the linkage analysis.
>    >
>    {: .comment}
> With these settings, we base the linkage analysis on crossed-in variants
> only, *i.e.*, the tool will look for variants, for which the L*er* sample
> appears homozygous mutant, and determine the contribution of each such
> variant allele to the recombinant pool highlighting regions of low
> contribution, *i.e.*, with suspected linkage to the causative variant, in its
> output.
>
> Running this job will generate two datasets, one with linkage tables for each
> chromosome in the *A. thaliana* genome, one with graphical representations
> of the data.
>
> > <question-title></question-title>
> >
> > 1. Have a look at the linkage plots dataset generated by the tool run. In
> >    which interval on which chromosome would you expect the causative
> >    mutation?
> > 2. Are the per-marker scatter plots and the histogram plots in agreement?
> > 3. What about those `No markers found!` messages (you saw them, right?)
> >    that the tool printed?
> >
> > > <solution-title></solution-title>
> > > 1. The plots suggest that the causative variant is likely to be found
> > >    in the region from 18,000,000 to 19,000,000 on chr2.
> > > 2. Look at the y axis labels! The scatter plots show the rates at
> > >    which marker sites appear to recombine with the causative mutation
> > >    in the pool of F2 recombinants, while the histograms show linkage
> > >    evidence. Hence, a dip of the scatter plot regression line towards zero
> > >    corresponds to a peak in the histograms and the two kinds of plots are,
> > >    thus, in very good agreement for this analysis.
> > >
> > >    In general, the histogram plots are easier to digest and often
> > >    provide more precise mapping intervals, and are often all you need for
> > >    mapping a causative variant. The per-marker scatter plots, on the other
> > >    hand, are closer to the raw data and can sometimes reveal additional
> > >    irregularities in the data that may indicate problems with the mapping
> > >    cross or unexpected factors influencing the inheritance pattern.
> > > 3. Since we included only reads mapping to chromosome 1 or 2 in
> > >    the original BAM files, the tool will not be able to detect any markers
> > >    for the other chromosomes.
> > {: .solution }
> >
> {: .question}
{: .hands_on}


# Identifying Candidate Mutations

With the mapping results obtained above we can now try to get a list of
candidates for the causative mutation. Here is what we know about this mutation
so far:

* The F2 recombinant pool should appear homozygous mutant for it.

  This is because the F2 recombinants got selected based on the phenotype of
  the recessive mutation so we can assume that all individuals in the pool were
  homozygous for it.

* The L*er* strain should not carry it.

* It should reside in the genomic region identified in the previous step.


> <hands-on-title>Variant Filtering</hands-on-title>
>
> With the above information, we can run **MiModD VCF Filter** {% icon tool %}
> to obtain a new VCF dataset with only those variants from the originally
> extracted ones that fulfill all three criteria. In the tool interface,
>
> - select as **VCF input file** the extracted variants dataset from your
>   history
> - add a **Sample-specific Filter** and set its
>   - **sample** to `outcrossed F2`
>   - **genotype pattern(s) for the inclusion of variants** to `1/1`
>
>   These settings will make the filter retain only variants for which the F2
>   pool has a `1/1`, *i.e.*, homozygous mutant genotype.
> - add another **Sample-specific Filter** and set its
>   - **sample** to `Ler`
>   - **genotype pattern(s) for the inclusion of variants** to `0/0`
>
>   This filter will retain only variants for which the mapping strain sample
>   has a homozygous wt genotype.
> - add a **Region Filter** and set its
>   - **Chromosome** to `chr2`
>   - **Region Start** to 18000000
>   - **Region End** to 19000000
>
>   to retain only variants falling into the previously determined mapping
>   interval.
>
>    > <comment-title>Sample and chromosome names</comment-title>
>    > This is the second time in this tutorial that we ask you to type sample
>    > or chromosome names exactly as they appear in a dataset, and so you
>    > may have started wondering how you are supposed to keep track of which
>    > names are used where in your own analyses.
>    >
>    > Because this is a valid concern, the MiModD tool suite features the
>    > **MiModD File Information** {% icon tool %} tool that can report the
>    > sample and sequence names stored in almost any dataset that you may
>    > encounter in a mapping-by-sequencing analysis (specifically, the
>    > tool can handle datasets in vcf, bcf, sam, bam and fasta format).
>    >
>    {: .comment}
>
> Running the job yields a new VCF dataset with variants that passed all
> filters.
>
> > <question-title></question-title>
> >
> > 1. How many variants passed the filters?
> > 2. How many variants would you have to consider as candidates if you did
> >    not have any mapping information?
> >
> > > <solution-title></solution-title>
> > > 1. If you used the suggested mapping interval of 1Mb on chr2, you
> > >    should have retained 5 variants. Filtering is a really powerful way to
> > >    reduce the number of variants to consider!
> > > 2. Re-running the job with the region filter removed shows that across
> > >    the two chromosomes, for which the input datasets provided sequencing
> > >    reads, there are 50 variants for which the pool appears homozygous
> > >    mutant and for which the L*er* strain is homozygous wt.
> > {: .solution }
> {: .question}
{: .hands_on}

Not all of the retained variants are equally likely candidates for being the
causative mutation. Since the input datasets consist of reads from whole-genome
sequencing, as opposed to exome sequencing, some of the variants may fall in
intergenic regions or introns where they are unlikely to cause a phenotype.
Other variants may fall in coding regions, but may be silent on the protein
level. Hence, we can prioritize our candidate variants after annotating them
with predicted functional effects.

> <hands-on-title>Variant Annotation</hands-on-title>
>
> To annotate our variants we are going to use **SnpEff** with its
> `Arabidopsis_thaliana` genome database as the source of functional genomic
> annotations, then generate a human-friendly variant report with the
> **MiModD Report Variants** {% icon tool %} tool.
>
> 1. Use **SnpEff Download** {% icon tool %} to download genome database
>    `Arabidopsis_thaliana`.
>
>    > <comment-title>SnpEff versions</comment-title>
>    > SnpEff genome databases can only be used with the specific `major.minor`
>    > version of SnpEff they were built for.
>    > Make sure you are using matching versions (e.g., `4.3` - differences
>    > past the second digit are fine and denote subversions with compatible
>    > database schemes) of **SnpEff Download** {% icon tool %} in this and of
>    > **SnpEff Variant effect and annotation** {% icon tool %} in the next
>    > step.
>    > 
>    > Please also note that before SnpEff version 4.3 the TAIR10 version of
>    > the *A. thaliana* genome database was named `athalianaTair10`.
>    > If your Galaxy server offers only older versions of the SnpEff tools,
>    > you will have to use that name instead of `Arabidopsis_thaliana`.
>    {: .comment}
> 2. In the **SnpEff Variant effect and annotation** {% icon tool %} interface,
>    set:
>    - **Sequence changes (SNPs, MNPs, InDels)** to your filtered VCF dataset
>    - **Genome source** to `Downloaded snpEff database in your history`
>    - **SnpEff4.3 Genome Data** to your just downloaded genome dataset
>    - under **Annotation options** check `Use 'EFF' field compatible with
>      older versions (instead of 'ANN')`
>
>      This last setting is required for compatibility of the resulting dataset
>      with the current version of **MiModD Report Variants** {% icon tool %}.
>
>    Running this job will produce another VCF dataset with the functional
>    annotation data added to its INFO column. Inspecting this dataset is not
>    particularly convenient, but you can turn it into a more user-friendly
>    report by
> 3. using **MiModD Report Variants** {% icon tool %} and setting:
>    - **The VCF input with the variants to be reported** to the annotated
>      variants dataset generated by **SnpEff**
>    - the **Format to use for the report** to `HTML`
>    - **Species** to `A. thaliana`.
>
>      The species information can be used by the tool to enrich the output
>      with species-specific database and genome browser links.
>
>    > <comment-title></comment-title>
>    > Variant reports in html format are meant to be used on relatively small
>    > lists of candidate variants like in this example. If the input VCF
>    > contains > 1000 variants, the relatively large size of the reports and
>    > slow loading in the browser may become annoying. In this case, or if you
>    > experience formatting problems with the generated html in your browser,
>    > it may be better to choose `Tab-separated plain text` as the report
>    > format although the output will then lack those convenient hyperlinks.
>    {: .comment}
>
> > <question-title></question-title>
> >
> > The published confirmed mutations in the mutant line are *chr2:18,774,111
> > C→T* and *chr2:18,808,927 C→T*. The latter was shown through follow-up
> > experiments to be the causative variant.
> >
> > 1. What are the functional effects of these two mutations according to the
> >    report?
> > 2. What biological functions are described for the two genes? How do these
> >    relate to the phenotype of our mutant line?
> >
> > > <solution-title></solution-title>
> > > 1. The C→T transition at 18,774,111 results in a premature
> > >    translational stop for the transcript AT2G45550.1. The C→T transition
> > >    at 18,808,927 affects a splice donor site and, thus, is likely to
> > >    interfere with correct splicing of the transcript AT2G45660.1.
> > > 2. From the linked gene pages at *TAIR* you can learn that the
> > >    gene AT2G45550 encodes a cytochrome P450, while AT2G45660 encodes a
> > >    protein called AGL20/SOC1, which is known to control flowering. Thus, there
> > >    is a direct connection between the late-flowering phenotype of the
> > >    mutant line studied here and the known role of the gene affected by
> > >    chr2:18,808,927 C→T.
> > {: .solution }
> {: .question}
{: .hands_on}

# Conclusion


Mapping-by-sequencing can greatly speed up and facilitate the molecular
identification of mutations recovered from mutagenesis screens. The method
makes extensive use of cross-sample comparison of variants and benefits strongly
from joint variant calling to generate reliable and manageable variant
information. A combination of variant linkage analysis and filtering often
results in very small lists of candidate variants that can then be confirmed
through further experimental work. Importantly, the selection of biological
samples for sequencing determines the meaningful variant comparisons that can
be made in the bioinformatics analysis so it is important to understand the
essence of the analysis method **before** the preparation of any biological
samples.


topic: variant-analysis
tutorial: pox-tiled-amplicon
title: Pox virus genome analysis from tiled-amplicon sequencing data
difficulty: advanced
time_estimation: 4H
tools_json: []
requirements_json: []
tags_json: ["virology", "one-health"]
content_hash: 25d02ffdce0a927c6d6d4c0b41aebeaf
last_modified: 2026-02-26T21:12:18.625267
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/pox-tiled-amplicon/tutorial.html

content:
Pox viruses (*Poxviridae*) are a large family of viruses, and members of it have various vertebrate and arthropod species as their natural hosts. The most widely known species in the family are the now extinct variola virus from the genus *orthopoxvirus* as the cause of smallpox, and vaccinia virus, a related, likely horsepox virus, which served as the source for the smallpox vaccine that allowed eradication of that disease.

However, various other pox viruses infect mammalian hosts and cause often severe diseases, and some of them have the ability to jump to humans. Mpox, caused by monkeypox virus, is the most recent and prominent example demonstrating the zoonotic potential of *orthopoxviruses*. The sister genus *capripoxvirus* is a major concern in livestock farming with all its three species - sheeppox virus (SPPV), goatpox virus (GTPV), and lumpy skin disease virus (LSDV) causing severe disease [in sheep and goats](https://www.woah.org/en/disease/sheep-pox-and-goat-pox/) (SPPV, GTPV) and [in cattle](https://www.woah.org/en/disease/lumpy-skin-disease/) (LSDV).

All pox viruses are characterized by a large double-stranded DNA genome (genome sizes of currently sequenced species range from 135 to 360kb) that gets replicated exclusively in the cytoplasm of host cells.

The ends of pox virus genomes consist of inverted terminal repeats (ITRs), i.e. regions of exact reverse-complementarity at the 5'- and 3'-end of the genome. At the very tips of the genome, these regions contain tandem repeats and terminal bases that are cross-linked to the opposite genome strand as first described for vaccinia virus and reviewed in {% cite Wittek1982 %}.

There is considerable length variation in the ITR regions of different species. For example, Capripox virus genomes have ITR lengths of 2.2 - 2.4 kb, monkeypox virus has 6.4 kb long ITRs, and variola virus had the shortest known
ITRs with a length of just 700 bases (reviewed in {% cite Haller2014 %}).

Independent of their size, ITRs pose a problem for the analysis of high-throughput sequencing data of pox viruses. Their repetitive nature can confuse de-novo assembly algorithms, but is, to some extent, also a challenge during reference-based mapping, particularly at the boundary between ITRs and the central region of the genome and with samples less related to the reference.
  
Recently, a tiled-amplicon approach with separate sequencing of half-genomes has been developed for *capripoxviruses* ({% cite Mathijs2022 %}) that makes it possible to assign ITR-derived reads unambiguously to the 5'- or 3'-end of a pox virus genome when combined with a suitable bioinformatics pipeline.
  
This tutorial demonstrates how such data can be analyzed correctly with Galaxy.

> <comment-title>Advanced tutorial</comment-title>
> This tutorial demonstrates a very close to real-world analysis of viral sequencing data with Galaxy[^prior_work].
> As such, it avoids many simplifications that are often made in training material to ease understanding.
> On the technical side, in particular, it makes use of many different Galaxy features and often assumes more than just basic familiarity with them.
> Like in other tutorials, you will find precise instructions for running every step in the analysis, but explanations focus very much on the biology of pox viruses and will often not offer much details on *why* a certain component of Galaxy is used the way it is described.
>
> To get the most out of this tutorial, you should have worked through several other parts of the training material, which are listed under **Requirements** above, in particular:
> - If you are not yet familiar with
>   - some typical ways of getting data into Galaxy,
>   - dataset collections and ways to manipulate them
>   - importing and running workflows,
>
>   you may benefit more from this tutorial if you first follow some of the introductory tutorials on these topics.
>
>   The tutorial also uses the rule-based uploader, which is an advanced feature, and treats it very much like a black box, which somehow lets you structure data exactly the way it is needed for the analysis. If you want to learn how to use this powerful feature for your own purposes, please follow the dedicated tutorials [Rule Based Uploader]({% link topics/galaxy-interface/tutorials/upload-rules/tutorial.md %}) and [Rule Based Uploader Advanced]({% link topics/galaxy-interface/tutorials/upload-rules-advanced/tutorial.md %}).
>
> - If this is your first exposure to sequencing data analysis, our more basic tutorials on quality control and mapping of such data are highly recommended to get you familiar with general concepts, data types, etc.
>
{: .comment}  

The following figure provides an overview of the wet lab data generation process described in {% cite Mathijs2022 %} and of the flow of the analysis that we will perform in this tutorial.

![A schematic drawing of a pox virus genome with a central coding region flanked by inverted terminal repeat regions. Below the genome 23 tiled amplicons are drawn as numbered lines to show their approximate distribution across the genome. Below the amplicons, the main steps in their wet lab processing and of the bioinformatics analysis of their sequences is shown as a flowchart](../../images/pox-virus-tiled-amplicon/pox-tiled-amplicon-wf.png "Tiled-amplicon approach with mapping to half-genomes")

On the wet lab side, the genome of a viral sample gets amplified in the form of 23 overlapping amplicons using a set of pan-genus primers suitable for all species of *capripoxvirus*.

Amplification happens in four separate multiplex PCR reactions: the primers for the odd-numbered amplicons (1-11) from the 5' amplicon pool are used in reaction 1a, the primers for the even-numbered amplicons (2-12) in reaction 1b, the odd-numbered amplicons (13-23) from the 3' amplicon pool in reaction 2a and the even numbered amplicons (12-22) from that pool in reaction 2b.

Note that the 5'-ITR gets amplified as part of amplicon 1, while the 3'-ITR is part of amplicon 23. Amplification of the two regions is specific because the right primer of amplicon 1 and the left primer of amplicon 23 both bind in the "central coding region", i.e. outside the reverse-complementary ITRs.

After amplification, reactions 1a and 1b, and reactions 2a and 2b are combined to form the two sequencing pools, "pool 1" and "pool 2". Note that amplicon 12 becomes part of both tools in the published protocol because overlap between the two sequencing pools is required for assembly-based analysis. For the mapping-based approach shown in this tutorial the overlap is not essential.

The DNA from the two pools is subjected separately to Illumina sequencing. This way, reads obtained from sequencing of "pool 1" are known to be derived from the 5'-half of the genome, while reads obtained from sequencing of "pool 2" are known to originate from the 3'-half.

In the tutorial analysis, we will exploit this information and map the "pool 1" data to a modified reference genome, in which the 3'-half of the sequence has been replaced with Ns. This masking prevents the read mapper from placing ambiguous reads (in particular reads derived from the 5'-ITR region) onto the 3'-half of the genome.
Likewise, we will use a 5'-masked genome for the mapping of the "pool 2" reads, again preventing erroneous mapping of reads that we know must be derived from the 3'-half of the genome.

Once the correct alignments of the reads from the two pools have been recorded, we can safely merge the two results, and use that merged data to construct a consensus sequence for the sample.

Compared to a standard tiled-amplicon approach involving just one sequencing pool, the half-genome sequencing approach is more accurate, especially at the boundaries between the "central coding region" and the ITRs, but the analysis is complicated by the doubled number of inputs, by the need to generate masked reference sequences and by the extra step of merging two read mapping results.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare analysis history and data

Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Prepare the Galaxy history</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md name="pox-tiled-amplicon tutorial" %}
>
{: .hands_on}

## Get reference and sequenced samples data

In this tutorial you are going to work on the sequencing data from two samples of lumpy skin disease virus that have been sequenced as amplified half-genomes in Illumina paired-end mode. This means that we will be dealing with a forward and a reverse read dataset per sample and amplified half-genome, and so will have to download a total of eight sequenced reads datasets for our analysis.

The samples are the first two (`SAMN20238786` and `SAMN20238785`) from BioProject `PRJNA746718` as found through the [European Nucleotide Archive browser](https://www.ebi.ac.uk/ena/browser/view/PRJNA746718).

If you follow the link, you should see that
- for each of the two samples there is data from two sequencing runs (representing the two sequenced half-genome pools of amplicons).

  If it isn't shown by default, enable display of the *Library Name* column by clicking on *Show Column Selection* and selecting the checkbox for *library_name*. The pool identity (`1` or `2`) is provided as the last part of each sequencing run's library name.

- for each pool the data consists of two downloadable `fastq.gz` results files, which represent the forward and reverse reads for that sequencing run.

> <comment-title>Choice of samples</comment-title>
> An in-depth analysis ({% cite Vandenbussche2022 %}) of the samples from BioProject PRJNA746718 has revealed that their genomes are the result of recombination between at least two different parent strains.
> So not only are these samples sequenced following our wet lab protocol of choice, but there is also some prior knowledge about their (rather interesting) biology.
> This gives us the opportunity to evaluate the correctness of our analysis by comparing the results we will generate in this tutorial to the published findings.
>
{: .comment}

In addition, we will require:
- a **reference genome** to map the sequenced reads to

  We will be using the public lumpy skin disease virus reference sequence [NC_003027.1](https://www.ncbi.nlm.nih.gov/nuccore/NC_003027.1/) from the [NCBI RefSeq](https://www.ncbi.nlm.nih.gov/refseq/) database here.

- a **primer scheme file** with information about
  - which amplicons were present in which half-genome sequencing pool
  - where each primer binds with respect to the reference genome
  
  This is a custom file, which you may have to modify, when analysing your own data, depending on the actual primer scheme used during sample preparation, and which we will just copy/paste into Galaxy. The version we are using describes the binding sites of the 46 primers published by ({% cite Mathijs2022 %}) with respect to the NC_003027.1 reference sequence.

> <hands-on-title>Get the data</hands-on-title>
>
> 1. Import all **pool 1** sequenced reads files of the two samples to analyze into your history.
>
>    You could collect the download links and additional metadata from the ENA page for `PRJNA746718`, but for simplicity lets just use this precompiled combined information:
>
>    ```
>    20L81    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/074/SRR15145274/SRR15145274_1.fastq.gz
>    20L81    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/074/SRR15145274/SRR15145274_2.fastq.gz
>    20L70    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/076/SRR15145276/SRR15145276_1.fastq.gz
>    20L70    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/076/SRR15145276/SRR15145276_2.fastq.gz
>    ```
>
>    and upload them as a *List of Pairs* collection.
>
>    1. Copy the links above
>    2. Open the {% tool [Upload](upload1) %} Manager
>    3. In the top row of tabs select **Rule-based**
>    4. Set **Upload data as** to `Collection(s)`
>    5. Paste the copied links into the text field on the right
>    6. Click **Build** to bring up the *Rule Builder* dialog
>    7. Click on the wrench icon {% icon tool %} next to **Rules** in the left panel of the window
>    8. In the text field, erase the prefilled content and copy/paste the following rule definition into it instead:
>
>       ```
>       {
>         "rules": [
>           {
>             "type": "add_column_regex",
>             "target_column": 1,
>             "expression": ".+_(\\d)\\.fastq\\.gz",
>             "group_count": 1,
>             "replacement": null
>           }
>         ],
>         "mapping": [
>           {
>             "type": "url",
>             "columns": [1]
>           },
>           {
>             "type": "list_identifiers",
>             "columns": [0],
>             "editing": false
>           },
>           {
>             "type": "paired_identifier",
>             "columns": [2]
>           }
>         ]
>       }
>       ```
>    9. Click on **Apply**
>    10. In the bottom row of options, change **Type** from "Auto-detect" to `fastqsanger.gz`
>    11. Set the name of the new collection to `pool1 reads`
>    12. Click on **Upload**
>
>    It is ok to continue with the next step even while the upload of the pool 1 reads is still ongoing.
>    Just click on the **Close** button of the information pop-up window if it is still open when you are ready to proceed.
> 2. {% tool [Upload](upload1) %} the **pool2** sequenced reads files of the same samples using the corresponding metadata:
>
>    ```
>    20L81    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/073/SRR15145273/SRR15145273_1.fastq.gz
>    20L81    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/073/SRR15145273/SRR15145273_2.fastq.gz
>    20L70    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/075/SRR15145275/SRR15145275_1.fastq.gz
>    20L70    ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR151/075/SRR15145275/SRR15145275_2.fastq.gz
>    ```
>
>    - Follow the same steps as for the pool 1 reads to get the data into your history.
>    - The rule definition used for the first pool will work unchanged for the second.
>    - Just make sure to choose `pool2 reads` as the name of the new collection.
>
>    As before, it is ok to proceed to the next step while the data upload is still ongoing.
>
> 3. Tag the uploaded collections of **pool1** and **pool2** reads
>
>    For each of the two collections
>
>    1. Click on the collection in the history panel to step into it
>    2. Click on the {% icon galaxy-pencil %} icon next to the collection name
>    3. Click on *"Add Tags"*
>    4. Use `#pool1` and `#pool2` as the tag for the pool1 and the pool2 collection of reads, respectively
>    5. Click on {% icon galaxy-save %} **Save**
>    6. Check that the tag appears below the collection name
>
>    Parts of the analysis in this tutorial will consist of identical steps performed on the data of each pool.
>    To make it easier to distinguish which analysis step has been conducted with which pool, we will be using name tags.
>    These are tags starting with `#`, and they will automatically propagate to new datasets and collections derived from the tagged collections.
>
> 4. Get the `NC_003027.1` LSDV reference.
>
>    A convenient public download link for this sequence is best obtained from the ENA again, where the sequence is known under its [INSDC](https://www.insdc.org/) alias [AF325528.1](https://www.ebi.ac.uk/ena/browser/view/AF325528.1):
>
>    ```
>    https://www.ebi.ac.uk/ena/browser/api/fasta/AF325528.1?download=true
>    ```
>
>    1. {% tool [Upload](upload1) %} the reference to your history via the link above and make sure the dataset format is set to `fasta`.
>
>       {% snippet faqs/galaxy/datasets_import_via_link.md format="fasta" %}
>    2. {% tool [Replace Text in entire line](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2) in entire line %} to simplify the reference sequence name
>       - {% icon param-file %} *"File to process"*: the uploaded reference sequence from the ENA
>       - In {% icon param-repeat %} *"1. Replacement"*:
>         - *"Find pattern"*: `^>.+`
>         - *"Replace with"*: `>AF325528.1`
>    3. When the Replace Text tool run is finished, **rename** the output dataset
>
>       {% snippet faqs/galaxy/datasets_rename.md name="LSDV reference" format="fasta" %}
>
> 5. {% tool [Upload](upload1) %} the primer scheme information in [BED](http://www.genome.ucsc.edu/FAQ/FAQformat.html#format1) format
>
>    ```
>    NC_003027.1    235       264       CaPV-V1_1_LEFT      pool1a    +
>    NC_003027.1    6789      6819      CaPV-V1_2_LEFT      pool1b    +
>    NC_003027.1    7771      7801      CaPV-V1_1_RIGHT     pool1a    -
>    NC_003027.1    13515     13543     CaPV-V1_3_LEFT      pool1a    +
>    NC_003027.1    14451     14482     CaPV-V1_2_RIGHT     pool1b    -
>    NC_003027.1    20192     20219     CaPV-V1_4_LEFT      pool1b    +
>    NC_003027.1    21201     21229     CaPV-V1_3_RIGHT     pool1a    -
>    NC_003027.1    26787     26814     CaPV-V1_5_LEFT      pool1a    +
>    NC_003027.1    27786     27814     CaPV-V1_4_RIGHT     pool1b    -
>    NC_003027.1    33277     33305     CaPV-V1_6_LEFT      pool1b    +
>    NC_003027.1    34363     34395     CaPV-V1_5_RIGHT     pool1a    -
>    NC_003027.1    39894     39921     CaPV-V1_7_LEFT      pool1a    +
>    NC_003027.1    40867     40897     CaPV-V1_6_RIGHT     pool1b    -
>    NC_003027.1    46393     46420     CaPV-V1_8_LEFT      pool1b    +
>    NC_003027.1    47419     47453     CaPV-V1_7_RIGHT     pool1a    -
>    NC_003027.1    52864     52890     CaPV-V1_9_LEFT      pool1a    +
>    NC_003027.1    54039     54069     CaPV-V1_8_RIGHT     pool1b    -
>    NC_003027.1    59439     59472     CaPV-V1_10_LEFT     pool1b    +
>    NC_003027.1    60530     60559     CaPV-V1_9_RIGHT     pool1a    -
>    NC_003027.1    66008     66041     CaPV-V1_11_LEFT     pool1a    +
>    NC_003027.1    66960     66989     CaPV-V1_10_RIGHT    pool1b    -
>    NC_003027.1    72534     72560     CaPV-V1_12_LEFT     pool1b    +
>    NC_003027.1    72534     72560     CaPV-V1_12_LEFT     pool2b    +
>    NC_003027.1    73568     73601     CaPV-V1_11_RIGHT    pool1a    -
>    NC_003027.1    79081     79111     CaPV-V1_13_LEFT     pool2a    +
>    NC_003027.1    80176     80202     CaPV-V1_12_RIGHT    pool1b    -
>    NC_003027.1    80176     80202     CaPV-V1_12_RIGHT    pool2b    -
>    NC_003027.1    85743     85777     CaPV-V1_14_LEFT     pool2b    +
>    NC_003027.1    86753     86781     CaPV-V1_13_RIGHT    pool2a    -
>    NC_003027.1    92208     92232     CaPV-V1_15_LEFT     pool2a    +
>    NC_003027.1    93323     93351     CaPV-V1_14_RIGHT    pool2b    -
>    NC_003027.1    98974     99002     CaPV-V1_16_LEFT     pool2b    +
>    NC_003027.1    99970     100000    CaPV-V1_15_RIGHT    pool2a    -
>    NC_003027.1    105634    105662    CaPV-V1_17_LEFT     pool2a    +
>    NC_003027.1    106584    106612    CaPV-V1_16_RIGHT    pool2b    -
>    NC_003027.1    112082    112108    CaPV-V1_18_LEFT     pool2b    +
>    NC_003027.1    113111    113144    CaPV-V1_17_RIGHT    pool2a    -
>    NC_003027.1    118617    118649    CaPV-V1a_19_LEFT    pool2a    +
>    NC_003027.1    119649    119675    CaPV-V1_18_RIGHT    pool2b    -
>    NC_003027.1    125361    125390    CaPV-V1_20_LEFT     pool2b    +
>    NC_003027.1    126297    126328    CaPV-V1_19_RIGHT    pool2a    -
>    NC_003027.1    131713    131742    CaPV-V1_21_LEFT     pool2a    +
>    NC_003027.1    132961    132988    CaPV-V1_20_RIGHT    pool2b    -
>    NC_003027.1    138437    138467    CaPV-V1_22_LEFT     pool2b    +
>    NC_003027.1    139333    139365    CaPV-V1_21_RIGHT    pool2a    -
>    NC_003027.1    143117    143145    CaPV-V1_23_LEFT     pool2a    +
>    NC_003027.1    146188    146219    CaPV-V1_22_RIGHT    pool2b    -
>    NC_003027.1    150505    150538    CaPV-V1_23_RIGHT    pool2a    -
>    ```
>
>    Since BED is a tabular format, but the above display uses spaces to separate columns, please make sure, you have **Convert spaces to tabs** checked when creating the dataset from the copied content!
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md name="LSDV primer scheme" format="bed" convertspaces="true" %}
>
{: .hands_on}

At this point you should have these four items in your history:

1. The collection of pool 1 reads (representing the 5'-half of the genome)
   - This should be a list of two elements (representing the two samples we are going to analyze).
   - Each of these elements should itself be a paired collection of a *forward* and a *reverse* reads dataset in `fastqsanger.gz` format.
   - A `#pool1` name tag should be attached to the collection in the history.

2. The collection of pool 2 reads (representing the 3'-half of the genome)
   - This should be structured exactly as the first collection.
   - Also the names of the elements in both collections should be identical.
   - A `#pool2` name tag should be attached to the collection in the history.

3. The LSDV reference as a single *fasta* dataset
4. The primer scheme as a single *bed* dataset

If this is the case for you, you are ready to start the actual data analysis.

# Generating masked reference genome versions for mapping

As illustrated in Figure 1, we would like to map the "pool 1" reads of each sample to a version of the LSDV reference, in which the 3'-half of the genome (starting after the right end of amplicon 12, the 3'-most amplicon still contained in pool 1) is masked (i.e., bases are replaced by Ns). This ensures that the read mapper will only consider the 5'-half of the genome when placing reads from pool 1.
Likewise, for the "pool2" read mapping, we would like to use a reference version, in which the 5'-half that is not covered by the second amplicon pool is masked out.

To build these two masked reference versions we need to extract the information about amplicon pool boundaries from the primer scheme BED dataset and use it twice on the LSDV reference to replace the appropriate ranges of its bases with Ns.

As this is a complex task that would take a considerable amount of time to perform step-by-step via manual tool runs, we are providing a pre-built workflow for you that you only need to import and run on the input data in your history.

> <hands-on-title>Import and run the reference masking workflow</hands-on-title>
>
> 1. Go to the training material [page for this workflow](workflows/) and use one of the options listed there to import the workflow into your account on your Galaxy server.
> 2. Back on the list of workflows page, click the *Run workflow* icon {% icon workflow-run %} to the right of the newly imported workflow.
> 3. On the next page, select
>    - *"Primer Scheme (BED file with Pool identifiers)"*: the uploaded ``LSDV primer scheme`` BED dataset
>    - *"Reference FASTA"*: the ``LSDV reference`` fasta dataset generated with **Replace Text**
>
>    Running the workflow will produce two key output datasets:
>    - a *Masked reference for mapping of pool1*
>    - a *Masked reference for mapping of pool2*
>
> 4. Check whether the workflow produced correctly masked half-genomes
>
>    You can {% icon galaxy-eye %} inspect the two masked reference datasets by eye, which quickly reveals that one sequence has its beginning N-masked, while for the other it is the end, but it is hard to find out whether the regions are exactly right this way. Luckily, we can use a tool to determine the exact extent of Ns for each dataset:
>    1. {% tool [Fasta regular expression finder](toolshed.g2.bx.psu.edu/repos/mbernt/fasta_regex_finder/fasta_regex_finder/0.1.0) %}
>       - {% icon param-files %} *"Input"*: select the two masked reference datasets; output of the masking workflow run
>       - *"Regular expression"*: `N+`
>
>         This looks for stretches of one or more Ns.
>       - In *"Specify advanced parameters"*:
>         - *"Do not search the reverse complement"*: `Yes`
>
>           We are only interested in the forward strand (and an N is an N on both strands anyway) so we can save some compute by turning on this option.
>         - *"Maximum length of the match to report"*: `1`
>
>           It would be boring to have the matched stretches of Ns written out again.
>
>    > <question-title></question-title>
>    >
>    > Inspect the two result datasets created by the tool and compare the regions of Ns reported in them to the primer scheme.
>    > Did the correct regions get masked?
>    >
>    > Tip: Galaxy's {% icon galaxy-scratchbook %} Window Manager, which you can enable (and disable again) from the menu bar can be very helpful for comparing multiple datasets.
>    >
>    > > <solution-title>Answer</solution-title>
>    > >
>    > > The masked reference for mapping of pool1 has Ns from 80,202 - 150,773.
>    > > The 3'-most primer contained in pool1, CaPV-V1_12_RIGHT, has its binding site listed as 80,176 - 80,202.
>    > > Bed format specifies zero-based half-open intervals, i.e. the first base of a sequence has position 0, and the end position of any interval is the first base *not* included in it!
>    > > So the 3'-most base covered by the primer is base number 80,201 (when counting from 0) and the reference is masked correctly from 80,202 onwards.
>    > >
>    > > The masked reference for mapping of pool2 has Ns from 0 - 72,534.
>    > > The 5'-most primer of pool2 is CaPV-V1_12_LEFT and the primer scheme lists the start of its binding site as 72,534.
>    > > Again, the way bed format is specified means that position 72,534 is the first base *not* masked in the reference for pool2 mapping, and that is correct.
>    > > Also note that amplicon 12 is contained in both pools and the masking takes this fact into account. 
>    > >
>    > > Feel free to open the workflow in the workflow editor and inspect the steps it took to work out the masking, but it's also ok to treat this part as a black box and move on. There is still a lot of analysis that is waiting to be done.
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# Quality control of the sequencing data of all samples

The next step is to check the quality of the sequencing data of both samples and to eliminate, if necessary, poor-quality reads or parts of them.

The tool **fastp** lets us perform these tasks and obtain a nice quality report for our reads before and after processing in one go, but many other options exist to perform sequenced reads quality control and trimming/filtering in Galaxy, and the dedicated tutorial on [quality control]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}) introduces more of them.

> <hands-on-title>QC and read trimming/filtering with fastp</hands-on-title>
>
> 1. {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.23.2+galaxy0) %} on the collection of **pool1** reads
>    - *"Single-end or paired reads"*: `Paired Collection`
>    - *"Select paired collection(s)"*: the uploaded collection of **pool1** reads
>    - In *"Filter Options"*
>      - In *"Length filtering options"*
>        - *"Length required"*: `30`
>
>    > <details-title>fastp default settings</details-title>
>    >
>    > fastp comes with basic default settings for some of its many parameters.
>    > In particular, with no explicit parameter specifications, the tool will still remove reads that
>    > - have base qualities < 15 for more than 40% of their bases
>    > - have more than 5 Ns in their sequence
>    > - have a length < 15 bases (after removing poly-G tails from Illumina reads)
>    >
>    >   Because our input data has far longer MiSeq-generated reads, we can increase this length threshold to 30 bases as done above.
>    >
>    > These are very relaxed criteria that will only remove the least usable reads, but still assure minimal
>    > quality standards that are good enough for many (including training) purposes.
>    {: .details}
>
>    Running the tool will result in two outputs. One is a new paired collection with the processed reads, the other one is a report of initial quality, the processing actions performed and their effect on key quality metrics.
>
> 2. {% icon galaxy-eye %} **Inspect** the reports for both samples and try to answer the following questions:
>
>    > <question-title>Questions</question-title>
>    >
>    > 1. After filtering, which of the two samples has more reads retained?
>    > 2. Which sample had more reads filtered out?
>    > 
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. Sample 20L81 has more reads retained (947,000) than 20L70 (856,000).
>    > > 2. Sample 20L81 also had more reads filtered out.
>    > >    "Only" 93.2% of its original reads got retained compared to 99.1% for 20L70.
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. Now repeat the steps above for the collection of pool 2 reads.
>
>    {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.23.2+galaxy0) %} on the collection of **pool2** reads
>    - *"Single-end or paired reads"*: `Paired Collection`
>    - *"Select paired collection(s)"*: the uploaded collection of **pool2** reads
>    - In *"Filter Options"*
>      - In *"Length filtering options"*
>        - *"Length required"*: `30`
>
>    > <question-title>Questions</question-title>
>    > 1. Are the results for the pool 2 sequences similar to those of pool 1?
>    > 2. What's your overall conclusion regarding the quality of the sequences and the effect of filtering?
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. Results are rather similar although for pool2 the metrics for the two samples are even more similar than for poo1, and it is now 20L70, which has a bit more reads retained after filtering.
>    > > 2. The input sequences are of rather good quality and the relaxed filtering criteria we have configured had little impact. This is not a bad thing though and we can proceed with our analysis.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# Mapping to the masked reference genomes

We are now ready to map each pool of each sample to the corresponding masked reference. We will use the tool BWA-MEM with essentially default settings for this purpose. As shown in figure 1, however, we want to merge the mapping results afterwards, but still be able to tell mapped reads from pool 1 and pool 2 apart. To this end, we need to tag reads with `pool1` and `pool2` read group identifiers, respectively, at the mapping step. Just because a sample has pool1 and pool2 reads it is still just one sample though, and we are going to express that fact through a common sample name (which we let BWA-MEM autodetect as the name of the collection elements).

> <hands-on-title>Read mapping and quality control</hands-on-title>
>
> 1. {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2) %} for **pool1** data
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>      - {% icon param-file %} *"Use the following dataset as the reference sequence"*: Masked reference for mapping of **pool1**
>    - *"Single or Paired-end reads"*: `Paired Collection`
>    - *"Select a paired collection"*: the `#pool1` collection of trimmed and filtered reads; output of **fastp** run on the pool1 data
>    - *"Set read groups information?"*: `Set read groups (SAM/BAM specification)`
>
>      for *Read group identifier*:
>      - *"Auto-assign"*: `No`
>      - *"Read group identifier"*: `pool1`
>
>      for *Read group sample name*:
>      - *"Auto-assign"*: `Yes`
>
>        This interprets names of collection elements as sample names.
>
> 2. {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2) %} for **pool2** data
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>      - {% icon param-file %} *"Use the following dataset as the reference sequence"*: Masked reference for mapping of **pool2**
>    - *"Single or Paired-end reads"*: `Paired Collection`
>    - *"Select a paired collection"*: the `#pool2` collection of trimmed and filtered reads; output of **fastp** run on the pool2 data
>    - *"Set read groups information?"*: `Set read groups (SAM/BAM specification)`
>
>      for *Read group identifier*:
>      - *"Auto-assign"*: `No`
>      - *"Read group identifier"*: `pool2`
>
>      for *Read group sample name*:
>      - *"Auto-assign"*: `Yes`
>
{: .hands_on}

# Merging the mapped reads of both pools

We have mapped both pool 1 and pool 2 reads to masked versions of the same reference sequence. This means that any mapping in any of the separate BWA-MEM outputs also describes a valid mapping to the complete, unmasked reference so all we need to do is merge the pool 1 and pool 2 mapped reads datasets of each sample into one to obtain complete mapping results for the entire genome of the sample.

A tool that can perform this task for us is **samtools merge**. However, if we gave this tool our collection of pool1 reads with one BAM dataset per sample and the collection of pool2 reads with again one BAM dataset per sample, you would be very disappointed to see that it would merge all datasets into one, i.e. it would not only combine pools, but also sample data, which is not what we want.

Why does **samtools merge** in Galaxy behave like this? Well, the tool is written so that it knows how to merge multiple datasets *and* it knows how to merge the contents of simple collections - and it's just too eager to apply that knowledge. What we can do, however, is try and pass it something it does *not* know how to handle in a single run, and that's a nested collection. So instead of providing the input as two separate collections (one for pool1 and one for pool2 data) that each hold two elements (representing data from one sample), we want to provide a collection of two samples, each one consisting of a collection of pool 1 and pool 2 reads, in turn.
When we do this, Galaxy will recognize that the tool cannot handle this, but that it would know how to handle each outer element (sample) separately (as a simple collection). In this situation, Galaxy will automatically schedule two parallel excutions of the tool for us, each one receiving one element of the outer collection as input.

We can do the necessary restructuring in two steps, then pass the result to **samtools merge**.

> <hands-on-title>Reorganize the data and merge across pools within samples</hands-on-title>
>
> 1. {% tool [Zip collections](__ZIP_COLLECTION__) %} with the following parameters
>    - {% icon param-collection %} *"Input 1"*: mapped reads `#pool1`; output of **BWA-MEM** run on the pool1 data
>    - {% icon param-collection %} *"Input 2"*: mapped reads `#pool2`; output of **BWA-MEM** run on the pool2 data
>
>    This combines the two separate collections into one nested structure, but, unfortunately, that structure is not organized the way we want it because **Zip collections** creates lists of pairs, i.e. list collections that hold paired collections inside. Now we still need to turn the list of pairs into a list of lists.
>
> 2. {% tool [Apply rules](__APPLY_RULES__) %} with the following parameters
>    - *"Input Collection"*: zipped collection of mapped reads from both pools
>    - Under *"Rules"*
>      - {% icon galaxy-wf-edit %} **Edit**
>    - In the *"Build Rules for Applying to Existing Collection"* window
>      1. Click {% icon tool %} next to **Rules**
>      2. Erase the prefilled rules description
>      3. Copy the rules below and paste them into the empty rules description area
>
>         ```
>         {
>           "rules": [
>             {
>               "type": "add_column_metadata",
>               "value": "identifier0"
>             },
>             {
>               "type": "add_column_metadata",
>               "value": "identifier1"
>             }
>           ],
>           "mapping": [
>             {
>               "type": "list_identifiers",
>               "columns": [
>                 0,
>                 1
>               ],
>               "editing": false
>             }
>           ]
>         }
>         ```
>      4. Click **Apply**
>      5. Click **Save**, then run the tool
>
>    This will reorganize the data once more, this time into the desired structure, which will give us the correct merge result.
>
> 3. {% tool [Samtools merge](toolshed.g2.bx.psu.edu/repos/iuc/samtools_merge/samtools_merge/1.15.1+galaxy0) %}
>    - {% icon param-collection %} *"Alignments in BAM format"*: the reorganized collection; output of **Apply rules**
>    - *"Make @RG headers unique"*: `No`
>    - *"Make @PG headers unique"*: `Yes`
>
> 4. {% icon galaxy-eye %} Now lets confirm that things have worked as expected:
>
>    > <question-title>Questions</question-title>
>    >
>    > 1. What are the datasets inside the **Samtools merge** collection?
>    > 2. {% icon galaxy-eye %} Display each of the datasets. What information about read groups and samples is stored in each dataset?
>    > 
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. The collection consists of two BAM datasets, one for sample 20L81 and one for 20L70.
>    > > 2. The header of the 20L81 dataset lists
>    > >
>    > >    ```
>    > >    @RG ID:pool1 SM:20L81 PL:ILLUMINA
>    > >    @RG ID:pool2 SM:20L81 PL:ILLUMINA
>    > >    ```
>    > >
>    > >    , which declares that the mapped reads in this file come from two read groups with the IDs pool1 and pool2 that belong to the same sample (SM), 20L81, sequenced on the Illumina platform.
>    > >
>    > >    The 20L70 dataset contains analogous lines in its header declaring the two read groups of the other sample.
>    > >
>    > >    When scrolling to the very right in the body of the datasets, you can see that a RG tag of the form RG:ID (where ID is one of the alternatives declared in the header) is attached to each mapped read line (because the dataset is sorted by mapped coordinates and pool2 reads should have been mapped only to the 3'-half of the genome, you will have to scroll down rather far in the dataset to find a RG tag with a pool2 ID value).
>    > >
>    > >    All this read group information in the dataset headers and on the mapped reads has been put there by BWA-MEM and Samtools merge has, apparently, merged the reads and the read group information correctly!
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


# Mapped reads filtering, trimming, and QC

So now that we have one dataset of mapped reads per sample, we will do some cleaning and quality control on the mapped data to be sure we pass only reliable mapping data into consensus calling.

We will first use **samtools view** to discard any alignments that we are not fully confident in.

Next, we will run **ivar trim** on the retained reads. This tool can detect primer sequences at the ends of the mapped reads that represent the tiled amplicon primers used during preparation of the samples for sequencing, and will mask them so that they will not be considered during generation of the consensus sequence.

> <details-title>Primer trimming</details-title>
>
> The primers used for the tiled-amplicon generation will get incorporated physically at the start and end of each amplicon. These ends, therefor, do *not* represent amplified sequence from the sample genome, but merely reflect the primer sequences. Consequently, the amplicon ends will never capture any mutations in the genome region they cover, and we cannot meaningfully use them for building a consensus sequence for the sample.
>
> Luckily, since we are dealing with *tiled* (i.e., overlapping) amplicons, each amplicon end should be overlapped by the non-primer part of another amplicon, and we can use these sequences to obtain information about the sequenced sample at primer binding sites. By masking parts of reads that are derived from incorporated primers, we make sure these "pseudo"-sequences are not interfering with mutation detection based on truly informative reads, and that we can faithfully reconstruct the genome of the sample.
>
{: .details}

Finally, we will use **Qualimap BamQC** to generate a quality report on the trimmed and filtered mapped reads, which we'll inspect to judge whether read mapping worked well enough for consensus generation purposes.

> <hands-on-title>Mapped reads filtering and trimming and quality control for mappings</hands-on-title>
>
> 1. {% tool [Samtools view](toolshed.g2.bx.psu.edu/repos/iuc/samtools_view/samtools_view/1.15.1+galaxy0) %} for filtering the mapped reads
>    - {% icon param-collection %} *"SAM/BAM/CRAM data set"*: collection of merged mapped reads; output of **Samtools merge**
>    - *"What would you like to look at?"*: `A filtered/subsampled selection of reads`
>    - In *"Configure filters"*:
>      - *"Filter by quality"*: `20`
>      - *"Require that these flags are set"*: `Read is paired`
>      - *"Exclude reads with any of the following flags set"*: `Read is unmapped` and `Mate is unmapped` and `Alignment of the read is not primary` and `Alignment is supplementary`
>
> 2. {% tool [ivar trim](toolshed.g2.bx.psu.edu/repos/iuc/ivar_trim/ivar_trim/1.4.2+galaxy0) %} for trimming of amplicon primers from mapped reads
>    - {% icon param-collection %} *"BAM file"*: collection of filtered mapped reads BAM dataset; output of **Samtools view**
>    - *"Source of primer information"*: `History`
>      - *"BED file with primer sequences and positions"*: the uploaded `LSDV primer scheme` BED dataset
>    - *"Filter reads based on amplicon info"*: `Yes, drop reads that extend beyond amplicon boundaries`
>    - *"Include reads not ending in any primer binding sites?"*: `Yes`
>    - *"Require a minimum length for reads to retain them after any trimming?"*: `Yes, and provide a custom threshold`
>      - *"Minimum trimmed length threshold"*: `30
>
> 3. {% tool [QualiMap BamQC](toolshed.g2.bx.psu.edu/repos/iuc/qualimap_bamqc/qualimap_bamqc/2.2.2d+galaxy3) %} for mapped reads quality control
>    - {% icon param-collection %} *"Mapped reads input dataset"*: filtered and trimmed mapped reads BAM dataset; output of **ivar trim**
>    - *"Skip duplicate reads"*: `Deselect all`
>    - In *"Settings affecting specific plots"*:
>      - *"Number of bins to use in across-reference plots"*: `200`
>
> 4. {% icon galaxy-eye %} Study the report generated with QualiMap
>
>    > <question-title>Questions</question-title>
>    >
>    > 1. What is the mean coverage for each of the samples across the entire genome?
>    > 2. Which sample might be a bit more problematic coverage-wise?
>    > 3. What can you observe for the "Mapping Quality Across Reference" plots for both samples?
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. Mean coverage for sample 20L81 is 2,073, for 20L70 it is 2,155.
>    > > 2. Despite its slightly higher mean coverage, sample 20L70 has the slightly worse coverage *distribution*.
>    > >
>    > >    The "Coverage across reference" plot shows higher variability of coverage for this sample.
>    > >    While it peaks at higher values, it also drops below a coverage of 500 more clearly and in more regions across the genome than 20L81, and even drops to potentially problematically low values in a short stretch of sequence just after the middle of the genome.
>    > >    The "Coverage Histogram" plot also reveals that the coverage distribution for 20L70 is a bit skewed towards lower values than that for 20L81.
>    > >
>    > >    All these effects are not very dramatic and should not prevent us from continuing with the analysis of both samples. We should watch out for a potentially incomplete consensus sequence in that low-coverage stretch of the 20L70 genome though.
>    > > 3. Mapping quality is almost constant at or very near 60 (which happens to be the maximum mapping quality value emitted by BWA-MEM) across the entire genome for both samples. Interestingly, there is a tiny dip in mapping quality around ~ 117,000 - 118,000 on the genome, which seems reproducible across the two samples.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# Consensus sequence construction

And with that we are ready for consensus sequence generation!

To accept any base suggested by the mapped sequenced reads as the consensus base for the corresponding genome position, we ask for the following requirements to be fulfilled:

- at least 20 sequenced reads have to provide information about the base in question
- at a minimum, 70% of these reads have to agree on the base at this position (80% for indels)

To avoid getting misled too much by sequencing errors, we are also going to ignore bases with a base calling quality less than 20 in the above counts (i.e., we are going to base our decisions only on bases in sequenced reads that the basecaller of the sequencer was reasonably sure about.

Now what if we cannot obtain a consensus base for a position with the above criteria? In such cases of uncertainty we want to insert an N (i.e. an unknown base) to express that we either did not have enough information about the position or that this information was ambiguous.

{% icon galaxy-info %} All of the above limits for consensus base calling are arbitrary to some degree, and depend somewhat on the quality of the sequencing data. With very high overall coverage, for example, it is possible to increase the coverage threshold, but if you increase that threshold too much, you may end up with a consensus sequence consisting mostly of Ns.

> <hands-on-title>Consensus building</hands-on-title>
>
> 1. {% tool [ivar consensus](toolshed.g2.bx.psu.edu/repos/iuc/ivar_consensus/ivar_consensus/1.4.2+galaxy0) %}
>    - {% icon param-collection %} *"BAM file"*: collection of trimmed mapped reads BAM dataset; output of **ivar trim**
>    - *"Minimum frequency threshold"*: `0.7`
>    - *"Minimum indel frequency threshold"*: `0.8`
>    - *"Minimum depth to call consensus"*: `20`
>
> 2. {% icon galaxy-eye %} Inspect the consensus sequences generated for the two samples
>
>    > <question-title></question-title>
>    >
>    > Does everything look ok? What about those leading and trailing Ns in the sequences?
>    >
>    > > <solution-title>Answer</solution-title>
>    > >
>    > > Certainly both sequences start and end with stretches of Ns.
>    > >
>    > > Since the sequencing data is ampliconic in nature, and the primer scheme lists the first and the last primer as binding close to but not exactly at the end of the genome, it is expected that we cannot resolve the first and last couple of hundreds bases.
>    > >
>    > > However, to determine whether those terminal stretches of Ns are exactly as long as we expect them to be would be a tedious task to perform manually. Likewise, it is not trivial to assert that there are no additional Ns somewhere in these long sequences.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# Quality check on results and interpretation

## Composition of the consensus sequences

So we have consensus sequences for our samples. Are there better ways for checking their quality than just looking through them by eye?

Luckily, there is a tool to help us, and we have even used it before to check the masking of the reference genome!

> <hands-on-title>Check consensus sequences for unresolved bases</hands-on-title>
>
> 1. {% tool [Fasta regular expression finder](toolshed.g2.bx.psu.edu/repos/mbernt/fasta_regex_finder/fasta_regex_finder/0.1.0) %}
>    - {% icon param-collection %} *"BAM file"*: collection of consensus sequences; output of **ivar consensus**
>    - *"Regular expression"*: `[^ATGC]+`
>
>      This expression matches stretches of sequences that do *not* (`^`) contain any of the standard bases (any of the characters in `[]`). The expression is case-sensitive, but ivar consensus only uses uppercase in sequences.
>    - In *"Specify advanced parameters"*:
>      - *"Do not search the reverse complement"*: `Yes`
>      - *"Maximum length of the match to report"*: `1`
>
> 2. {% icon galaxy-eye %} Inspect the output of the tool for both samples and compare it to the primer scheme
>
>    > <question-title></question-title>
>    >
>    > Has consensus building worked well?
>    >
>    > > <solution-title>Answer</solution-title>
>    > >
>    > > The tool reports only two stretches consisting of Ns at both ends of the genome, and their positions are exactly identical between the samples. There are no interior Ns, nor other ambiguity characters.
>    > >
>    > > The 5'-end stretch of Ns extends up to position 264 (in bed format, so zero-based and the position itself not included). The binding site of the first primer in the primer scheme is from (bed format again) 235 up to 264.
>    > >
>    > > Remember that we used ivar trim to mask primer sequences? This seems to have worked well, and we got a consensus sequence from the first position for which we could reasonably expect information!
>    > >
>    > > The 3'-end stretch of Ns starts at position 150,453 and continues (as we have seen by eye before) until the end of the genome, which seems to be at 150,721.
>    > > Hmm, these values are surrounding the binding site of the last primer in the primer scheme, but do not match perfectly. But wait:
>    > >
>    > > It is possible that the consensus sequences incorporate deletions, i.e. it could be that they are shorter than the reference we have used for mapping. So obviously the **consensus sequences have a length of 


topic: variant-analysis
tutorial: somatic-variant-discovery
title: Somatic Variant Discovery from WES Data Using Control-FREEC
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: []
content_hash: c5a4bec885ea40f968edaf8dc55c3deb
last_modified: 2026-02-26T21:12:18.627802
zenodo_link: https://doi.org/10.5281/zenodo.2582555
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/somatic-variant-discovery/tutorial.html

content:
Human Copy Number Variations (hCNVs) are the result of structural genomic rearrangements that result in the duplication or deletion of DNA segments. These changes contribute significantly to human genetic variability, diseases, and somatic genome variations in cancer and other diseases {% cite Nam2015 %}. hCNVs can be routinely investigated by genomic hybridisation and sequencing technologies
. There is a range of software tools that can be used to identify and quantify hCNVs. Unfortunately, locating hCNVs is still a challenge in standardising formats for data representation and exchange. Furthermore, the sensitivity, specificity, reproducibility, and reusability of hCNV detection and analysis research software varies. As a result, there is a need for the adoption of community-developed standards for data discovery and exchange. To address that ELIXIR developed {% cite Beacon %} protocol and {% cite GA4GH %} standards, as well as mechanisms for annotating, benchmarking, creating reproducible and sharable tools and workflows, such as {% cite WorkflowHub %}, Galaxy, and {% cite OpenEBench %}, and, most importantly, accessible training resources and infrastructure.

 This tutorial is a modification of a Galaxy Training Network tutorial [Somatic variant calling]({% link topics/variant-analysis/tutorials/somatic-variants/tutorial.md %}) tutorial to provide training on how to preprocess, identify and visualise hCNV regions using Control-FreeC tool using tumor/normal samples pairs.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data Preparation
First, start with uploading and preparing the input data to analyze. The sequencing reads used in this analysis are from real-world data from a cancer patient’s tumor and normal tissue samples. For the sake of an acceptable speed of the analysis, the original data has been downsampled though to include only the reads from human chromosomes 5, 12 and 17.

| Name | Format | Origin | Encoding | Sequence length | Total Sequences | Chromosome  | Data size (MB) |
| ---- | -----| -------| ------  | -------- | --------------- | -------------- | ------------ |
| SLGFSK-N_231335_r1_chr5_12_17 | fastq | Normal tissue | Sanger / Illumina 1.9 | 101 |10602766 | 5, 12 and 7 | 530.4 MB |
| SLGFSK-N_231335_r2_chr5_12_17 | fastq | Normal tissue | Sanger / Illumina 1.9 | 101 | 10602766 | 5, 12 and 7 | 582.0 MB |
| SLGFSK-T_231336_r1_chr5_12_17 | fastq | Cancer tissue | Sanger / Illumina 1.9 | 101 |16293448 | 5, 12 and 7 | 811.3 MB |
| SLGFSK-T_231336_r2_chr5_12_17 | fastq  | Cancer tissue | Sanger / Illumina 1.9 | 101 |16293448 | 5, 12 and 7 | 868.7 MB |

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. For this tutorial, make a new history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import the data files from
>    [Zenodo](https://zenodo.org/record/2582555):
>
>    ```
>    https://zenodo.org/record/2582555/files/SLGFSK-N_231335_r1_chr5_12_17.fastq.gz
>    https://zenodo.org/record/2582555/files/SLGFSK-N_231335_r2_chr5_12_17.fastq.gz
>    https://zenodo.org/record/2582555/files/SLGFSK-T_231336_r1_chr5_12_17.fastq.gz
>    https://zenodo.org/record/2582555/files/SLGFSK-T_231336_r2_chr5_12_17.fastq.gz
>    ```
>   This will download four sequenced files ordered as: 
>   The first two files are for the forward and the reverse reads for the sample **normal**
>   tissue sequence.
>   The other two belong to the **tumor** reads.
> 
>    In some cases the same dataset can be found in the Galaxy shared data library. 
>    Ask the instructor for more details about this.
>
>    The dat aset can also be downloaded a local storage.  
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="fastqsanger.gz" %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Make sure to upload the sequences in fastaq format. Look at the history and
>    check if the created datasets have their data types assigned correctly with two reads for
>    the tumor tissues and two reads for the normal tissues. If not, fix any
>    missing or wrong data type assignments.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="fastqsanger.gz" %}
>
> 4. Give the data meaningful names and tags to facilitate analysis.
>     
>
>    When uploading data from a link, Galaxy names the files after the link address.
>    It might be useful to change or modify the name to something more meaningful.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
>
> 5. This tutorial has a set of shared steps performed on the data. To track the
>    data in the history, it is recommended to tag the datasets by attaching a meaningful tag '#'
>    to them. The tagging will automatically be attached to any file generated
>    from the original tagged dataset.
>   *e.g.*, `#normal` for normal tissue datasets (with `-N_` in the name) and
>   *e.g.*, `#tumor` for tumor dataset (with `-T_` in the name).
>
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
>
{: .hands_on}

# Quality control and mapping of NGS reads
The data was obtained following a series of laboratory procedures, including DNA preparation, extraction, and sequencing, which means there is a possibility of errors occurring during those steps, which could affect data quality. To address that, it is necessary to test the quality of the fastq reads. The data quality needs to be within an acceptable range before looking for hCNVs. The low-quality data can lead us to false results. To detect low-quality data,  preprocessing step is required to trim or discard the low-quality reads before proceeding with the mapping and hCNV detection steps.

> <comment-title>More on quality control and mapping</comment-title>
>  To read more about quality control this is tutorial on Galaxy training network 
> [ Quality Control]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}) 
> For mapping [Mapping]({% link topics/sequence-analysis/tutorials/mapping/tutorial.md %})
{: .comment}


## Quality Control

To perform quality control, we will use [Falco](https://falco.readthedocs.io/en/latest/) an efficiency-optimized rewrite of [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/).

> <hands-on-title>Quality control of the input datasets</hands-on-title>
> 1. Run {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} on the fastq datasets
>       - {% icon param-files %} *"Raw read data from the current history"*: all 4 FASTQ  datasets selected with **Multiple datasets**
>
>    {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>
>    This job will generate eight new datasets to the history. To
>    parse the quality results view the html report of each dataset.
>
>    For the next step use the raw data findings from Falco.
>
> 2. Use {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0) %} to aggregate the raw **Falco** data of all four input datasets into one report
>      - In *"Results"*
>        - *"Which tool was used generate logs?"*: `FastQC` (no matter whether FastQC or Falco was used)
>        - In *"FastQC output"*
>           - *"Type of FastQC output?"*: `Raw data`
>           - {% icon param-files %} *"FastQC output"*: all four *RawData*
>             outputs of **Falco** {% icon tool %})
>
> 3. Inspect the *Webpage* output produced by the tool
>
>    > <question-title></question-title>
>    >
>    > What do you feel about the sequence's overall quality? 
>    > ![fastqbefore](../../images/Control_Freec_fastqc_per_base_sequence_quality_plot_before.png "Sequence quality per base generated by Falco before end trimming.")
>    > ![GCbefore](../../images/Control_Freec_fastqc_per_sequence_gc_content_plot_before.png "Sequence quality per Sequence GC content generated by Falco before end trimming.")
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The forwards and reversed reads show good 
>    > >    quality, , with no major issues discovered
>    > >    discovered during the preparation process..
>    > >
>    > > 2. TThe GC content plots for the samples' forward and reverse reads 
>    > >    show an unusual bimodal distribution.
>    > >
>    > >    The unnormal distribution of the GC content of reads from a sample is 
>    > >    usually interpreted as a sign of possible contamination.
>    > >    However, we are dealing with sequencing data from captured exomes,
>    > >    which means that the reads do not represent random sequences from a  genome.
>    > >    They rather represent an arbitrary selection. Indeed, the samples were 
>    > >    prepared using Agilent's SureSelect V5 technology for exome enrichment, 
>    > >    and bimodal GC content distributions have been identified as a hallmark 
>    > >    of that capture method
>    > >    for example, see Fig. 4C in
>    > >   {% cite Meienberg2015 %}.
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

## Read trimming and filtering
As previously demonstrated, The data have relatively high-quality sequenced reads.
However, the aim is to detect clear reads for hCNVs and will use a trimming step to see if the analysis can be improved.


> <hands-on-title>Read trimming and filtering of the normal tissue reads</hands-on-title>
> 1. Run {% tool [Trimmomatic](toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.36.5) %} to trim and filter the normal tissue reads
>    - *"Single-end or paired-end reads?"*: `Paired-end (two separate
>      input files)`
>      
>      This makes the tool treat the forward and reverse reads simultaneously.
>
>      - {% icon param-file %} *"Input FASTQ file (R1/first of pair)"*: the forward reads (r1) dataset of the normal tissue sample
>      - {% icon param-file %} *"Input FASTQ file (R2/second of pair)"*: the reverse reads (r2) dataset of the normal tissue sample
>
>    - *"Perform initial ILLUMINACLIP step?"*: `Yes`
>       - *"Select standard adapter sequences or provide custom?"*: `Standard`
>       - *"Adapter sequences to use"*: `TruSeq3 (paired-ended, for MiSeq and HiSeq)`
>       - *"Maximum mismatch count which will still allow a full match to be performed"*: `2`
>       - *"How accurate the match between the two 'adapter ligated' reads must be for PE palindrome read alignment"*: `30`
>       - *"How accurate the match between any adapter etc. sequence must be against a read"*: `10`
>       - *"Minimum length of adapter that needs to be detected (PE specific/ palindrome mode)"*: `8`
>       - *"Always keep both reads (PE specific/palindrome mode)?"*: `Yes`
>
>       These parameters are used to cut ILLUMINA-specific adapter sequences
>       from the reads.
>
>    - *"Trimmomatic Operation"*
>      - *"Select Trimmomatic operation to perform"*: `Cut the specified number of bases from the start of the read (HEADCROP)`
>      - *"Number of bases to remove from the start of the read"*: `3`
>      - {% icon param-repeat %} "Insert Trimmomatic Operation"*
>        - *"Select Trimmomatic operation to perform"*: `Cut bases off the end of a read, if below a threshold quality (TRAILING)`
>        - *"Minimum quality required to keep a base"*: `10`
>      - {% icon param-repeat %} "Insert Trimmomatic Operation"*
>        - *"Select Trimmomatic operation to perform"*: `Drop reads below a
>       specified length (MINLEN)`
>        - *"Minimum quality required to keep a base"*: `25`
>
>    
{: .hands_on}

This step will creates four files in the history. The sizes of those two files vary depending on the original data quality and trimming intensity. The first two files are for mated forward and reverse reads, respectively.
The other two are for unmated reads as a result of excessive trimming.
However, because of the high average data quality, there was no need to perform excessive trimming by selecting the previous three trimming conditions, so those files should be empty. Those files can be heden to keep the history cleaner.

> <details-title>Keeping attention on the data type</details-title>
> Track whether the reads are paired or unpaired, and remember to include them in any tool to be used.
> The reason is that there are some tools, such as read mappers,
> that expect reads to be in a specific order and having unmapped reads can result in significant.
{: .details}

> <hands-on-title>Read trimming and filtering of the tumor tissue reads</hands-on-title>
> 2. repeat the previous step for **tumor tissue** reads following the same steps as above.
{: .hands_on}

> <hands-on-title>Exercise: Quality control of the polished datasets</hands-on-title>
> Use {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} and {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0) %} like before,
> but using the four trimmed datasets produced by Trimmomatic as input.
>
>    > <question-title></question-title>
>    >
>    > Is there any difference between the reads before and after the trimming?
>    > ![fastqafter](../../images/Control_Freec_fastqc_per_base_sequence_quality_plot_after.png "Sequence quality per base generated by Falco after end trimming. The x-axis on the grapgh is for the read length and the y-axis shows the quality scores. The higher the score the better the base call. The background of the graph divides the y axis into very good quality calls (green), calls of reasonable quality (orange), and calls of poor quality (red).")
>    > ![GCafter](../../images/Control_Freec_fastqc_per_sequence_gc_content_plot_afrer.png "Sequence quality per Sequence GC content generated by Falco after end trimming the plot shows an unusually shaped distribution that could indicate a contaminated library or some other kinds of biased subset")
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The quality of the data is good, so 
>    > > trimming them didn't lead to dramatic changes. However,
>    > > we can point out that some of the adapters were removed.  
>    > {: .solution}
>   {: .question}
{: .hands_on}

## Read Mapping

> <hands-on-title>Read Mapping</hands-on-title>
> 1. Use {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.1) %} to map the reads from the **normal tissue** sample to the reference genome
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>        - *"Using reference genome"*: `Human: hg19` (or a similarly named option)
>
>      > <comment-title>Using the imported `hg19` sequence</comment-title>
>      > If you have imported the `hg19` sequence as a fasta dataset into your
>      > history instead:
>      >   - *"Will you select a reference genome from your history or use a
>      >     built-in index?"*: `Use a genome from history and build index`
>      >      - {% icon param-file %} *"Use the following dataset as the reference sequence"*: your imported `hg19` fasta dataset.
>     {: .comment}
>
>    - *"Single or Paired-end reads"*: `Paired`
>       - {% icon param-file %} *"Select first set of reads"*: the trimmed
>         forward reads (r1) dataset of the **normal tissue** sample; output of
>        **Trimmomatic** {% icon tool %}
>       - {% icon param-file %} *"Select second set of reads"*: the trimmed
>         reverse reads (r2) dataset of the **normal tissue** sample; output of
>        **Trimmomatic** {% icon tool %}
>    - *"Set read groups information?"*: `Set read groups (SAM/BAM specification)`
>      - *"Auto-assign"*: `No`
>        - *"Read group identifier (ID)"*: `Not available.`
>      - *"Auto-assign"*: `No`
>        - *"Read group sample name (SM)"*: `Not available.`
>      - *"Platform/technology used to produce the reads (PL)"*: `ILLUNINA`
>      - *"Select analysis mode"*: `Simple illumina mode`
>
>    > <comment-title>More on read group identifiers and sample names</comment-title>
>    > In general, we can choose our own ID and SM values, but the ID should
>    > unambiguously identify the sequencing run that produced the reads,
>    > while the SM value should identify the biological sample.
>   {: .comment}
>
> 2. Use {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.1) %} to map the reads from the **tumor tissue** sample
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>        - *"Using reference genome"*: `Human: hg19` (or a similarly named option)
>
>      Adjust these settings as before in case of using imported reference
>      genome.
>    - *"Single or Paired-end reads"*: `Paired`
>       - {% icon param-file %} *"Select first set of reads"*: the trimmed
>         forward reads (r1) dataset of the **tumor tissue** sample; output of
>        **Trimmomatic** {% icon tool %}
>       - {% icon param-file %} *"Select second set of reads"*: the reverse
>         reads (r2) dataset of the **tumor tissue** sample; output of
>        **Trimmomatic** {% icon tool %}
>    - *"Set read groups information?"*: `Set read groups (SAM/BAM specification)`
>      - *"Auto-assign"*: `No`
>        - *"Read group identifier (ID)"*: `Not available.`
>      - *"Auto-assign"*: `No`
>        - *"Read group sample name (SM)"*: `Not available.`
>      - *"Platform/technology used to produce the reads (PL)"*: `ILLUNINA`
>      - *"Select analysis mode"*: `Simple illumina mode`
{: .hands_on}

Name the created list as **Mapping-lsit**




# Copy Number Variation detection (hCNV).

After the mapping step, the data are ready to the hCNV detection step. This tutorial focuses on the **Control-FreeC** tool as a method for hCNV identification:

- Identifies variant alleles in tumor/normal pair samples.
- Visualize the hCNV using the Circos tool.

## Mapped reads filtering

The remaining data preprocessing until the Control-FreeC step is the same for Normal and Tumor reads.  Create a data collection to include those two files.

> <hands-on-title>Filtrate the mapped reads</hands-on-title>
> 1. Use {% tool [Build list](__BUILD_LIST__) %} to creat a list from the maped reads of the **normal tissue** and **tumor tissue**
>    - {% icon param-file %} *"Dataset"*: `Insert dataset`
>       - *"Input dataset"*: `The output of map with BWA-MEM for normal tissue`
>    - {% icon param-file %} *"Dataset"*: `Insert dataset`
>       - *"Input dataset"*: `The output of map with BWA-MEM for cancer tissue`
> 2. Run {% tool [Create text file](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_text_file_with_recurring_lines/1.1.0) %}  with the following parameters
>   - *"Characters to insert"*: `normal reads`
>   - *"Specify the number of iterations by"*: `User defined number`
>   - *"How many times?"*: `1`
>   - {% icon param-repeat %} "Insert selection"*
>      - *"Characters to insert"*: `tumor reads`
>      - *"Specify the number of iterations by"*: `User defined number`
>      - *"How many times?"*: `1`
>   - This will create a text file with only two lines normal reads and tumor reads
>
> 3. Run {% tool [Relabel identifiers](__RELABEL_FROM_FILE__) %}  with the following parameters
>   - {% icon param-collection %} *"Input Collection"*: the creat a list from `Build lsit tool`.
>   - *"How should the new labels be specified?"*: `Using lines in a simple text file.`
>   - *"New Identifiers"*: The outcom from `Create txt file tool`
>   - *"Ensure strict mapping"*: `False`
>
>   - Make sure to use the same files order for the both of **Build list** tool and **Creat text file**.   
{: .hands_on}

It is essential to filtrate the reads before the mapping step.
It is also required to filtrate the reads after. The preprocessing step
works by filtrating the mapped reads by removing the low-quality regions
and the duplicated reads. This step is needed to reduce the running time
and in results interpretation.

> <hands-on-title>Data filtration and Remove duplicates</hands-on-title>
> 1. Run {% tool [Samtools view](toolshed.g2.bx.psu.edu/repos/iuc/samtools_view/samtools_view/1.9+galaxy3) %}  with the following parameters
>   - {% icon param-collection %} *"SAM/BAM/CRAM dataset"*: The outpot of  `Relabel identifiers` dataset cpllection.
>   - *"What would you like to look at?"*: `A filtered/subsampled selection of reads`
>   - *"Configure filters:"*:
>     - *"Filter by quality"*: `1`
>     - *"Exclude reads with any of the following flags set"*: `Read is unmapped` `Mate  is unmapped`
>   - *Produce extra dataset with dropped reads?"*: `False`
>   - *"Output format"*: `BAM (-b)`
>   - *"Reference data"*: `No`
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
> 2. Run {% tool [RmDup](toolshed.g2.bx.psu.edu/repos/devteam/samtools_rmdup/samtools_rmdup/2.0.1) %}  with the following parameters
>   - {% icon param-collection %} *"BAM File"*:  The outpot of  `Samtools view`
>   - *"Is this paired-end or single end data"*: `BAM is paired-end`
>   - *"Treat as single-end"*: `False`
{: .hands_on}




## Homogenize mapped reads

To detect hCNVs expression accurately. The reads must go through the lift alignment process. Lift Alignment works by shifting reads that contain indels to the left of a reference genome until they can not be shifted anymore. As a result, it will only extract reads with indels, with no false reads (reads that mismatch with the reference genome other than the indels).


> <hands-on-title>Homogenize the positional distributed indels</hands-on-title>
> 1. Run {% tool [BamLeftAlign](toolshed.g2.bx.psu.edu/repos/devteam/freebayes/bamleftalign/1.3.1) %}  with the following parameters
>   - *"Choose the source for the reference genome"*: `Locally cached `
>   - {% icon param-collection %} *"Select alignment file in BAM format"*: The outpot of  `tool RmDup`.
>   - *"Using reference genome"*: `hg19`
>   - *"Maximum number of iterations"*: `5`
>
> 2. Run {% tool [Samtools calmd](toolshed.g2.bx.psu.edu/repos/devteam/samtools_calmd/samtools_calmd/2.0.2) %}  with the following parameters
>   - {% icon param-collection %} *"BAM file to recalculate"*: The outpot of  `BamLeftAlign`.
>   - *"Choose the source for the reference genome"*: `Use a built-in genome`
>   - *"Using reference genome"*: `hg19`
>   - *"Do you also want BAQ (Base Alignment Quality) scores to be calculated?"*: `No`
>   - *"Additional options"*: `Advanced options`
>   - *"Change identical bases to '='"*: `False`
>   - *"Coefficient to cap mapping quality of poorly mapped reads"*: `50`
{: .hands_on}


After the Homogenizing step, it is now to extract the reads which hold indels from the mapped reads.

## Filtrate indels

> <hands-on-title>Filtrate the indels reads</hands-on-title>
> 1. Run {% tool [Samtools view](toolshed.g2.bx.psu.edu/repos/iuc/samtools_view/samtools_view/1.9+galaxy3) %}  with the following parameters
>   - {% icon param-collection %} *"BAM file to recalculate"*: The outpot of  `Samtools CalMD`.
>   - *"What would you like to look at?"*: `Just the input header (-H)`
>   - *"What would you like to have reported?"*: `The header in ...`
>   - *"Output format"*: `SAM`
>   - *"Reference data"*: `No, see help (-output-fmt-option no_ref)`  
> 2. Run {% tool [Select](Grep1) %}  with the following parameters
>   - {% icon param-collection %} *"Select lines from"*: The outpot of  `Samtools view`.
>   - *"that"*: `Matching`
>   - *"the pattern"*: `^@SQ\tSN:(chr[0-9]+|chrM|chrX|chrY)\tLN:[0-9]+`
>   - *"Keep header line"*: `False`
>
>
> 3. Run {% tool [Replace Text in entire line](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2) %} with the following parameters:
>   - {% icon param-collection %} *"Select lines from"*: The output of `Select`
>   - *"pattern"*: `^@SQ\tSN:(chr[0-9]+|chrM|chrX|chrY)\tLN:([0-9]+)`
>   - *"Replace with"*: `\1\t0\t\2`
>
>    This step will generate a data collection folder with two files inside. Change the datatype for the files inside it to **BED format**.
>
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
>
> 4. Run {% tool [Samtools view](toolshed.g2.bx.psu.edu/repos/iuc/samtools_view/samtools_view/1.9+galaxy3) %}  with the following parameters
>   - {% icon param-collection %} *"SAM/BAM/CRAM data set"*: The outpot of  `CalMD`.
>   - *"What would you like to look at?"*: `A filtered/subsampled selection of reads`
>   - *Configure filters*
>   - *"Filter by quality"*: `255`
>   - *What would you like to have reported?"*: `Reads dropped during filtering and subsampling`
>   - *Produce extra dataset with dropped reads?"*: `False`
>   - *"Output format"*: `BAM (-b)`
>   - *"Reference data"*: `No`
> 5. Run {% tool [Samtools view](toolshed.g2.bx.psu.edu/repos/iuc/samtools_view/samtools_view/1.9+galaxy3) %}  with the following parameters
>   - {% icon param-collection %} *"SAM/BAM/CRAM data set"*: The outpot of  `Samtools view`.
>   - *"What would you like to look at?"*: `A filtered/subsampled selection of reads`
>   - *"Configure filters"*
>   - *"Filter by regions:"*: `Regions from BED file`
>   - {% icon param-collection %} *"Filter by intervals in a bed file:"*: The outpot of  ` Replace Text` BED format.
>   - *"Filter by readgroup"*: `NO`
>   - *"Filter by quality"*: `1`
>   - *Produce extra dataset with dropped reads?"*: `False`
>   - *"Output format"*: `BAM (-b)`
>   - *"Reference data"*:`No`
{: .hands_on}

> <hands-on-title>Extract files form list</hands-on-title>
> extract the files from the list to handel them separitly
> 1. Run {% tool [Extract Dataset](__EXTRACT_DATASET__) %}  with the following parameters:
>   - {% icon param-collection %} *"Input List"*: `The outpot of Samtools view`.
>   - *"How should a dataset be selected?"*: `Select by element identifier`
>   - *"Element identifier:"*: `tumor reads`
>
> 2. Run {% tool [Extract Dataset](__EXTRACT_DATASET__) %}  with the following parameters:
>   - {% icon param-collection %} *"Input List"*: The outpot of  `Samtools view`.
>   - *"How should a dataset be selected?"*: `Select by element identifier`
>   - *"Element identifier:"*: `normal reads`
{: .hands_on}







## Control_FREEC for hCNV detection  
The data are now ready to detect hCNV. Control-FREEC detects copy-number alterations and allelic imbalances (including loss of heterozygosity; LOH) by automatically computing, normalising, and segmenting copy number profile and beta allele frequency (BAF) profile, and then calling copy number alterations and LOH. Control-FREEC differentiates between somatic and germline variants. Based on those profiles. The control reads display the gene status for each segment.
![Controlfreec](../../images/Control_Freec_output.png "1. Control-FREEC calculates copy number and BAF profiles and detects regions of copy number gain/loss and LOH regions in chromosomes 5,12 17. Gains, losses and LOH are shown in red, blue and light blue, respectively.")

> <comment-title>More on control_FREEC and hCNVs detection</comment-title>
> Control-freec works by: 
> 1. Annotating genomic changes and heterozygosity loss in the sample dataset.
> 2. Distinguishes between germline and somatic variants by creating copy number profile and BAF profile.
> 3. Employs the information in those profiles to detect copy number changes in sample reads.
>
{: .comment}


> <question-title></question-title>
> Can you expect the essential factors that afflict hCNVs detection?
>
> > <solution-title></solution-title>
> >
> > 1. Coverage bias in reads 
> > Changes in reading mobility and GC content may favour the duplication of specific reads over others.
> > 2. Bias in reading alignment
> > Because normal cells have higher read coverage than allelic reads during alignment, they may be classified as noise findings.
> > 3. Normal cell contamination 
> > The presence of normal cells within tumour cells can have an impact on the construction of a tumour genome's copy number profile.
> {: .solution}
{: .question}





> <hands-on-title>Detection of copy-number changes</hands-on-title>
> 1. Import the BED file for the captured reagions from [Zenodo](https://doi.org/10.5281/zenodo.5697358):
>
>    ```
>    https://zenodo.org/record/5697358/files/capture_targets_chr5_12_17.bed
>    ```
>
 > 2. Run {% tool [Control-FREEC](toolshed.g2.bx.psu.edu/repos/iuc/control_freec/control_freec/11.6+galaxy1) %}  with the following parameters
>   - *"Select the sequencing method of the input file(s)?"*: `whole-exome sequencing (WES)`
>   - {% icon param-file %} *"Sample file"*: The outpot of  `Extract Dataset` (**cancer reads**)
>   - {% icon param-file %} *"Control file"*: The outpot of  `Extract Dataset` (**normal reads**)
>   - {% icon param-file %} *"BED file with capture regions"*: `chr 5, 7, 12 capture reagions **Input file**` 
>   - *"Format of reads"*: `Illumina paired-end (FR)`
>
>   - *"Advanced WES settings:"*:
>     - *"Degree of polynomial:"*: `control-read-count-based normalization, WES (1)`
>     - *"Read Count (RC) correction for GC-content bias and low mappability:"*: `normalize the sample and the control RC using GC-content and then calculate the ratio "Sample RC/contol RC" (1)`
>     - *"Minimal number of consecutive windows to call a CNA"*: `WES (3)`
>     - *"Segmentation of normalized profiles (break point)"*: `1.2`
>     - *"Desired behavior in the ambiguous regions"-1" "*: `make a separate fragment of this "unknown" region and do not assign any copy number to this region at all (4)`
>     - *"Adjust sample contamination?"*: `True`
>     - *"Sample contamination by normal cells"*: `0.30000000000000004`
>     - *"Intercept of polynomial"*: `with GC-content (1)`
>     - *"Sample sex"*: `XX`
>
>   - *"Choose the source for the reference genome"*: `Locally cached`
>   - *"Reference genome"*: `hg19`
>   - *"Outputs:"*:
>     - *"BedGraph Output for UCSC genome browser"*: `False`
>     - *"Visualize normalized copy number profile with predicted CNAs"*: `True`
>     - *"2D data track file for Circos"*: `True`
>
> > <question-title></question-title>
> >
> > In your opinion, what are the challenges in hCNVs detection?
> > > <solution-title></solution-title>
> > >
> > > 1. The bias in Reads coverage changes in reads mobility and GC content can lead to 
> > > favouring the duplication of specific reads over others.
> > > 2. The bias in reads alignment. 
> > > Normal cells' reads coverage is higher than the allelic reads, for 
> > > that the allelic reads some times expressed as noise findings
> > > 3. Contamination with normal cells 
> > > The availability of normal cells within tumor cells can
> > > effect on the construction of the copy number profile of a 
> > > tumor genome
> > {: .solution}
> {: .question}
{: .hands_on}


## Visualise detected hCNVs

Cicros demonstrates the relationship and the positions of different objects with an appealing,
high quality and illustrative multilayers circular plot. Circos gives the user flexibility to
present the link between their data at a high rate by providing the ability to control the features 
and elements in creating the plot. Circos visualise the genomic alterations in genome structure and 
the relationships between the genomic intervals [Krzywinski, Schein et al. 2009](https://genome.cshlp.org/content/19/9/1639.short).

> <hands-on-title>Visualise the hCNV findings</hands-on-title>
> 1. Run {% tool [Circos](toolshed.g2.bx.psu.edu/repos/iuc/circos/circos/0.69.8+galaxy7) %}  with the following parameters
>   - *"Reference Genome Source"*: `Custom Karyotype`
>   - {% icon param-file %} *"Sample file"*: The outpot of  `Output dataset out_chr_sorted_circos from control freec`
>   - *"Ideogram:"*:
>     - *Spacing Between Ideograms (in chromosome units)"*: `3.0`
>     - *"Radius"*: `0.8`
>     - *"Thickness"*: `45.0`
>     - *"Labels:"*:
>       - *"Radius"*: `0.01`
>       - *"Label Font Size"*: `40`
>     - *"Cytogenic Bands:"*:
>       - *"Band Stroke Color"*: `Black`
>    - *"2D Data Tracks:"*:
>      - {% icon galaxy-wf-new %} "Insert 2D Data Plot"*
>        - *"Outside Radius"*: `0.95`
>        - *"Plot Type"*: `Scatter`
>        - {% icon param-file %} *"Scatter Plot Data Source"*: `Output dataset 'out_ratio_log2_circos' from Control-FreeC`
>        - *"Plot Format Specific Options:"*:
>          - *"Glyph"*: `Circle`
>          - *"Glyph Size"*: `4`
>          - *"Fill Color"*: `Gray`
>          - *"Stroke Color"*: `Black`
>          - *"Stroke Thickness"*: `0`  
>         - *"Rules:"*:
>           - {% icon galaxy-wf-new %} *"Insert Rule"*:
>             - {% icon galaxy-wf-new %} *"Insert Conditions to Apply"*
>               - *"Condition"*: `Based on value (ONLY for scatter/histogram/heatmap/line)`
>               - *"Points above this value"*: `0.0`
>             - {% icon galaxy-wf-new %} *"Insert Actions to Apply"*
>               - *"Action"*: `Change Fill Color for all points`
>               - *"Fill Color"*: `Red`
>             - *"Continue flow"*: `False`
>           - {% icon param-repeat %} *"Insert Rule"*:
>               - {% icon galaxy-wf-new %} *"Insert Conditions to Apply"*:
>                 - *"Condition"*: `Based on value (ONLY for scatter/histogram/heatmap/line)`
>                 - *"Points below this value"*: `0.0`
>               - {% icon galaxy-wf-new %} *"Insert Actions to ApplyInsert Actions to Apply"*:
>                 - *"Action"*: `Change Fill Color for all points`
>                 - *"Fill Color"*: `Blue`
>               - *"Continue flow"*: `False`
>         -  In*"Axes:"*:
>           - {% icon galaxy-wf-new %} *"Insert Conditions to Apply"*
>             - *"Radial Position"*: `Absolute position (values match data values)`
>             - *"Spacing"*: `1.0`
>             - *"y0"*: `-4.0`
>             - *"y1"*: `4.0`
>             - *"Color"*: `Gray`
>             - *"Color Transparency"*: `1.0`
>             - *"Thickness"*: `2`
>           - *"When to show"*: `Always`
>   - *"Limits:"*:
>     - *"Maximum number of links to draw"*: `2500000`
>     - *"Maximum number of points per track"*: `2500000`
{: .hands_on}

> <question-title></question-title>
> Can you interpret generated plot from the Circos tool?
> ![Cicros](../../images/Control_Freec-%5BCircos_Plot%5D.png "	This circos plot shows of genetic alterations in chromosomes 5,12 and 17 tutmer tissues. The outermost circle represents the targeted chromosomes and inner circle for deletion related and amplificated related variations ")
> ![Cicros](../../images/Control_Freec_Cicros_zoomed.png "Zoomed hCNV regions detected by Control_FREEC using Cicros")
> > <solution-title></solution-title>
> > The outermost circle represents the targeted chromosomes. 
> > The inner circle
> > shows the copy number changes in those regions.
> >
> > The inner circle has two parts. The deletion-related variations are the red dots directed to the centre 
> > (below the centre line), while the amplificated variations are the green dots pointed out of the center (toward chromosomes).
> {: .solution}
{: .question}

# Conclusion

In this tutorial, we introduced Contol-FreeC as an alternative tool for detecting hCNVs and highlighted the steps for preparing reads and analysis.


topic: variant-analysis
tutorial: dunovo
title: Calling very rare variants
difficulty: intermediate
time_estimation: 3h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 43252ea709ff77d7bce6fb1043aa6c9c
last_modified: 2026-02-26T21:12:18.616747
zenodo_link: https://doi.org/10.5281/zenodo.3554549
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/dunovo/tutorial.html

content:
This page explains how to perform discovery of low frequency variants from duplex sequencing data. As an example we use the *ABL1* dataset published by [Schmitt and colleagues](https://www.ncbi.nlm.nih.gov/pubmed/25849638) (SRA accession [SRR1799908](https://www.ncbi.nlm.nih.gov/sra/?term=SRR1799908)).

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Background

### Finding rare variants

Most popular variant callers focus on the common case of sequencing a diploid individual to find heterozygous and homozygous variants. This is a well-studied problem with its own challenges, but at least you can expect your variants to be present in either 100%, 50%, or 0% of your sample DNA. If you observe a variant present in 99%, 56%, or 2% of the reads at a site, you can probably assume the allele is actually present at 100%, 50%, or 0%, respectively, in your sample.

But in this tutorial, we're looking for **rare variants**. So our true frequency might actually be 13%, 1%, or even 0.4%. The challenge then becomes distinguishing these situations from sequencing errors. Next-generation sequencers produce noise at this level, making it challenging to make this distinction in data produced with standard resequencing methods.

### Duplex sequencing

[Duplex sequencing](http://www.pnas.org/content/109/36/14508.short) is a method that addresses the problem of distinguishing sequencing signal from noise. It can increase sequencing accuracy by over four orders of magnitude. Duplex sequencing uses randomly generated oligomers to uniquely tag each fragment in a sample after random shearing. The tagged fragments are then PCR amplified prior to sequencing, so that many reads can be obtained from each original molecule. The tags in each read can then be used to identify which original fragment the read came from. Identifying multiple reads from each fragment allows building a consensus of the original sequence of the fragment, eliminating errors.

The key to duplex sequencing, as opposed to other types of consensus-based methods ([review here](https://www.nature.com/articles/nrg.2017.117)), is that both ends of the original fragment are tagged such that its strands can be distinguished. Knowing which strand each read comes from allows us to recognize errors even in the first round of PCR.

Processing the raw reads into consensus sequences consists of four main steps:
1. Group reads by their tags.
2. Align reads in the same tag group.
3. Build single-strand consensus sequences (**SSCS**) of reads coming from the same original strand.
4. Build duplex consensus sequences (**DCS**) from pairs of SSCS.

Du Novo is a tool which can carry out these steps. Unlike most other such tools, it can do so without the use of a reference sequence, and it can correct for errors in the tags which can contribute to data loss.

> <comment-title>Terminology</comment-title>
>
> Du Novo processes the tags from each fragment by concatenating them into a single **barcode**.
>
> - For a standard protocol with two 12bp tags, this results in a 24bp barcode which identifies each family.
>
{: .comment}

[Schmitt *et al.* 2012](http://www.pnas.org/content/109/36/14508.short) provides this overview of the whole method:
![duplex](../../images/ds.png)

**Figure 1:** The logic of duplex sequencing. The computational process is shown in part **C**.

### The value of single-strand consensus sequences

The DCSs have the ultimate accuracy, yet the SSCSs can also be very useful when ampliconic DNA is used as an input to a duplex experiment. Let us illustrate the utility of SSCSs with the following example. Suppose one is interested in quantifying variants in a virus that has a very low titer in body fluids. Since the duplex procedure requires a substantial amount of starting DNA (between [between 0.2 and 3 micrograms](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4271547/)) the virus needs to be enriched. This can be done, for example, with a PCR designed to amplify the entire genome of the virus. Yet the problem is that during the amplification heterologous strands will almost certainly realign to some extent forming heteroduplex molecules:

![hd](../../images/het.png){:height="444px" width="605px"}

**Figure 2:** Heteroduplex formation in ampliconic templates. Image by Barbara Arbeithuber from [Stoler *et al.* 2016](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-016-1039-4). Here there are two distinct types of viral genomes: carrying `A` and `G`. Because the population of genomes is enriched via PCR, heteroduplex formation takes place, skewing frequency estimates performed using DCSs.

In the image above there are two alleles: green (`A`) and red (`G`). After PCR a fraction of molecules are in heteroduplex state. If this PCR-derived DNA is now used as the starting material for a DS experiment, the heteroduplex molecules will manifest themselves as having an `N` base at this site (because Du Novo interprets disagreements as `N`s during consensus generation). So, DSCs produced from this dataset will have `A`, `G`, and `N` at the polymorphic site. Yet, SSCSs will only have `A` and `G`. Thus SSCS will give a more accurate estimate of the allele frequency at this site in this particular case. In Du Novo SSCSs are generated when the {% icon param-check %} *Output single-strand consensus sequences* option of {% icon tool %} **Du Novo: Make consensus reads** tool is set to `Yes` (see [below](#making-consensus-sequences)).

## How to use this tutorial

The entire analysis described here is accessible as a [Galaxy history](https://usegalaxy.org/u/nstoler/h/du-novo-gtn-tutorial) that you can copy and play with.

> <comment-title>Running the tools</comment-title>
> * Leave all parameters on their default settings, unless instructed otherwise.
>   > <comment-title>Helping Du Novo</comment-title>
>   > But if you'd like to help improve Du Novo, consider checking `Yes` under {% icon param-check %} *Send usage data*.
>   {: .comment}
{: .comment}

This analysis can be divided into three parts:
 1. [Generating consensus sequences](#generating-consensus-sequences)
 2. [Calling variants with duplex consensus sequences](#calling-variants-with-duplex-consensus-sequences)
 3. [Calling variants with single strand consensus sequences](#calling-variants-with-single-strand-consensus-sequences)

Here are the steps, displayed as the Galaxy history you'll end up with if you follow the instructions:
- Note: Galaxy histories show the first step at the bottom!

![steps](../../images/history-dunovo.png)

**Figure 3:** Analysis outline

# Generating consensus sequences

The starting point of the analysis is sequencing reads (in [FASTQ](https://en.wikipedia.org/wiki/FASTQ_format) format) produced from a duplex sequencing library.

## Getting data in and assessing quality

You can obtain the data from [Schmitt *et al.* 2015](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4414912/) from [Zenodo](https://doi.org/10.5281/zenodo.3554549), or just import the data from the Galaxy Data library under the "GTN Materials" folder.

> <hands-on-title>Importing the raw data</hands-on-title>
>
> 1. **Import** the datasets from Zenodo, or from a data library.
>
>    ```
>    https://zenodo.org/record/3554549/files/SRR1799908_forward.fastq
>    https://zenodo.org/record/3554549/files/SRR1799908_reverse.fastq
>    ```
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. **Rename** {% icon galaxy-pencil %} the datasets to
>    - `SRR1799908_forward`
>    - `SRR1799908_reverse`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

This creates two datasets in our galaxy history: one for forward reads and one for reverse.

We then evaluated the quality of the data by running FastQC on both datasets (forward and reverse):

> <hands-on-title> Evaluating input read quality </hands-on-title>
>
> 1. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *Short read data from your current history*: One of the raw FASTQ datasets (`SRR1799908_forward`)
>    - Leave the rest of the options as their defaults.
>
> 2. Repeat with the other FASTQ file (`SRR1799908_reverse`)
>
{: .hands_on}


This created two datasets in our galaxy history: one for forward reads and one for reverse. We then evaluated the quality of the data by running FastQC on both datasets (forward and reverse). You can read about using {% icon tool %} **FastQC** in the dedicated [quality-control tutorial]({{ site.baseurl }}/topics/sequence-analysis/tutorials/quality-control/tutorial.html#assess-quality-with-fastqc---short--long-reads).

This gave us the following plots:

![Quality scores across all bases: foward](../../images/abl1-f-qc.png) | ![Quality scores across all bases: reverse](../../images/abl1-r-qc.png)

**Figure 4:** FastQC assessment of the quality of the raw reads. **Left:** Forward reads. **Right:** Reverse reads.

One can see that these data are of excellent quality and no additional processing is required before we can start the actual analysis.

## Processing reads into duplex consensus sequences with Du Novo

Now we are ready to collapse the raw reads into duplex consensus sequences.

### Sorting reads into families

The {% icon tool %} **Du Novo: Make families** tool will separate the 12bp tags from each read pair and concatenate them into a 24bp barcode. Then, it will use the barcodes to sort the reads into families that all descend from the same original fragment.

> <hands-on-title>Sorting reads into families</hands-on-title>
>
> 1. {% tool [Du Novo: Make families](toolshed.g2.bx.psu.edu/repos/nick/dunovo/make_families/3.0.2) %} with the following parameters:
>    - {% icon param-file %} *Sequencing reads, mate 1*: The forward raw reads (`SRR1799908_forward`)
>    - {% icon param-file %} *Sequencing reads, mate 2*: The reverse raw reads (`SRR1799908_reverse`)
>    - {% icon param-text %} *Tag length*: `12`
{: .hands_on}

### Correcting barcodes

The grouping reads based on barcode relies on exact barcode matches. Any PCR or sequencing error in the barcode sequence will prevent the affected reads from being joined with their other family members.

Du Novo includes a tool which can correct most of these errors and recover the affected reads. This can increase the final yield of duplex consensus reads by up to 11% (Stoler *et al.* 2018, in preparation).

> <hands-on-title>Correcting barcodes</hands-on-title>
>
> 1. {% tool [Du Novo: Correct barcodes](toolshed.g2.bx.psu.edu/repos/nick/dunovo/correct_barcodes/3.0.2) %} with the following parameters:
>    - {% icon param-file %} *Input reads*: The output of {% icon tool %} **Make families**
>    - {% icon param-text %} *Maximum differences*: `3`
{: .hands_on}

### Aligning families

After grouping reads that came from the same original fragment, we need to align them with each other. This next tool will perform a multiple sequence alignment on each family.

> <comment-title>Analysis bottleneck</comment-title>
> This is by far the most time-consuming step.
>
> On this dataset, it took 2 hours to complete when run on [Galaxy Main](https://usegalaxy.org/).
> - At the time, Galaxy allocated 6 cores to the job.
{: .comment}

> <hands-on-title>Aligning families</hands-on-title>
>
> 1. {%  tool [Du Novo: Align families](toolshed.g2.bx.psu.edu/repos/nick/dunovo/align_families/3.0.2) %} with the following parameters:
>    - {% icon param-file %} *Input reads*: The output of {% icon tool %} **Correct barcodes**
>    - {% icon param-select %} *Multiple sequence aligner*: `Kalign2`
{: .hands_on}

### Making consensus sequences

Now, we need to collapse the aligned reads into consensus sequences. This next tool will process each group of aligned reads that came from the same single-stranded family into a consensus. Then it will align the consensus sequences from the two strands of each original molecule, and call a consensus between them.

Normally, the tool only produces the final double-stranded consensus sequences. But we will make use of the single-stranded consensus sequences [later](#calling-variants-with-single-strand-consensus-sequences), so we'll tell it to keep those as well.

> <hands-on-title>Making consensus sequences</hands-on-title>
>
> 1. {% tool [Du Novo: Make consensus reads](toolshed.g2.bx.psu.edu/repos/nick/dunovo/dunovo/3.0.2) %} with the following parameters:
>    - {% icon param-file %} *Aligned input reads*: The output of {% icon tool %} **Align families**
>    - {% icon param-text %} *Minimum reads for a consensus sequence*: `3`
>    - {% icon param-text %} *Consensus % threshold*: `0.7`
>    - {% icon param-select %} *Output format*: `FASTQ`
>    - {% icon param-check %} *Output single-strand consensus sequences as well*: `Yes`
>
> 2. This tool will produce 4 output files, in two sets of paired-end FASTQ files:
>    - one pair for the single-stranded consensus sequences (called `SSCS mate 1` and `2`)
>    - one pair for the double-stranded consensus sequences (called `mate 1` and `2`).
{: .hands_on}

> <comment-title>Setting output formats</comment-title>
>
> You may have to set the datatype of the outputs from {% icon tool %} **Du Novo: Make consensus reads** tool.
>
> Versions below 2.16 only set the datatype to `fastq`, not the more specific `fastqsanger`. Many tools (like {% icon tool %} **Map with BWA-MEM**) won't accept FASTQ input without it specifying what subtype it is.
>
> 1. In your history, click on the pencil icon next to the dataset name.
> 2. Click on the **Datatypes** tab.
> 3. In the **Change datatype** pane, click on the dropdown where it says `fastq`.
> 4. Enter `fastqsanger`, then click the **Change datatype** button in the upper right of the pane.
{: .comment}

> <details-title>Where do the FASTQ quality scores come from?</details-title>
>
> There is no easy way to assign a PHRED score to a consensus base derived from many duplex reads.
>
> So Du Novo does not attempt to give a meaningful score. It assigns the same arbitrary score to all bases.
> - It produces FASTQ for compatibility, but the output contains no more information than a FASTA file.
>
> You may have noticed the {% icon param-text %} *Output PHRED score* parameter in the {% icon tool %} **Du Novo: Make consensus reads** tool. This allows you to specify which score to assign to (all) the bases.
>
{: .details}


## Filtering consensuses

You may have realized that when calling a "consensus" between two sequences, if the two disagree on a base, there's no way to know which is correct. So in these situations, Du Novo uses the [IUPAC ambiguity letter](https://en.wikipedia.org/wiki/Nucleic_acid_notation) for the two different bases (e.g. `W` = `A` or `T`). Also, when calling single-stranded consensus sequences, if there aren't enough high-quality bases to call a position (in the [above hands-on](#hands-on-making-consensus-sequences), we set this threshold to 70%), it gives an `N`.

This information could be useful for some analyses, but not for our variant calling. The tool {% icon tool %} **Sequence Content Trimmer** will help with filtering these out. With the settings below, it will move along the read, tracking the frequency of ambiguous (non-ACGT) bases in a 10bp window. If it sees more than 2 ambiguous bases in a window, it will remove the rest of the read, starting with the first offending base in the window. We'll also tell it to remove entirely any read pair containing a read that got trimmed to less than 50bp.

> <hands-on-title>Filtering the consensus sequences</hands-on-title>
>
> 1. {% tool [Sequence Content Trimmer](toolshed.g2.bx.psu.edu/repos/nick/sequence_content_trimmer/sequence_content_trimmer/0.2.3) %} with the following parameters:
>    - {% icon param-select %} *Paired reads?*: `Paired`
>    - {% icon param-file %} *Input reads (mate 1)*: The double-stranded output of {% icon tool %} **Make consensus reads** (mate 1)
>    - {% icon param-file %} *Input reads (mate 2)*: The double-stranded output of {% icon tool %} **Make consensus reads** (mate 2)
>    - {% icon param-text %} *Bases to filter on*: `ACGT`
>    - {% icon param-text %} *Frequency threshold*: `0.2`
>    - {% icon param-text %} *Size of the window*: `10`
>    - {% icon param-check %} *Invert filter bases*: `Yes`
>    - {% icon param-check %} *Set a minimum read length*: `Yes`
>    - {% icon param-text %} *Minimum read length*: `50`
{: .hands_on}

# Calling variants with duplex consensus sequences

At this point we have trimmed DCSs. We can now proceed to call variants. This involves aligning the variants against the reference genome, then counting variants.

We're not specifically interested in the reference sequence, since all we care about is sequence content of the consensus reads. But we'll be using the reference sequence to figure out where all the reads come from. This lets us stack them on top of each other, with equivalent bases lined up in columns. Then we can step through each column, count how many times we see each base, and and compile a list of variants.

## Mapping the reads

### Align against the genome with BWA-MEM

Here, we'll use {% icon tool %} **Map with BWA-MEM** to map the DCS reads to the human reference genome.

> <hands-on-title>Align with BWA-MEM</hands-on-title>
>
> 1. {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.18) %} with the following parameters:
>    - {% icon param-select %} *Using reference genome?*: `Human (Homo sapiens) (b38): hg38`
>    - {% icon param-file %} *Select first set of reads*: The first output from the {% icon tool %} **Sequence Content Trimmer**
>    - {% icon param-file %} *Select second set of reads*: The second output from the {% icon tool %} **Sequence Content Trimmer**
{: .hands_on}

### Left Aligning indels

To normalize the positional distribution of indels we use the {% icon tool %} **BamLeftAlign** utility from the [FreeBayes](https://github.com/ekg/freebayes#indels) package. You can find it in the **NGS: Variant Analysis** section. This is necessary to avoid erroneous polymorphisms flanking regions with indels (e.g., in low complexity loci):

> <hands-on-title>Left-align indels</hands-on-title>
>
> 1. {% tool [BamLeftAlign](toolshed.g2.bx.psu.edu/repos/devteam/freebayes/bamleftalign/1.3.8) %} with the following parameters:
>    - {% icon param-file %} *Select alignment file in BAM format*: The output of {% icon tool %} **Map with BWA-MEM**
>    - {% icon param-select %} *Using reference genome*: `Human (Homo sapiens): hg38`
>      - The same genome we aligned to.
{: .hands_on}

## Calling the variants

Now we'll use our aligned consensus reads to find variants.

Normally, in a diploid resequencing experiment, you would call variants relative to the reference. So, you'd report sites which are different from the reference (and whether they're hetero- or homozygous).

In our case, we're interested in *rare* variants. So what we'll report is the sites where there is more than one allele, and what the frequency is of the less-common allele (the **minor allele**). This has the potential to include every small sequencing error (even though we're using duplex, there still are errors). So to reduce the noise, we'll set a lower threshold at 1% minor allele frequency (**MAF**).

### Finding variants in the alignment

To identify sites containing variants we use the {% icon tool %} **Naive Variant Caller (NVC)** tool from the **NGS: Variant Analysis** section. This reads the alignment and counts the number of bases of each type at each site.

> <hands-on-title>Count the variants</hands-on-title>
>
> 1. {%  tool [Naive Variant Caller (NVC)](toolshed.g2.bx.psu.edu/repos/blankenberg/naive_variant_caller/naive_variant_caller/0.0.4) %} with the following parameters:
>    - {% icon param-file %} *BAM file*: The output of {% icon tool %} **BamLeftAlign**
>    - {% icon param-select %} *Using reference genome*: `hg38`
>      - The same genome we aligned to.
>    - {% icon param-check %} *Insert Restrict to regions*: Click to add a region.
>    - {% icon param-text %} *Chromosome*: `chr9`
>      - *ABL1* is on chr9. Restricting it to this region saves some processing time.
>    - {% icon param-text %} *Minimum base quality*: `0`
>      - In our case, base quality [isn't meaningful](#details-where-do-the-fastq-quality-scores-come-from), so we set the threshold to 0.
>    - {% icon param-text %} *Minimum mapping quality*: `20`
>    - {% icon param-text %} *Ploidy*: `1`
>      - Ploidy is irrelevant here as it is a mixture of multiple genomes.
{: .hands_on}

The {% icon tool %} **Naive Variant Caller (NVC)** generates a [VCF](https://en.wikipedia.org/wiki/Variant_Call_Format) file that can be viewed at genome browsers such as [IGV](https://www.broadinstitute.org/igv/). Yet one rarely finds variants by looking at genome browsers. We'll want to use tools to search for variants that fit our criteria.

### Finding minor alleles

Now we'll want to parse the VCF produced by the NVC, determine what the major and minor allele is at each site, and calculate their frequencies. The {% icon tool %} **Variant Annotator** from the **NGS: Variant Analysis** section can do this.

> <hands-on-title>Read the variants file</hands-on-title>
>
> 1. {% tool [Variant Annotator](toolshed.g2.bx.psu.edu/repos/nick/allele_counts/allele_counts_1/1.3.2) %} with the following parameters:
>    - {% icon param-file %} *Input variants from Naive Variants Detector*: The output of {% icon tool %} **Naive Variant Caller (NVC)**
>    - {% icon param-text %} *Minor allele frequency threshold*: `0`
>    - {% icon param-text %} *Coverage threshold*: `10`
>    - {% icon param-check %} *Output stranded base counts*: `Yes`
>      - To be able to filter for strand bias.
{: .hands_on}

### Filtering out the noise

Now we have a file containing the base counts for every site covered by at least 10 reads. We'd like to filter through this data to find sites with a reasonable chance of being a real variant, not sequencing error.

The {% icon tool %} **Variant Annotator** produces a simple tab-delimited file, with one site per line. We can use the {% icon tool %} **Filter** tool from the **Filter and Sort** section to process this kind of file. We'll use the filter `c16 >= 0.01` to remove lines where the value in column 16 is less than 0.01. Column 16 contains the minor allele frequency, so this will remove all sites with a MAF less than 1%.

> <hands-on-title>Filter the raw variants list</hands-on-title>
>
> 1. {% tool [Filter - data on any column using simple expressions](Filter1) %} with the following parameters:
>    - {% icon param-file %} *Filter*: The output of {% icon tool %} **Variant Annotator**
>    - {% icon param-text %} *With following condition*: `c16 >= 0.01`
>    - {% icon param-check %} *Number of header lines to skip*: `1`
{: .hands_on}

## Results

Now we're down to just two sites:

Position (chr9)| Major allele | Minor allele |    MAF    |
   Column 3    |  Column 14   | Column 15    | Column 16 |
|:-------------|:------------:|:------------:|:---------:|
  130,872,141  |      G       |      A       |  0.01259  |
  130,880,141  |      A       |      G       |  0.47764  |

The polymorphism we are interested in (and the one reported by [Schmitt *et al.* 2015](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4414912/)) is at the position 130,872,141 and has a frequency of 1.3%. The other site (position 130,880,141) is a known common variant [rs2227985](https://www.ncbi.nlm.nih.gov/SNP/snp_ref.cgi?type=rs&rs=rs2227985), which is heterozygous in this sample.

# Calling variants with single strand consensus sequences

To analyze the SSCS data, go back to the [Filtering consensuses](#filtering-consensuses) step, and replace the input to the {% icon tool %} **Sequence Content Trimmer** with the single-stranded output of {% icon tool %} **Make consensus reads** instead of the double-stranded output. Then continue through the same steps as before, with the same parameters, to the end.

Then you will have a new set of variants, but produced with the SSCS reads, giving you a higher sensitivity (fewer false negatives), but lower specificity (more false positives).

> <tip-title>Re-running with the same settings</tip-title>
>
> There's a shortcut to avoid setting every parameter the second time you run a tool.
>
> 1. In your history, click on an output of the first run to expand it.
> 2. Click on the button with the circular "re-run" arrows.
> 3. Now the parameters will all be the same as the last run. All you have to do is change the input file(s).
{: .tip}

## Re-running analyses with workflows

Instead of manually re-running all the tools in the variant calling section, you can use a **workflow** to automatically run the same tools, but on the SSCS reads. Workflows let you run a chain of tools on different input data with a single click of a button. You can find more information on using workflows in the [Galaxy 101 introductory tutorial](../../../introduction/tutorials/galaxy-intro-101/tutorial.html#run-workflow-on-different-data).

We've prepared two workflows which split the above analysis into two steps:

1. [Using Du Novo](./workflows/) to create consensus sequences from raw reads.
  - This will generate trimmed DCS and SSCS files from raw sequencing data.
  - This does not include the FastQC step. You should always run FastQC on your raw reads first, to check the quality of your sequencing run before proceeding with the analysis.

   > <comment-title>Helping Du Novo</comment-title>
   > The {% icon param-check %} *Send usage data* option is left off in the above workflow.
   > This is because we want to make sure you only share data knowingly.
   >
   > But again, if you'd like to help improve Du Novo, consider turning it on.
   {: .comment}

2. [Calling variants](./workflows/) from consensus sequences.
  - This takes a pair of FASTQ files and calls variants using them.
  - If you'd like variants from both DCS and SSCS, you'll have to run this twice, once on each.
  - **N.B.** Remember that this workflow is designed for the above *ABL1* analysis. If you want to use it for any other dataset, you'll have to change the relevant options.

You can use the variant calling workflow to call variants using the SSCS instead of the DCS.


# Conclusion

You should now understand duplex sequencing, rare variants, and be able to process the former to find the latter.

### If things don't work...
...you need to complain. Use [Galaxy's Help Forum](https://help.galaxyproject.org/) to do this.


topic: variant-analysis
tutorial: aiv-analysis
title: Avian influenza viral strain analysis from gene segment sequencing data
difficulty: intermediate
time_estimation: 4H
tools_json: []
requirements_json: []
tags_json: ["virology", "one-health"]
content_hash: 1f8c4b3c91fff0e4c28aa78fd8bfe2d2
last_modified: 2026-02-26T21:12:18.612654
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/aiv-analysis/tutorial.html

content:
Of the four species of influenza viruses (Influenza A-D), *Influenza A* is the most virulent in human hosts and subtypes of it have been responsible for all historic flu pandemics.

The different influenza species infect distinct animal hosts (though all of them can infect humans and pigs) and the most important natural reservoir for *Influenza A* are birds (in particular, wild aquatic birds), in which it causes **Avian influenza**.

Importantly, all flu pandemics of the last century have been triggered by **reassortment** events between avian and human *Influenza A* strains (although at least the 2009 swine flu pandemic involved an additional mixing event with an *Influenza A* strain from pigs).

Reassortment events are a key trait of *Influenza A* made possible by its wide range of natural hosts combined with two molecular characteristics of the species:

1. the linear negative-sense single-stranded RNA genomes of influenza viruses (and of all viruses of the *Orthomyxoviridae* family) are segmented, i.e. consist of several (eight in the case of *Influenza A* and *B*) distinct pieces of RNA, which typically encode just one, sometimes two viral proteins.
2. the mutation rate in the genome is high in influenza viruses, and particularly high in *Influenza A*, since their RNA polymerase lacks exonuclease activity necessary for proof-reading.

Together these characteristics enable *Influenza A* to evolve relatively rapidly in non-human hosts, then re-adapt to humans through exchange of segments (reassortment) in a host co-infected with a human and, *e.g.*, an avian strain. If such a reassortment introduces new variants of the two most antigenic proteins of influenza, HA (hemagglutinin) and NA (neuraminidase), this antigenic shift provides the reassorted strain with immune escape potential, which, if it happens in a genetic background compatible with efficient transmission in humans, can trigger an unusually strong wave of influenza or even a new pandemic.

In order to estimate the likelihood of an epidemic or pandemic influenza event in the near future it is, therefor, important to monitor closely the genome composition of circulating Avian Influenza strains in wild and domestic birds and the huge technological advances over the last decade make it possible to use next-generation sequencing for this purpose.

At the same time, the segmented nature of their genomes combined with high genetic variability, especially in the HA and NA genes, requires rather specialized bioinformatics workflows for the analysis of data from influenza viruses compared to other viruses with similar genome size (like, *e.g.* SARS-CoV-2):

The viral surface proteins HA and (to a lesser extent) NA are the main targets of the host antibody response and are, thus, under constant selection pressure to mutate into forms capable of evading an existing host immune response. As a consequence, these segments have evolved into a much richer panel of sequence variants than the other segments, to the point that sequences of the HA segment can, at present, be classified into 18 distinct subfamilies, H1-H18, while there are 11 recognized subfamilies of NA, and *Influenza A* strains are subtyped (as, for example, H5N1, H3N2, etc.) according to the combination of HA- and NA-encoding segments they are carrying.

Importantly, the sequence diversity of HA and (again to a lesser extent) of NA segments is big enough to prevent a naive approach of mapping sequenced reads to one specific agreed-upon *Influenza A* reference sequence. While this would work for the other six segments, mapping software would regularly fail to find enough plausible mappings for sequenced reads of HA and NA origin to continue analysis with. This is why, in this tutorial we are going to explore an alternative approach, which is also mapping-based but chooses a suitable reference for each segment dynamically based on the input sequencing data.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare analysis history and data

In this tutorial you are going to work on a single avian influenza sample sequenced in paired-end mode on the Illumina platform, *i.e.* we are going to download two datasets of sequenced reads for that sample.

In addition, we are going to base the analysis on a small collection of multiple reference sequences for each influenza gene segment. We prepared this collection for you from public [INSAFlu](https://insaflu.insa.pt/dashboard/) [data](https://insaflu.readthedocs.io/en/latest/uploading_data.html?highlight=references#uploading-reference-data).

If you have your own curated collection of reference sequences, you should be able to use it to follow this tutorial without any problem. Note, however, that the analysis results referred to in many of the questions will be different if you are exchanging the reference collection.
If you want to learn how you can create a Galaxy collection from your own references that is structured like the default we are suggesting here, you may want to follow the dedicated tutorial on [Using dataset collections]({% link topics/galaxy-interface/tutorials/collections/tutorial.md %}).

# Get a history with reference data

Any analysis should get its own Galaxy history, but in this tutorial we *won't* start with creating one.

Instead, to save some time, you can copy an existing shared history on Galaxy Europe.

> <hands-on-title>Prepare the Galaxy history</hands-on-title>
>
> 1. On Galaxy Europe, open the [shared history](https://usegalaxy.eu/published/history?id=0fc6b637aadbc8a2) with pre-prepared INSAFlu reference data.
>
>    On the Galaxy US server, use this [shared history](https://usegalaxy.org/published/history?id=31cb33f8e6126823) instead.
>
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md name="Avian influenza tutorial" %}
>
{: .hands_on}

## Get the sequencing data

> <hands-on-title>Get the data</hands-on-title>
>
> 1. Import the forward and reverse reads of the sequenced sample into your history and organize them into a *Paired Collection*.
>
>    ```
>    https://usegalaxy.eu/api/datasets/4838ba20a6d86765171e4f201205515c/display?to_ext=fastqsanger.gz
>    https://usegalaxy.eu/api/datasets/4838ba20a6d867657d09280baae4e1ec/display?to_ext=fastqsanger.gz
>    ```
>
>    1. Copy the links above
>    2. Open the {% tool [Upload](upload1) %} Manager
>    3. In the top row of tabs select **Collection**
>    4. Configure the drop-down select boxes on that tab like this:
>       - *"Collection Type"*: `Pair`
>       - *"File Type"*: `fastqsanger.gz`
>    5. Click on **Paste/Fetch data** and paste the links you copied into the empty text box
>    6. Press **Start**
>    7. Wait for the **Build** button to become enabled, click it, and, in the next dialogue, give a suitable **Name**, like `Sequencing data`, to the new collection.
>    8. Click on **Create collection**
>
{: .hands_on}

## Inspect the reference data

> <hands-on-title>What is in the reference collection?</hands-on-title>
>
> 1. Step inside the imported reference collection by clicking on it.
> 2. Expand individual elements of the collection by clicking on them to reveal more details about them.
> 3. View the content of any element by clicking its {% icon galaxy-eye %}.
>
>    You can then scroll through the data and use your browser's search function to look for particular sequences.
>
>    > <question-title>Questions</question-title>
>    >
>    > 1. How many elements are in the collection? What do they represent?
>    > 2. How many sequences are stored in each element?
>    > 3. Are all subtypes of HA and NA represented in the sequences?
>    > 4. Are there sequences of non-A Influenza species?
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. The collection consists of eight elements and each element provides reference sequences for one genome segment.
>    > > 2. Each element has data for 56 sequences (Galaxy knows how to interpret the fasta format of the sequencing data and counts sequences for you; you can see the count for any of the elements by expanding it).
>    > > 3. The collection has reference sequence data for H1-H18 (with the exception of H11) and for N1-N11. For many of those subtypes, however, there is only one reference, i.e., within-subtype variation is not captured well by the collection.
>    > > 4. There are 6 *Influenza B* references in the collection. Since *Influenza B* is not normally seen in birds and we are going to analyze an avian influenza sample here, these can be considered controls: if we get an *Influenza B* assignment for the sample at any step in our analysis, we know something is very suspicious. This also means that we only have 50 *informative* references in the collection.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

{% icon galaxy-info %} There are lots of text processing and filtering tools for Galaxy, of which this tutorial is going to introduce just a small subset. You could use them to extract statistics about which subtypes are found how many times in the collection of references instead of relying on scrolling and searching. If you are interested and have the time, you may want to try the [Data Manipulation Olympics]({% link topics/introduction/tutorials/data-manipulation-olympics/tutorial.md %}) to learn more about available such tools and how to combine them.

# Quality control

As a very first step, we would like to make sure that we base our analysis only on the high-quality parts of the sequenced reads.

NGS reads often have lower base calling quality near their ends, so here we are going to trim low-quality stretches of bases from both ends. In addition, we are going to discard reads shorter than 30 bases after trimming. This is reasonable since our next step will involve extracting all possible 21-mers from the reads and there are very few possibilities for a read shorter than 30 bases.

The tool **fastp** lets us perform these tasks and obtain a nice quality report for our reads before and after processing in one go, but many other options exist to perform sequenced reads quality control and trimming/filtering in Galaxy and the dedicated tutorial on [quality control]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}) introduces more of them.

> <hands-on-title>QC and read trimming/filtering with fastp</hands-on-title>
>
> 1. {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.23.4+galaxy1) %}
>    - *"Single-end or paired reads"*: `Paired Collection`
>    - *"Select paired collection(s)"*: the uploaded paired collection of sequenced reads
>    - In *"Filter Options"* under *"Length filtering options"*:
>      - *"Length required"*: `30`
>    - In *"Read Modification Options"* under *"Per read cutting by quality options"*:
>      - *"Cut by quality in front (5')"*: `Yes`
>      - *"Cut by quality in tail (3')"*: `Yes`
>      - *"Cutting mean quality"*: `30`
>
>    > <details-title>fastp default settings</details-title>
>    >
>    > fastp comes with basic default settings for some of its many parameters.
>    > In particular, with no explicit parameter specifications, the tool will still remove reads that
>    > - have base qualities < 15 for more than 40% of their bases
>    > - have more than 5 Ns in their sequence
>    > - have a length < 15 bases (after removing poly-G tails from Illumina reads)
>    >
>    >   Because our input data has far longer MiSeq-generated reads, we can increase this length threshold to 30 bases as done above.
>    >
>    > These are very relaxed criteria that will only remove the least usable reads, but still assure minimal
>    > quality standards that are good enough for many (including training) purposes.
>    {: .details}
>
>    Running the tool will result in two outputs. One is a new paired collection with the processed reads, the other one is a report of initial quality, the processing actions performed and their effect on key quality metrics.
>
> 2. {% icon galaxy-eye %} **Inspect** the report and try to answer the following questions:
>
>    > <question-title>Questions</question-title>
>    >
>    > 1. Which set of reads (forward or reverse reads) did profit most quality-wise from our low-quality base trimming?
>    > 2. What percentage of reads got discarded completely with our settings, and what percentage of bases?
>    > 3. Why, according to the Filtering result, have some reads been discarded because of too many Ns?
>    > 
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. The forward reads were of worse quality in particular near their 3'-ends than the reverse reads and, consequently, were affected more strongly by trimming (see the different "quality" plots in the report).
>    > > 2. Less than 2% of reads got discarded completely (the "Filtering result" section of the report says 98.03% of all reads were retained). In contrast, around 10% of all bases got discarded (compare total bases before and after filtering) so most reads got trimmed but not discarded.
>    > > 3. As explained in the "fastp default settings" box above, **fastp** is a tool with many hidden default actions in its various sections.
>    > >    Inside the *Filter Options* inspect *Quality filtering options* and the various defaults mentioned in the parameters help therein. One of these defaults is to discard reads with more than 5 Ns. These defaults are often reasonable, but it's good to be aware of them.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# Per-segment subtyping and hybrid reference construction

Our goal at this step is to find best matches between our sequencing data and the reference sequences of each genome segment. This will:
- give us preliminary subtyping results with regard to the HA and NA segments for the sequenced sample
- suggest the best reference to map the sequencing data to for each segment.

We are then going to combine the best reference of each segment into a hybrid reference genome to use for mapping our sequenced reads against.

To identify the best matching reference segments, we are going to run the tool **VAPOR** asking it to report the stats for any hits it identifies.

> <hands-on-title>Exploring best-matching reference segment scores</hands-on-title>
>
> 1. {% tool [VAPOR](toolshed.g2.bx.psu.edu/repos/iuc/vapor/vapor/1.0.2+galaxy3) %}
>    - {% icon param-collection %} *"Reference sequences"*: `References per segment (INSAFlu)`
>    - *"Type of sequencing data"*: `Paired-end as collection`
>    - *"Paired collection of sequenced reads"*: quality-trimmed reads; output of **fastp**
>    - *"Desired output"*: `Return scores of best matches`
>    - *"Limit number of reported matches to"*: `0`
>    - In *"Optional arguments"*:
>      - *"Read kmer filtering threshold"*: `0.1`
>      - *"Minimum k-mer proportion"*: `0.0`
>
> 2. {% icon galaxy-eye %} Explore the output collection produced by the tool
>
>    > <question-title>Questions</question-title>
>    >
>    > 1. Is there a difference between the results for segment 4 (encoding HA), segment 6 (encoding NA) and those for the remaining segments?
>    > 2. According to the tool, what is the likely subtype with regard to HA and NA of the sample?
>    > 
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. The values for `% of query bases in reads` and for `Total score` are drastically lower for the HA and somewhat lower for the NA segment than for any of the other segments. This could be due to the structure of the collection of references we have used, but also fits very well with the higher selection pressure on NA and, in particular on HA, as the main antigenic proteins of the virus, which leads to higher variability in these segments than in the others.
>    > > 2. The best match (assigned an almost 2x higher score than the second-best match) found for the HA segment is from an H4N6 strain. The top two matches with regard to the NA segment (again assigned ~ 2x higher scores than the next best matches) are from strains with the N6 subtype of NA.
>    > >
>    > >    The most likely subtype of the sample, thus, appears to be H4N6.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

Now that we have established that things *may* make sense, we can use the output of **VAPOR** to extract the actual sequence of the top hit for each reference segment. We then concatenate these best matches into a hybrid reference genome for mapping.

> <hands-on-title>Obtaining sequences of top hits identified by VAPOR</hands-on-title>
>
> 1. {% tool [Replace parts of text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.3+galaxy1) %} to extract just the name of the sequence from each line of VAPOR's output
>    - {% icon param-collection %} *"File to process"*: collection of score outputs of **VAPOR**
>    - In {% icon param-repeat %} *"1. Find and Replace"*:
>      - *"Find pattern"*: `^.+\t>(.+)$`
>      - *"Replace with"*: `$1`
>      - *"Find-Pattern is a regular expression"*: `Yes`
>
> 2. {% tool [Select first lines from a dataset](Show beginning1) %} to get the first line, i.e. the best match from VAPOR's output
>    - *"Select first"*: `1` lines
>    - {% icon param-collection %} *"from"*: output of **Replace**
>
> 3. {% tool [seqtk_subseq](toolshed.g2.bx.psu.edu/repos/iuc/seqtk/seqtk_subseq/1.4+galaxy0) %} to extract the reference sequences based on their names reported by VAPOR
>    - {% icon param-collection %} *"Input FASTA/Q file"*: `References per segment (INSAFlu)`
>    - *"Select source of sequence choices"*: `FASTA/Q ID list`
>    - {% icon param-collection %} *"Input ID list"*: the collection output of **Select first**
>
> 4. {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0) %} to combine the best-matching sequence of each segment into one dataset
>    - {% icon param-collection %} *"Collection of files to collapse into single dataset"*: the selected sequences collection produced by **seqtk_subseq**
>
{: .hands_on}

At this point, you may want to {% icon galaxy-eye %} inspect the output of the last step to see if it is an eight-segments reference genome as expected, and if the segments correspond to the top hits found by VAPOR.

# Mapping to a hybrid reference

If things went well, the hybrid reference we just obtained should be close enough across all segments to our sample to allow successful mapping of reads. Before we start the mapping we may want to truncate the segment names in our hybrid reference genome though because currently these names still reflect the full origin of the segment sequences, but from now on we are fine with just the segment ID. We can use the **Replace** tool from before again to truncate the names.

> <hands-on-title>Shortening sequence titles</hands-on-title>
>
> 1. {% tool [Replace parts of text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.3+galaxy1) %}
>    - {% icon param-file %} *"File to process"*: the hybrid reference genome; output of **Collapse Collection**
>    - In {% icon param-repeat %} *"1. Find and Replace"*:
>      - *"Find pattern"*: `^>([^|]+).+$`
>      - *"Replace with"*: `>$1`
>      - *"Find-Pattern is a regular expression"*: `Yes`
>
> 2. Add a tag to the output dataset
>
>    This is the hybrid reference genome that we are going to use for mapping now.
>    It is an important intermediate result, to which we will need to return later, and to make that easier we are going to add a tag to it now for extra visibility in the history.
>
>    1. Click on the dataset to expand it
>    2. Click on {% icon galaxy-tags %} **Add Tags**
>    3. Enter a tag like `HybridRef` (tags have to consist of a single word)
>    4. Confirm your choice by pressing Enter on the keyboard or clicking on the new name.
>
{: .hands_on}

Having polished the titles of the segments in our hybrid reference genome we are finally ready for mapping, which we will carry out with **BWA-MEM**, clean up a bit with **Samtools view** and produce a quality report for with **QualiMap BamQC**.

> <hands-on-title>Read mapping and quality control</hands-on-title>
>
> 1. {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.18) %}
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>    - {% icon param-file %} *"Use the following dataset as the reference sequence"*: hybrid reference genome with shortened names; output of **Replace**
>    - *"Single or Paired-end reads"*: `Paired Collection`
>    - *"Select a paired collection"*: quality-trimmed reads; output of **fastp** in QC and Trimming
>
> 2. {% tool [Samtools view](toolshed.g2.bx.psu.edu/repos/iuc/samtools_view/samtools_view/1.20+galaxy3) %}
>    - {% icon param-file %} *"SAM/BAM/CRAM data set"*: mapped reads BAM dataset; output of **BWA-MEM**
>    - *"What would you like to look at?"*: `A filtered/subsampled selection of reads`
>    - In *"Configure filters"*:
>      - *"Filter by quality"*: `20`
>      - *"Require that these flags are set"*: `Read is paired` and `Read is mapped in a proper pair`
>
> 3. {% tool [QualiMap BamQC](toolshed.g2.bx.psu.edu/repos/iuc/qualimap_bamqc/qualimap_bamqc/2.2.2d+galaxy3) %}
>    - {% icon param-file %} *"Mapped reads input dataset"*: filtered mapped reads BAM dataset; output of **Samtools view**
>    - *"Skip duplicate reads"*: `Unselect all`
>    - In *"Settings affecting specific plots"*:
>      - *"Number of bins to use in across-reference plots"*: `40`
>
> 4. {% icon galaxy-eye %} Study the report generated with QualiMap
>
>    > <question-title>Questions</question-title>
>    >
>    > 1. What is the coverage of each segment by the sequenced reads, and is it uniform?
>    > 2. Look for a plot showing read mapping quality across the reference. What can you conclude?
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. Coverage is rather different for the different segments. Looking at "Mean coverage" and its "Standard deviation" in the "Chromosome stats" table and at the "Coverage across reference" plot, coverage of the HA segment seems to be most critical since it approaches zero in some regions.
>    > > 2. Mapping quality is almost constant at or very near 60 (which happens to be the maximum mapping quality value emitted by BWA-MEM) across all segments with the exception of HA.
>    > >
>    > > These two observations combined with the VAPOR statistics for HA show again that even the best matching reference for this segment is not exactly close to the sequence of our sample. The read mapper had difficulties placing the sequenced reads on that sub-optimal reference and it looks as if we might have very little sequence information for some HA regions.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# Consensus sequence construction

From the polished mapping of reads to our custom reference we can now construct the consensus sequence of our sample.

Unfortunately, the tool we are going to use for this, **ivar consensus**, is not capable of working with more than one reference name at a time, but because this is influenza data we have mappings to 8 different segments described in our data. So we need to take a little detour and split the mapped reads data into a collection of datasets each containing the mappings for just one segment first again, then perform the consensus construction for all of them in parallel.

> <hands-on-title>Splitting mapped reads by genome segment</hands-on-title>
>
> 1. {% tool [Split BAM by Reference](toolshed.g2.bx.psu.edu/repos/iuc/bamtools_split_ref/bamtools_split_ref/2.5.2+galaxy2) %}
>    - {% icon param-file %} *"BAM dataset to split by reference"*: filtered mapped reads; output of **Samtools view**
>
{: .hands_on}

The output from this step has the desired collection structure, but the names of the collection elements are not the nicest. Ideally, we would just reuse the segment names, which are already provided in our mapping reference genome. So lets extract these names again and use them as new element labels

> <hands-on-title>Relabeling collection elements</hands-on-title>
>
> 1. {% tool [Select lines that match an expression](Grep1) %}
>    - {% icon param-file %} *"Select lines from"*: the mapping reference; output of **Replace** in Mapping to a hybrid reference
>
>      {% icon tip %} This is the dataset you should have {% icon galaxy-tags %} tagged before. Lets hope you did!
>    - *"that"*: `Matching`
>    - *"the pattern"*: `^>.+`
>
> 2. {% tool [Replace parts of text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.3+galaxy1) %}
>    - {% icon param-file %} *"File to process"*: output of **Select**
>    - In {% icon param-repeat %} *"1. Find and Replace"*:
>      - *"Find pattern"*: `^>(.+)$`
>      - *"Replace with"*: `$1`
>      - *"Find-Pattern is a regular expression"*: `Yes`
>
> 3. {% tool [Relabel identifiers](__RELABEL_FROM_FILE__) %}
>    - {% icon param-file %} *"Input collection"*: the split mappings; output of **Split BAM by Reference**
>    - *"How should the new labels be specified?"*: `Using lines in a simple text file.`
>    - {% icon param-file %} *"New Identifiers"*: output of **Replace**
>    - *"Ensure strict mapping"*: `Yes`
>
{: .hands_on}

And with that we are ready for consensus sequence generation!

To accept any base suggested by the mapped sequenced reads as the consensus base for the corresponding genome position, we ask for the following requirements to be fulfilled:

- at least ten sequenced reads have to provide information about the base in question
- at a minimum, 70% of these reads have to agree on the base at this position.

To avoid getting misled too much by sequencing errors, we are also going to ignore bases with a base calling quality less than 20 in the above counts (i.e., we are going to base our decisions only on bases in sequenced reads that the basecaller of the sequencer was reasonably sure about.

Now what if we cannot obtain a consensus base for a position with the above criteria? In such cases of uncertainty we want to insert an N (i.e. an unknown base) to express that we either did not have enough information about the position or that this information was ambiguous.

{% icon galaxy-info %} All of the above limits for consensus base calling are arbitrary to some degree, and depend somewhat on the quality of the sequencing data. With very high overall coverage, for example, it is possible to increase the coverage threshold, but if you increase that threshold too much, you may end up with a consensus sequence consisting mostly of Ns.

> <hands-on-title>Per-segment consensus construction</hands-on-title>
>
> 1. {% tool [ivar consensus](toolshed.g2.bx.psu.edu/repos/iuc/ivar_consensus/ivar_consensus/1.4.3+galaxy0) %}
>    - {% icon param-collection %} *"Bam file"*: the relabeled collection of mapped reads; output of **Relabel identifiers**
>    - *"Minimum quality score threshold to count base"*: `20`
>    - *"Minimum frequency threshold"*: `0.7`
>    - *"Minimum indel frequency threshold"*: `0.8`
>    - *"Minimum depth to call consensus"*: `10`
>    - *"How to represent positions with coverage less than the minimum depth threshold"*: `Represent as N (-n N)`
>
>    The output is a consensus sequence in FASTA format, one per segment, with
>    the names just providing a bit too much detail for our purpose.
>
> 2. {% tool [Replace parts of text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.3+galaxy1) %}
>    - {% icon param-collection %} *"File to process"*: consensus sequences; output of **ivar consensus**
>    - In {% icon param-repeat %} *"1: Find and Replace"*:
>      - *"Find pattern"*: `^>Consensus_(.*)_threshold_.*`
>      - *"Replace with"*: `>$1`
>      - *"Find-Pattern is a regular expression"*: `Yes`
>
> 3. {% icon galaxy-eye %} Inspect each consensus sequence generated for the different segments
>
>    > <question-title></question-title>
>    >
>    > Does everything look ok?
>    >
>    > > <solution-title>Answer</solution-title>
>    > >
>    > > As expected from the findings so far, the consensus sequence for the HA segment has stretches of Ns in it, which likely reflect the mapping issues and associated loss of coverage caused by our insufficiently sized collection of references.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# Placing segments on a phylogenetic tree

The next logical step after obtaining the consensus sequences of segments of our sample is to explore how those sequences are related to the sequences in our reference collection.
To do so, we are going to combine the reference sequences of all segments with their corresponding consensus sequence into one multiple sequence alignment (MSA) per segment, and use these to generate phylogenetic trees, again one per segment. We are going to use two rather standard tools, **MAFFT** and **IQTree**, for generating MSAs and trees, respectively.

> <hands-on-title>Exploring phylogeny</hands-on-title>
>
> 1. {% tool [MAFFT](toolshed.g2.bx.psu.edu/repos/rnateam/mafft/rbc_mafft/7.526+galaxy0) %}
>    - *"For multiple inputs generate"*: `one or several MSAs depending on input structure`
>      - In *"Input batch"*:
>        - {% icon param-repeat %} *"1: Input batch"*
>          - {% icon param-collection %} *"Sequences to align"*: collection of `References per segment (INSAFlu)`
>        - {% icon param-repeat %} *"2: Input batch"*
>          - {% icon param-collection %} *"Sequences to align"*: collection of renamed consensus sequences; output of **Replace** on consensus sequences
>    - *"Type of sequences"*: `Nucleic acids`
>
>    Because both input batches are collections of eight elements each, the result is also a collection of eight MSAs, each aligning all reference sequences of one genome segment plus the consensus sequence we have obtained for that segment against each other.
>
> 2. {% tool [IQ-TREE](toolshed.g2.bx.psu.edu/repos/iuc/iqtree/iqtree/2.3.6+galaxy0) %}
>    - {% icon param-collection %} *"Specify input alignment file in PHYLIP, FASTA, NEXUS, CLUSTAL or MSF format."*: output of **MAFFT**
>    - *"Specify sequence type ..."*: `DNA`
>
> 3. {% icon galaxy-eye %} Explore each of the final trees produced by IQTree for the different segments
>
>    There are two ways to explore the trees in Galaxy:
>
>    One option is to open the *Report and Final Tree* datasets produced by IQ-Tree.
>    When you scroll down through their content, you will find that these feature simple text representations of the trees.
>
>    Another option is to expand the *MaxLikelihood Tree* datasets generated by IQ-Tree, then click on {% icon galaxy-barchart %} **Visualize**.
>    This will let you launch the *Phylogenetic Tree Visualization* in the middle panel, which lets you explore the tree interactively and in different representations.
>
>    > <question-title></question-title>
>    >
>    > What are your conclusions about the sample in general and its HA and NA segments in particular?
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > - For most of its segments the sample resembles relatively recent (from the last decade) Eurasian reference sequences.
>    > > - For HA and NA the sample clusters with the few available samples of the corresponding subtype.
>    > > - None of the references closest to the sample with respect to HA and NA are close to the recent Eurasian reference cluster for their remaining segments.
>    > > - A plausible explanation is that the H4 and N6 segments of the sample have been brought into the recent Eurasian background through a reassortment event. Caveat: interpretations like this can be heavily influenced by the size of the reference collection!
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# Conclusion

Analysis workflows for influenza whole-genome sequencing data need to take into account the specific characteristics of the viral genome. Due to their higher natural variability this is especially true for avian influenza samples and for the HA- and NA-encoding segments of the genome.

Nevertheless, it looks possible, with carefully chosen reference segment sequences and bioinformatic tools, to avoid a computationally expensive de-novo assembly approach and to use mapping against a dynamically compiled reference genome instead.

The rather small reference segment collection suggested for this tutorial consists of 56 different samples, of which only a single one has the H4N6 subtype of the sample analyzed here. Still it allowed us to perform subtyping of the sample, to construct complete consensus sequences for 7 segments including NA, and to draw valuable conclusions about the origin of the sample. It is conceivable that a larger collection of references chosen to capture several strains from each HA subtype could solve the remaining issue of the incomplete HA consensus sequence.


topic: variant-analysis
tutorial: non-dip
title: Calling variants in non-diploid systems
difficulty: intermediate
time_estimation: 1h30m
tools_json: []
requirements_json: []
tags_json: ["prokaryote", "microgalaxy"]
content_hash: 82de85cfcbe606758a62d97c874bfbde
last_modified: 2026-02-26T21:12:18.623218
zenodo_link: https://doi.org/10.5281/zenodo.1251112
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/non-dip/tutorial.html

content:
The majority of life on Earth is non-diploid and represented by prokaryotes, viruses, and their derivatives, such as our own mitochondria or plant's chloroplasts. In non-diploid systems, allele frequencies can range anywhere between 0 and 100% and there could be multiple (not just two) alleles per locus. The main challenge associated with non-diploid variant calling is the difficulty in distinguishing between the sequencing noise (abundant in all NGS platforms) and true low frequency variants. Some of the early attempts to do this well have been accomplished on human mitochondrial DNA although the same approaches will work equally good on viral and bacterial genomes ({% cite Rebolledo-Jaramillo2014 %}, {% cite Li2015 %}).

As an example of non-diploid systems, we will be using human mitochondrial genome. However, this approach will also work for most bacterial and viral genomes.

There are two ways one can call variants:

1. By comparing reads against an existing genome assembly
1. By assembling a genome first and then mapping against that assembly

![2 approaches](../../images/ref_vs_assembly.jpg "This figure from {% cite Olson2015 %} contrasts the two approaches.")

In this tutorial we will take the *first* path, in which we map reads against an existing assembly. Later in the course, after we learn about assembly approaches, we will try the second approach as well.

The goal of this example is to detect heteroplasmies - variants within mitochondrial DNA. Mitochondria are transmitted maternally, and heteroplasmy frequencies may change dramatically and unpredictably during the transmission due to a germ-line bottleneck ({% cite Cree2008 %}). As we mentioned above, the procedure for finding variants in bacterial or viral genomes will be essentially the same.

Datasets representing a child and a mother are available in [Zenodo](https://doi.org/10.5281/zenodo.1251112). These datasets were obtained by paired-end Illumina sequencing of human genomic DNA enriched for mitochondria. The enrichment was performed using long-range PCR with two primer pairs that amplify the entire mitochondrial genome. Samples will therefore still contain a lot of DNA from the nuclear genome, which, in this case, is a contaminant.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Importing data

For this tutorial we have prepared a subset of data previously by our group ({% cite Rebolledo-Jaramillo2014 %}). Let's import these data into Galaxy. They are available from [this Galaxy Library](https://usegalaxy.org/library/folders/Fe4842bd0c37b03a7) or via [Zenodo](https://zenodo.org/record/582600)

> <hands-on-title>Get the data</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a meaningful name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 1. Import files from [Zenodo](https://zenodo.org/record/1251112):
>
>    ```
>    https://zenodo.org/record/1251112/files/raw_child-ds-1.fq
>    https://zenodo.org/record/1251112/files/raw_child-ds-2.fq
>    https://zenodo.org/record/1251112/files/raw_mother-ds-1.fq
>    https://zenodo.org/record/1251112/files/raw_mother-ds-2.fq
>    ```
>
> 1. Check that all newly created datasets in your history are assigned datatype
>    `fastqsanger`, and fix any missing or wrong datatype assignment
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="fastqsanger" %}
>
{: .hands_on}

## Checking data quality

Before proceeding with the analysis, we need to find out how good the data actually is. For this will use [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/).

> <hands-on-title>Assess quality of data</hands-on-title>
>
> 1. Run {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72+galaxy1) %} on each of the four FASTQ datasets with the following parameters:
>       - {% icon param-files %} *"Short read data from your current history"*: all 4 FASTQ datasets selected with **Multiple datasets**
>
>    {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>
{: .hands_on}

Once the FastQC jobs runs, you will be able to look at the HTML reports generated by this tool. The data have generally high quality in this example:

![FastQC output plot](../../images/mt_qc_plot.png "FastQC plot for one of the mitochondrial datasets shows that qualities are acceptable for 250 bp reads (mostly in the green, which is at or above [Phred score](https://en.wikipedia.org/wiki/Phred_quality_score) of 30).")

# Mapping reads to a reference

Our reads are long (250 bp) so we will use BWA-MEM ({% cite Li2013 %}) to align them against the reference genome as it has good mapping performance for longer reads (100bp and up).

> <hands-on-title>Map reads</hands-on-title>
>
> 1. Use {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.1) %} to map the reads to the reference genome with the following parameters:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>        - *"Using reference genome"*: `Human: hg38` (or a similarly named option)
>    - *"Single or Paired-end reads"*: `Paired`
>       - {% icon param-files %} *"Select first set of reads"*: select both `-1` datasets selected with **Multiple datasets**
>       - {% icon param-files %} *"Select second set of reads"*: select both `-2` datasets selected with **Multiple datasets**
>    - *"Set read groups information?"*: `Set read groups (SAM/BAM specification)`
>      - *"Auto-assign"*: `Yes`
>      - *"Auto-assign"*: `Yes`
>      - *"Platform/technology used to produce the reads (PL)"*: `ILLUMINA`
>      - *"Auto-assign"*: `Yes`
>
{: .hands_on}

> <comment-title>More about selecting datasets</comment-title>
>
> By selecting datasets 1 and 3 as **Select the first set of reads** and datasets 2 and 4 as **Select the second set of reads**, Galaxy will automatically launch two BWA-MEM jobs using datasets 1,2 and 3,4 generating two resulting BAM files.
> By setting **Set read groups information** to `Set read groups (SAM/BAM specifications)` and clicking **Auto-assign** we will ensure that the reads in the resulting BAM dataset are properly set.
{: .comment}

# Postprocessing mapped reads

## Merging BAM datasets

Because we have set read groups, we can now merge the two BAM dataset into one. This is because read groups label each read as belonging to either *mother* or *child*.

> <hands-on-title>Merge multiple datasets into one</hands-on-title>
>
> 1. Use {% tool [MergeSAMFiles](toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_MergeSamFiles/2.18.2.1) %} to merge the BAM datasets with the following parameters:
>    - {% icon param-files %} *"Select SAM/BAM dataset or dataset collection"*: select both BAM datasets produced by **BWA-MEM** {% icon tool %}
>    - *"Select validation stringency"*: `Lenient`
>
{: .hands_on}

## Removing duplicates

Preparation of sequencing libraries (at least at the time of writing) for technologies such as Illumina (used in this example) involves PCR amplification. It is required to generate sufficient number of sequencing templates so that a reliable detection can be performed by base callers. PCR has its own biases which are especially profound in cases of multi-template PCR used for construction of sequencing libraries ({% cite Kanagawa2003 %}).

Duplicates can be identified based on their outer alignment coordinates or using sequence-based clustering. One of the common ways for identification of duplicate reads is the **MarkDuplicates** utility from [Picard](https://broadinstitute.github.io/picard/command-line-overview.html) package which is designed to identify both PCR and optical duplicates.

> <details-title>More about MarkDuplicates</details-title>
>
> Duplicates are identified as read pairs having identical 5' positions (coordinate and strand) for both reads in a mate pair (and optionally, matching unique molecular identifier reads; see `BARCODE_TAG` option). Optical, or more broadly sequencing, duplicates are duplicates that appear clustered together spatially during sequencing and can arise from optical/imagine-processing artifacts or from biochemical processes during clonal amplification and sequencing; they are identified using the `READ_NAME_REGEX` and the `OPTICAL_DUPLICATE_PIXEL_DISTANCE` options. The tool's main output is a new SAM or BAM file in which duplicates have been identified in the SAM flags field, or optionally removed (see `REMOVE_DUPLICATE` and `REMOVE_SEQUENCING_DUPLICATES`), and optionally marked with a duplicate type in the 'DT' optional attribute. In addition, it also outputs a metrics file containing the numbers of `READ_PAIRS_EXAMINED`, `SECONDARY_OR_SUPPLEMENTARY_RDS`, `UNMAPPED_READS`, `UNPAIRED_READS`, `UNPAIRED_READ DUPLICATES`, `READ_PAIR_DUPLICATES`, and `READ_PAIR_OPTICAL_DUPLICATES`.
>
> Usage example: `java -jar picard.jar MarkDuplicates I=input.bam O=marked_duplicates.bam M=marked_dup_metrics.txt`
{: .details}

> <hands-on-title>De-duplicate mapped data</hands-on-title>
>
> 1. Use {% tool [MarkDuplicates](toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_MarkDuplicates/2.18.2.2) %} to de-duplicate the merged BAM datasets with the following parameters:
>    - {% icon param-file %} *"Select SAM/BAM dataset or dataset collection"*: select the merged BAM dataset produced by **MergeSAMFiles** {% icon tool %}
>    - *"The scoring strategy for choosing the non-duplicate among candidates"*: `SUM_OF_BASE_QUALITIES`
>    - *"The maximum offset between two duplicate clusters in order to consider them optical duplicates"*: `100`
>    - *"Select validation stringency"*: `Lenient`
>
{: .hands_on}

MarkDuplicates produces a BAM dataset with duplicates removed and also a metrics file. Let's take a look at the metrics data:

```
raw_child-ds-	55	27551	849	55	50	1658	1	0.06103	219750
raw_mother-ds-	95	54973	1951	95	89	4712	2	0.08645	302188
```

Column headers are:

- LIBRARY (read group in our case)
- UNPAIRED_READS_EXAMINED
- READ_PAIRS_EXAMINED-
- SECONDARY_OR_SUPPLEMENTARY_RDS
- UNMAPPED_READS
- UNPAIRED_READ_DUPLICATES
- READ_PAIR_DUPLICATES
- READ_PAIR_OPTICAL_DUPLICATES
- PERCENT_DUPLICATION
- ESTIMATED_LIBRARY_SIZE

> <question-title></question-title>
>
> 1. What percent of read duplication are found in each read group (child and mother)?
>
> > <solution-title></solution-title>
> >
> > 1. The two datasets have ~6% and ~9% duplicates for child and mother, respectively.
> >
> {: .solution}
{: .question}

## Left-aligning indels

Left aligning of indels (a variant of re-aligning) is extremely important for obtaining accurate variant calls. For illustrating how left-aligning works, we expanded on an example provided by {% cite Tan2015 %}. Suppose you have a reference sequence and a sequencing read:

```
Reference GGGCACACACAGGG
Read      GGGCACACAGGG
```

If you look carefully you will see that the read is simply missing a `CA` repeat. But it is not apparent to a mapper, so some of possible alignments and corresponding variant calls include:

```
Alignment                 Variant Call

GGGCACACACAGGG            Ref: CAC
GGGCAC--ACAGGG            Alt: C

GGGCACACACAGGG            Ref: ACA
GGGCA--CACAGGG            Alt: A

GGGCACACACAGGG            Ref: GCA
GGG--CACACAGGG            Alt: G
```

The last of these is *left-aligned*. In this case gaps (represented by dashes) are moved as far left as possible. For a formal definition of left-alignment and variant normalization, see {% cite Tan2015 %}.

> <hands-on-title>Left-align indels</hands-on-title>
>
> 1. Use {% tool [BamLeftAlign](toolshed.g2.bx.psu.edu/repos/devteam/freebayes/bamleftalign/1.3.1) %} to perform left alignment with the following parameters:
>    - *"Choose the source for the reference genome"*: `Locally cached`
>    - {% icon param-file %} *"Select alignment file in BAM format"*: select the BAM dataset produced by **MarkDuplicates** {% icon tool %}
>    - *"Using reference genome"*: `hg38`
>    - *"Maximum number of iterations'*: `5`
>
{: .hands_on}

## Filtering reads

Remember that we are trying to call variants in mitochondrial genome. Let focus only on the reads derived from mitochondria genome by filtering everything else out.

> <hands-on-title>Filter BAM data</hands-on-title>
>
> 1. Use {% tool [Filter BAM datasets on a variety of attributes](toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.4.1) %} with the following parameters:
>    - {% icon param-file %} *"BAM dataset(s) to filter"*: select the BAM dataset produced by **BamLeftAlign** {% icon tool %}
>    - In *"Condition"*:
>      - In *"1: Condition"*:
>        - In *"Filter"*:
>          - In *"1: Filter"*:
>            - *"Select BAM property to filter on"*: `mapQuality`
>            - *"Filter on read mapping quality (phred scale)"*: `>=20`
>          - In *"2: Filter"*:
>            - *"Select BAM property to filter on"*: `isPaired`
>            - *"Selected mapped reads"*: `Yes`
>          - Click on *"Insert Filter"*
>          - In *"3: Filter"*:
>            - *"Select BAM property to filter on"*: `isProperPair`
>            - *"Select reads with mapped mate"*: `Yes`
>          - Click on *"Insert Filter"*
>          - In *"4: Filter"*:
>            - *"Select BAM property to filter on"*: `reference`
>            - *"Select reads with mapped mate"*: `chrM`
>
>    > <comment-title>Filtering reads</comment-title>
>    > Further explanation of the filters used:
>    > - *mapQuality* is set to ≥20. Mapping quality reflects the probability that the read is placed incorrectly using [phred scale](https://en.wikipedia.org/wiki/Phred_quality_score). Thus 20 is 1/100 or 1% chance that the read is incorrectly mapped. By setting this parameter to ≥20, we will keep all reads that have 1% or less probability of being mapped incorrectly.
>    > - *isPaired* will eliminate singleton (unpaired) reads.
>    > - *isProperPair* will only keep reads that map to the same chromosome and are properly placed.
>    > - *reference* is set to the mitochondrial chromosome, chrM.
>    {: .comment}
{: .hands_on}

# Calling non-diploid variants

FreeBayes is widely used for calling variants in diploid systems. However, it can also be used for calling variants in pooled samples where the number of samples is not known. This is the exact scenario we have here: in our sample we have multiple mitochondrial (or bacterial or viral) genomes, but we do not know exactly how many. Thus we will use the `--pooled-continuous` option of FreeBayes to generate frequency-based variant calls as well as some other options highlighted below.

> <hands-on-title>Calling variants</hands-on-title>
>
> 1. Use {% tool [FreeBayes](toolshed.g2.bx.psu.edu/repos/devteam/freebayes/freebayes/1.3.1) %} to call variants with the following parameters:
>    - *"Choose the source for the reference genome"*: `Locally cached`
>    - *"Run in batch mode?"*: `Run individually`
>    - {% icon param-file %} *"BAM dataset"*: select the BAM dataset produced by last **Filter** {% icon tool %} step
>    - *"Using reference genome"*: `hg38`
>    - *"Limit variant calling to a set of regions?"*: `Limit to region`
>      - *"Region Chromosome"*: `chrM`
>      - *"Region Start"*: `1`
>      - *"Region End"*: `16000`
>    - *"Choose parameter selection level"*: `5: Full list of options`
>      - *"Population model options"*: `Set population model options`
>        - *"The expected mutation rate or pairwise nucleotide diversity among the population under analysis"*: `0.001`
>        - *"Set ploidy for the analysis"*: `1`
>        - *"Assume that samples result from pooled sequencing"*: `Yes`
>        - *"Output all alleles which pass input filters, regardless of genotyping outcome or model"*: `Yes`
>
>    > <comment-title>Population model options</comment-title>
>    > The "Population model options" are one of the most important parameter choices to make when calling variants in non-diploid systems.
>    {: .comment}
>
>      - *"Allelic scope options"*: `Set allelic scope options`
>        - *"Ignore SNP alleles"*: `No`
>        - *"Ignore indels alleles"*: `No`
>        - *"Ignore multi-nucleotide polymorphisms, MNPs"*: `Yes`
>        - *"Ignore complex events (composites of other classes)"*: `Yes`
>
>    > <comment-title>Allelic scope options</comment-title>
>    > Mitochondria has a number of low complexity regions (mononucleotide repeats). Setting the allelic scope parameters as described above will decrease noise from these regions.
>    {: .comment}
>
>      - *"Input filters"*: `Set input filters`
>        - *"Exclude alignments from analysis if they have a mapping quality less than"*: `20`
>        - *"Exclude alleles from analysis if their supporting base quality less than"*: `30`
>
>    > <comment-title>Filter options</comment-title>
>    > Setting *Exclude alignments from analysis if they have a mapping quality less than* to `20` (phred score of 20) will make FreeBayes only consider reliably aligned reads.
>    > Setting *Exclude alleles from analysis if their supporting base quality less than* to `30` (phred score of 30) will make FreeBayes only consider high quality bases.
>    {: .comment}
{: .hands_on}

FreeBayes will produce a [VCF](https://samtools.github.io/hts-specs/VCFv4.2.pdf) dataset similar to what is shown below (you may need to scroll sideways to see it in full). It lists 25 sites of interest. For brevity, the header lines have been removed:

```
#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO	FORMAT	raw_child-ds-	raw_mother-ds-
chrM	73	.	A	G	30438.6	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=949;CIGAR=1X;DP=950;DPB=950;DPRA=0;EPP=144.879;EPPR=5.18177;GTI=0;LEN=1;MEANALT=1;MQM=55.8314;MQMR=60;NS=2;NUMALT=1;ODDS=2187.64;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=34948;QR=37;RO=1;RPL=322;RPP=215.867;RPPR=5.18177;RPR=627;RUN=1;SAF=421;SAP=29.2075;SAR=528;SRF=0;SRP=5.18177;SRR=1;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:226:0,226:0:0:226:8314:-739.462,0	1:724:1,723:1:37:723:26634:-2346.86,0
chrM	263	.	A	G	11603.3	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=364;CIGAR=1X;DP=364;DPB=364;DPRA=0;EPP=16.755;EPPR=0;GTI=0;LEN=1;MEANALT=1;MQM=60;MQMR=0;NS=2;NUMALT=1;ODDS=982.556;PAIRED=1;PAIREDR=0;PAO=0;PQA=0;PQR=0;PRO=0;QA=13201;QR=0;RO=0;RPL=276;RPP=213.858;RPPR=0;RPR=88;RUN=1;SAF=172;SAP=5.39653;SAR=192;SRF=0;SRP=0;SRR=0;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:108:0,108:0:0:108:3896:-350.812,0	1:256:0,256:0:0:256:9305:-837.344,0
chrM	310	.	TCC	CCC	4471	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=150;CIGAR=1X2M;DP=171;DPB=180.333;DPRA=0;EPP=7.70068;EPPR=4.45795;GTI=0;LEN=1;MEANALT=2.5;MQM=59.88;MQMR=60;NS=2;NUMALT=1;ODDS=407.373;PAIRED=1;PAIREDR=1;PAO=11;PQA=237;PQR=0;PRO=0;QA=5147;QR=194;RO=6;RPL=113;RPP=86.6265;RPPR=8.80089;RPR=37;RUN=1;SAF=35;SAP=95.6598;SAR=115;SRF=1;SRP=8.80089;SRR=5;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:54:0,49:0:0:49:1645:-150.867,0	1:117:6,101:6:194:101:3502:-315.963,0
chrM	513	.	GCACACACACAC	GCACACACACACAC	2095.84	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=109;CIGAR=1M2I11M;DP=150;DPB=172.167;DPRA=0;EPP=109.173;EPPR=6.05036;GTI=0;LEN=2;MEANALT=3.5;MQM=60;MQMR=60;NS=2;NUMALT=1;ODDS=225.991;PAIRED=1;PAIREDR=1;PAO=5;PQA=101.5;PQR=101.5;PRO=5;QA=3824;QR=1196;RO=35;RPL=19;RPP=103.436;RPPR=4.56135;RPR=90;RUN=1;SAF=80;SAP=54.8268;SAR=29;SRF=24;SRP=13.4954;SRR=11;TYPE=ins;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:34:5,28:5:163:28:996:-74.9521,0	1:116:30,81:30:1033:81:2828:-161.475,0
chrM	750	.	A	G	47254.6	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=1459;CIGAR=1X;DP=1459;DPB=1459;DPRA=0;EPP=48.5904;EPPR=0;GTI=0;LEN=1;MEANALT=1;MQM=59.8705;MQMR=0;NS=2;NUMALT=1;ODDS=3230.11;PAIRED=1;PAIREDR=0;PAO=0;PQA=0;PQR=0;PRO=0;QA=53335;QR=0;RO=0;RPL=594;RPP=112.315;RPPR=0;RPR=865;RUN=1;SAF=1007;SAP=461.453;SAR=452;SRF=0;SRP=0;SRR=0;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:331:0,331:0:0:331:11988:-1073.7,0	1:1128:0,1128:0:0:1128:41347:-3719.34,0
chrM	1438	.	A	G	75374.8	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=2237;CIGAR=1X;DP=2238;DPB=2238;DPRA=0;EPP=47.8812;EPPR=5.18177;GTI=0;LEN=1;MEANALT=1;MQM=59.8605;MQMR=60;NS=2;NUMALT=1;ODDS=5003.07;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=84494;QR=37;RO=1;RPL=1397;RPP=304.171;RPPR=5.18177;RPR=840;RUN=1;SAF=925;SAP=148.392;SAR=1312;SRF=0;SRP=5.18177;SRR=1;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:488:0,488:0:0:488:18416:-1656.39,0	1:1750:1,1749:1:37:1749:66078:-5935.03,0
chrM	2706	.	A	G	39098.5	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=1222;CIGAR=1X;DP=1235;DPB=1235;DPRA=0;EPP=3.188;EPPR=3.17734;GTI=0;LEN=1;MEANALT=1;MQM=59.9926;MQMR=60;NS=2;NUMALT=1;ODDS=2496.46;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=45333;QR=474;RO=13;RPL=507;RPP=79.8897;RPPR=3.17734;RPR=715;RUN=1;SAF=225;SAP=1062.06;SAR=997;SRF=5;SRP=4.51363;SRR=8;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:246:1,245:1:32:245:9136:-818.843,0	1:989:12,977:12:442:977:36197:-3215.79,0
chrM	3197	.	T	C	130583	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=3879;CIGAR=1X;DP=3892;DPB=3892;DPRA=0;EPP=59.2643;EPPR=3.17734;GTI=0;LEN=1;MEANALT=1;MQM=59.9938;MQMR=60;NS=2;NUMALT=1;ODDS=12192.8;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=146498;QR=463;RO=13;RPL=2190;RPP=143.521;RPPR=7.18621;RPR=1689;RUN=1;SAF=1459;SAP=519.999;SAR=2420;SRF=7;SRP=3.17734;SRR=6;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:1358:4,1354:4:150:1354:50893:-4563.63,0	1:2534:9,2525:9:313:2525:95605:-8569.92,0
chrM	3243	.	A	G	10397.4	.	AB=0;ABP=0;AC=1;AF=0.5;AN=2;AO=1365;CIGAR=1X;DP=3092;DPB=3092;DPRA=0;EPP=116.418;EPPR=46.9792;GTI=0;LEN=1;MEANALT=1;MQM=59.956;MQMR=59.8917;NS=2;NUMALT=1;ODDS=2394.09;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=49004;QR=63273;RO=1727;RPL=849;RPP=179.415;RPPR=105.14;RPR=516;RUN=1;SAF=443;SAP=368.01;SAR=922;SRF=637;SRP=261.033;SRR=1090;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:1035:341,694:341:12439:694:24871:-1118.32,0	0:2057:1386,671:1386:50834:671:24133:0,-2399.65
chrM	4769	.	A	G	48890.4	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=1545;CIGAR=1X;DP=1545;DPB=1545;DPRA=0;EPP=162.63;EPPR=0;GTI=0;LEN=1;MEANALT=1;MQM=51.4796;MQMR=0;NS=2;NUMALT=1;ODDS=4468.76;PAIRED=1;PAIREDR=0;PAO=0;PQA=0;PQR=0;PRO=0;QA=57939;QR=0;RO=0;RPL=463;RPP=541.537;RPPR=0;RPR=1082;RUN=1;SAF=890;SAP=80.6281;SAR=655;SRF=0;SRP=0;SRR=0;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:517:0,517:0:0:517:19209:-1665.33,0	1:1028:0,1028:0:0:1028:38730:-3308.01,0
chrM	5539	.	A	G	3281.62	.	AB=0;ABP=0;AC=1;AF=0.5;AN=2;AO=379;CIGAR=1X;DP=787;DPB=787;DPRA=0;EPP=216.428;EPPR=224.5;GTI=0;LEN=1;MEANALT=1;MQM=54.1504;MQMR=53.777;NS=2;NUMALT=1;ODDS=755.62;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=13803;QR=15250;RO=408;RPL=74;RPP=308.741;RPPR=351.808;RPR=305;RUN=1;SAF=286;SAP=216.428;SAR=93;SRF=318;SRP=279.681;SRR=90;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	0:299:221,78:221:8252:78:2824:0,-485.016	1:488:187,301:187:6998:301:10979:-358.012,0
chrM	7028	.	C	T	70453.4	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=2109;CIGAR=1X;DP=2114;DPB=2114;DPRA=0;EPP=63.8084;EPPR=3.44459;GTI=0;LEN=1;MEANALT=1;MQM=55.9113;MQMR=59.2;NS=2;NUMALT=1;ODDS=6416.42;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=78988;QR=192;RO=5;RPL=1123;RPP=22.3353;RPPR=3.44459;RPR=986;RUN=1;SAF=969;SAP=33.1175;SAR=1140;SRF=3;SRP=3.44459;SRR=2;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:708:1,707:1:39:707:26465:-2367.64,0	1:1406:4,1402:4:153:1402:52523:-4692.5,0
chrM	7269	.	G	A	59443.2	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=1773;CIGAR=1X;DP=1773;DPB=1773;DPRA=0;EPP=129.209;EPPR=0;GTI=0;LEN=1;MEANALT=1;MQM=58.3328;MQMR=0;NS=2;NUMALT=1;ODDS=5563.82;PAIRED=1;PAIREDR=0;PAO=0;PQA=0;PQR=0;PRO=0;QA=66393;QR=0;RO=0;RPL=929;RPP=11.8591;RPPR=0;RPR=844;RUN=1;SAF=848;SAP=10.2718;SAR=925;SRF=0;SRP=0;SRR=0;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:619:0,619:0:0:619:23040:-2069.26,0	1:1154:0,1154:0:0:1154:43353:-3894.52,0
chrM	8860	.	A	G	48674.5	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=1552;CIGAR=1X;DP=1556;DPB=1556;DPRA=0;EPP=36.1924;EPPR=5.18177;GTI=0;LEN=1;MEANALT=1;MQM=46.5528;MQMR=58.25;NS=2;NUMALT=1;ODDS=4986.6;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=56983;QR=146;RO=4;RPL=845;RPP=29.6556;RPPR=5.18177;RPR=707;RUN=1;SAF=844;SAP=28.8889;SAR=708;SRF=2;SRP=3.0103;SRR=2;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:598:0,598:0:0:598:21921:-1876.69,0	1:958:4,954:4:146:954:35062:-3002.8,0
chrM	9477	.	G	A	31596.7	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=944;CIGAR=1X;DP=946;DPB=946;DPRA=0;EPP=58.9901;EPPR=3.0103;GTI=0;LEN=1;MEANALT=1;MQM=59.3178;MQMR=60;NS=2;NUMALT=1;ODDS=2909.29;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=35363;QR=67;RO=2;RPL=521;RPP=25.1023;RPPR=7.35324;RPR=423;RUN=1;SAF=469;SAP=3.09311;SAR=475;SRF=1;SRP=3.0103;SRR=1;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:329:2,327:2:67:327:12108:-1080.66,0	1:617:0,617:0:0:617:23255:-2090.4,0
chrM	9548	.	G	A	23689.1	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=729;CIGAR=1X;DP=730;DPB=730;DPRA=0;EPP=65.6375;EPPR=5.18177;GTI=0;LEN=1;MEANALT=1;MQM=59.7133;MQMR=60;NS=2;NUMALT=1;ODDS=2252;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=26539;QR=38;RO=1;RPL=394;RPP=13.3792;RPPR=5.18177;RPR=335;RUN=1;SAF=339;SAP=10.7579;SAR=390;SRF=1;SRP=5.18177;SRR=0;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:261:1,260:1:38:260:9395:-838.704,0	1:469:0,469:0:0:469:17144:-1542,0
chrM	11467	.	A	G	157655	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=4755;CIGAR=1X;DP=4759;DPB=4759;DPRA=0;EPP=620.69;EPPR=3.0103;GTI=0;LEN=1;MEANALT=1;MQM=59.9394;MQMR=42.5;NS=2;NUMALT=1;ODDS=15848.9;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=179486;QR=151;RO=4;RPL=3558;RPP=2548.64;RPPR=3.0103;RPR=1197;RUN=1;SAF=1995;SAP=270.266;SAR=2760;SRF=2;SRP=3.0103;SRR=2;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:1820:0,1820:0:0:1820:68685:-6177.29,0	1:2939:4,2935:4:151:2935:110801:-9948.16,0
chrM	11719	.	G	A	86257.7	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=2676;CIGAR=1X;DP=2687;DPB=2687;DPRA=0;EPP=15.4873;EPPR=12.6832;GTI=0;LEN=1;MEANALT=1;MQM=59.6039;MQMR=60;NS=2;NUMALT=1;ODDS=6603.9;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=96531;QR=397;RO=11;RPL=1384;RPP=9.87851;RPPR=4.78696;RPR=1292;RUN=1;SAF=1292;SAP=9.87851;SAR=1384;SRF=2;SRP=12.6832;SRR=9;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:714:3,711:3:109:711:25643:-2282.19,0	1:1973:8,1965:8:288:1965:70888:-6347.71,0
chrM	12308	.	A	G	63434.1	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=1900;CIGAR=1X;DP=1905;DPB=1905;DPRA=0;EPP=73.302;EPPR=3.44459;GTI=0;LEN=1;MEANALT=1;MQM=59.9621;MQMR=60;NS=2;NUMALT=1;ODDS=5056.51;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=71094;QR=184;RO=5;RPL=1169;RPP=222.265;RPPR=3.44459;RPR=731;RUN=1;SAF=907;SAP=11.463;SAR=993;SRF=3;SRP=3.44459;SRR=2;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:547:5,542:5:184:542:20196:-1799.85,0	1:1358:0,1358:0:0:1358:50898:-4577.25,0
chrM	12372	.	G	A	58077.1	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=1737;CIGAR=1X;DP=1741;DPB=1741;DPRA=0;EPP=74.4189;EPPR=5.18177;GTI=0;LEN=1;MEANALT=1;MQM=60;MQMR=60;NS=2;NUMALT=1;ODDS=4910.27;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=64920;QR=149;RO=4;RPL=775;RPP=46.726;RPPR=3.0103;RPR=962;RUN=1;SAF=941;SAP=29.2942;SAR=796;SRF=3;SRP=5.18177;SRR=1;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:539:3,536:3:112:536:19922:-1781.72,0	1:1202:1,1201:1:37:1201:44998:-4043.68,0
chrM	13617	.	T	C	27663.7	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=846;CIGAR=1X;DP=848;DPB=848;DPRA=0;EPP=303.228;EPPR=3.0103;GTI=0;LEN=1;MEANALT=1;MQM=59.8972;MQMR=60;NS=2;NUMALT=1;ODDS=2549.91;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=31932;QR=78;RO=2;RPL=638;RPP=477.603;RPPR=7.35324;RPR=208;RUN=1;SAF=314;SAP=124.993;SAR=532;SRF=1;SRP=3.0103;SRR=1;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:284:0,284:0:0:284:10656:-956.335,0	1:564:2,562:2:78:562:21276:-1906.39,0
chrM	14766	.	C	T	54363.3	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=1633;CIGAR=1X;DP=1633;DPB=1633;DPRA=0;EPP=21.2132;EPPR=0;GTI=0;LEN=1;MEANALT=1;MQM=59.9853;MQMR=0;NS=2;NUMALT=1;ODDS=4408.47;PAIRED=1;PAIREDR=0;PAO=0;PQA=0;PQR=0;PRO=0;QA=61054;QR=0;RO=0;RPL=852;RPP=9.71354;RPPR=0;RPR=781;RUN=1;SAF=1113;SAP=470.614;SAR=520;SRF=0;SRP=0;SRR=0;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:473:0,473:0:0:473:17508:-1575.01,0	1:1160:0,1160:0:0:1160:43546:-3916.74,0
chrM	14793	.	A	G	52354.7	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=1605;CIGAR=1X;DP=1605;DPB=1605;DPRA=0;EPP=123.965;EPPR=0;GTI=0;LEN=1;MEANALT=1;MQM=59.9688;MQMR=0;NS=2;NUMALT=1;ODDS=4357.82;PAIRED=1;PAIREDR=0;PAO=0;PQA=0;PQR=0;PRO=0;QA=58638;QR=0;RO=0;RPL=950;RPP=120.75;RPPR=0;RPR=655;RUN=1;SAF=940;SAP=105.327;SAR=665;SRF=0;SRP=0;SRR=0;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:475:0,475:0:0:475:17313:-1557.59,0	1:1130:0,1130:0:0:1130:41325:-3717.26,0
chrM	15301	.	G	A	67202.5	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=2029;CIGAR=1X;DP=2036;DPB=2036;DPRA=0;EPP=73.6971;EPPR=4.45795;GTI=0;LEN=1;MEANALT=1.5;MQM=60;MQMR=60;NS=2;NUMALT=1;ODDS=5288.64;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=75132;QR=224;RO=6;RPL=962;RPP=14.8095;RPPR=3.0103;RPR=1067;RUN=1;SAF=1129;SAP=59.1336;SAR=900;SRF=1;SRP=8.80089;SRR=5;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:563:2,561:2:75:561:20714:-1856.3,0	1:1473:4,1468:4:149:1468:54418:-4880.92,0
chrM	15326	.	A	G	73749.6	.	AB=0;ABP=0;AC=2;AF=1;AN=2;AO=2200;CIGAR=1X;DP=2201;DPB=2201;DPRA=0;EPP=68.7112;EPPR=5.18177;GTI=0;LEN=1;MEANALT=1;MQM=60;MQMR=60;NS=2;NUMALT=1;ODDS=5718.37;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=82190;QR=38;RO=1;RPL=1013;RPP=32.8937;RPPR=5.18177;RPR=1187;RUN=1;SAF=1088;SAP=3.57883;SAR=1112;SRF=0;SRP=5.18177;SRR=1;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:597:1,596:1:38:596:22276:-2000.05,0	1:1604:0,1604:0:0:1604:59914:-5388.94,0
```

## Filtering variants

After filtering the data with stringent input parameters (restricting base quality to a minimum of 30 and mapping quality to a minimum of 20) a considerable amount of variants due to read-alignment bias exists. [Erik Garrison](https://github.com/ekg) has a beautiful illustration of various biases potentially affecting called variants and making a locus sequence-able.

![Various biases potentially affecting called variants](../../images/mt_biases.png "Here you can see that in an ideal case (indicated with a green star) a variant is evenly represent by different areas of sequencing reads (cycle and placement biases) and is balanced across the two strands (strand bias). Allele imbalance is not applicable in our case as it reflects significant deviation from the diploid (50/50) expectation.")

A robust tool set for processing VCF data is provided by [vcflib](https://github.com/vcflib/vcflib) developed by Erik Garrison, the author of FreeBayes. One way to filter VCF is using `INFO` fields of the VCF dataset. If you look at the VCF dataset shown above you will see all comment lines beginning with `##INFO`.  These are `INFO` fields. Each VCF record contains a list of `INFO` tags describing a wide range of properties for each VCF record. You will see that FreeBayes and NVC differ significantly in the number and types of `INFO` fields each of these caller generates. This why the two require different filtering strategies.

Among numerous types of data generated by FreeBayes let's consider the following variant properties:

* `##INFO=<ID=DP,Number=1,Type=Integer,Description="Total read depth at the locus">` - This is the number of reads covering a given site.
* `##INFO=<ID=SRP,Number=1,Type=Float,Description="Strand balance probability for the reference allele: Phred-scaled upper-bounds estimate of the probability of observing the deviation between SRF and SRR given E(SRF/SRR) ~ 0.5, derived using Hoeffding's inequality">` - The higher this quantity the better the site as it diminishes the chance of the site having significant strand bias.
* `##INFO=<ID=SAP,Number=A,Type=Float,Description="Strand balance probability for the alternate allele: Phred-scaled upper-bounds estimate of the probability of observing the deviation between SAF and SAR given E(SAF/SAR) ~ 0.5, derived using Hoeffding's inequality">` - The higher this quantity the better the site as it diminishes the chance of the site having significant strand bias  (also see this [discussion on the freebayes forum](https://groups.google.com/forum/#!topic/freebayes/fX4TOAqXJrA)).
* `##INFO=<ID=EPP,Number=A,Type=Float,Description="End Placement Probability: Phred-scaled upper-bounds estimate of the probability of observing the deviation between EL and ER given E(EL/ER) ~ 0.5, derived using Hoeffding's inequality">` - The higher this number the lower the chance of having significant placement bias.
* `QUAL` - phred scaled variant quality.

> <hands-on-title>Filtering VCF data</hands-on-title>
>
> 1. Use {% tool [VCFfilter](toolshed.g2.bx.psu.edu/repos/devteam/vcffilter/vcffilter2/1.0.0_rc1+galaxy3) %} to filter the variants with the following parameters:
>    - {% icon param-file %} *"VCF dataset to filter"*: select the VCF dataset produced by **FreeBayes** {% icon tool %}
>    - In *"more filters"*:
>      - In *"1: more filters"*:
>        - *"Select the filter type"*: `Info filter (-f)`
>        - *"Specify filtering value"*: `SRP > 20`
>      - Insert more filters:
>      - In *"2: more filters"*:
>        - *"Select the filter type"*: `Info filter (-f)`
>        - *"Specify filtering value"*: `SAP > 20`
>      - Insert more filters:
>      - In *"3: more filters"*:
>        - *"Select the filter type"*: `Info filter (-f)`
>        - *"Specify filtering value"*: `EPP > 20`
>      - Insert more filters:
>      - In *"4: more filters"*:
>        - *"Select the filter type"*: `Info filter (-f)`
>        - *"Specify filtering value"*: `QUAL > 20`
>      - Insert more filters:
>      - In *"5: more filters"*:
>        - *"Select the filter type"*: `Info filter (-f)`
>        - *"Specify filtering value"*: `DP > 20`
>
>    > <comment-title>VCF filter options</comment-title>
>    > Filtering FreeBayes VCF for strand bias (`SPR` and `SAP`), placement bias (`EPP`), variant quality (`QUAL`), and depth of coverage (`DP`).
>    {: .comment}
{: .hands_on}

> <question-title></question-title>
>
> 1. How many variants remain after filtering the VCF?
>
> > <solution-title></solution-title>
> > 1. Two variants remain after filtering the VCF (your results might be different depending on the input and references you use):
> > ```
> > #CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO	FORMAT	raw_child-ds-	raw_mother-ds-
> > chrM	3243	.	A	G	10397.4	.	AB=0;ABP=0;AC=1;AF=0.5;AN=2;AO=1365;CIGAR=1X;DP=3092;DPB=3092;DPRA=0;EPP=116.418;EPPR=46.9792;GTI=0;LEN=1;MEANALT=1;MQM=59.956;MQMR=59.8917;NS=2;NUMALT=1;ODDS=2394.09;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=49004;QR=63273;RO=1727;RPL=849;RPP=179.415;RPPR=105.14;RPR=516;RUN=1;SAF=443;SAP=368.01;SAR=922;SRF=637;SRP=261.033;SRR=1090;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	1:1035:341,694:341:12439:694:24871:-1118.32,0	0:2057:1386,671:1386:50834:671:24133:0,-2399.65
> > chrM	5539	.	A	G	3281.62	.	AB=0;ABP=0;AC=1;AF=0.5;AN=2;AO=379;CIGAR=1X;DP=787;DPB=787;DPRA=0;EPP=216.428;EPPR=224.5;GTI=0;LEN=1;MEANALT=1;MQM=54.1504;MQMR=53.777;NS=2;NUMALT=1;ODDS=755.62;PAIRED=1;PAIREDR=1;PAO=0;PQA=0;PQR=0;PRO=0;QA=13803;QR=15250;RO=408;RPL=74;RPP=308.741;RPPR=351.808;RPR=305;RUN=1;SAF=286;SAP=216.428;SAR=93;SRF=318;SRP=279.681;SRR=90;TYPE=snp;technology.ILLUMINA=1	GT:DP:AD:RO:QR:AO:QA:GL	0:299:221,78:221:8252:78:2824:0,-485.016	1:488:187,301:187:6998:301:10979:-358.012,0
> > ```
> {: .solution}
{: .question}

# Examining the results

For visualizing VCFs Galaxy has two external tool options. The first is called [VCF.IOBIO](http://vcf.iobio.io/) developed by [Gabor Marth's group](http://marthlab.org/) at the University of Utah. The second is called [IGV](http://software.broadinstitute.org/software/igv/) developed by Broad Institute.

## Visualising with VCF.IOBIO

VCF.IOBIO can be invoked by expanding a VCF dataset in Galaxy's history by clicking on it:

> <hands-on-title>Display data in VCF.IOBIO</hands-on-title>
>
>![Expanded output](../../images/mt_vcf_dataset_expanded.png "At the bottom there is a link 'display at vcf.iobio'. Clicking on this link will start indexing of VCF datasets, which is required to display them.")
>
>![VCF.IOBIO overview](../../images/mt_vcfiobio.png "There are not that many variants to look at in this example. Nevertheless there are helpful statistics such as Transition/Transversion (Ts/Tn) ratio.")
{: .hands_on}

## Visualising with IGV

Similarly to VCF.IOBIO expanding a history item representing a VCF dataset will reveal an IGV link:

> <hands-on-title>Display data in IGV</hands-on-title>
>
>![VCFfilter output expanded](../../images/mt_vcf_dataset_expanded.png "At the bottom there is a link 'display at IGV: local Human hg38'.")
>
>The difference between 'local' and 'Human hg38' links is explained in the following video:
>
> <div class="embed-responsive embed-responsive-16by9"><iframe src="https://player.vimeo.com/video/123414437?portrait=0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe></div>
>
> Visualizing our FreeBayes dataset will produce this:
>
>![IGV interface](../../images/mt_igv.png "Here we focus on one particular variant at position 3243 for reasons that will become apparent in the next section.")
{: .hands_on}

## Comparing frequencies

Visualizing VCF datasets is a good way to get an overall idea of the data, but it does not tell a lot of details. For example, above we have visualized site 3243 using IGV. It is interesting but we need to find out more. One thing we can do is to convert VCF dataset into a tab-delimited representation and with it play a bit more.

> <hands-on-title>Convert VCF to tab-delimited data</hands-on-title>
>
> 1. Use {% tool [VCFtoTab-delimited](toolshed.g2.bx.psu.edu/repos/devteam/vcf2tsv/vcf2tsv/1.0.0_rc1+galaxy0) %} to filter the variants with the following parameters:
>    - {% icon param-file %} *"Select VCF dataset to convert"*: select the VCF dataset produced by **VCFfilter** {% icon tool %}
>    - *"Report data per sample"*: `Yes`
>    - *"Fill empty fields with"*: `Nothing`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Why are there four rows in the VCFtoTab-delimited {% icon tool %} output when the input VCF had only two rows?
> 1. How many columns are in the VCFtoTab-delimited {% icon tool %} output?
>
> > <solution-title></solution-title>
> >
> > 1. Since *"Report data per sample"* was set to `Yes`, VCFtoTab-delimited {% icon tool %} produced two rows for each of the two variants, one row per sample (child and mother).
> > 1. There are 62 (!) columns in the output of VCFtoTab-delimited {% icon tool %}.
> {: .solution}
{: .question}

![VCFtoTab-delimited output](../../images/mt_tab.png "The columns in the above dataset represent INFO and Genotype fields on the original VCF dataset.")

 Let's restrict ourselves to a few key columns:

* 2 `POS` - position along mitochondrial genome
* 4 `REF` - reference allele
* 5 `ALT` - alternative allele
* 52 `SAMPLE` - name of the sample
* 54 `AO` - number of alternative observations (how many times do we see the alternative allele at this position in this sample)
* 55 `DP` - depth of coverage at this site for this sample

> <hands-on-title>Cut columns from a file</hands-on-title>
>
> 1. Use {% tool [Cut columns from a table](Cut1) %} to select specific columns with the following parameters:
>    - *"Cut columns"*: `c2,c4,c5,c52,c54,c55`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: select the tabular dataset produced by **VCFtoTab-delimited** {% icon tool %}
{: .hands_on}

Running Cut will generate the following dataset:

```
POS  REF ALT     SAMPLE       AO    DP
--------------------------------------
3243	A	G	raw_child-ds-	694	1035
3243	A	G	raw_mother-ds-	671	2057
5539	A	G	raw_child-ds-	78	299
5539	A	G	raw_mother-ds-	301	488
```

Let's look at position 3243. At this site, the mother sample has 671 `G`s (since `G` is the alternative allele) and 2057-671=1386 `A`s. The child sample has 694 `G`s and 1035-694=341 `A`s:

```
Allele     A     G
-------------------
Mother  1386   691
Child   341    694
```

> <question-title></question-title>
>
> 1. What do you notice about the relative frequencies of the `A` and `G` alleles between mother and child?
>
> > <solution-title></solution-title>
> >
> > 1. The *major* allele in the mother (`A`) becomes the *minor* allele in the child -- a remarkable frequency change due to mitochondrial bottleneck!
> {: .solution}
{: .question}

# Conclusion

This entire analysis is available as a [Galaxy history](https://usegalaxy.org/u/aun1/h/non-diploid-freebayes) that you can import into your Galaxy account and play with.

Now you know how to call variants in non-diploid system, so try it on bacteria, viruses, etc.


topic: variant-analysis
tutorial: sars-cov-2
title: From NCBI's Sequence Read Archive (SRA) to Galaxy: SARS-CoV-2 variant analysis
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["covid19", "virology", "one-health"]
content_hash: 4169377939231b03ee4dc2e0481b1f5c
last_modified: 2026-02-26T21:12:18.626972
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/sars-cov-2/tutorial.html

content:
The aim of this tutorial is twofold:

1. It introduces general patterns of working with high-throughput sequencing data in Galaxy,
2. It uses SARS-CoV-2 sequencing data analysis as a concrete example of how to identify genomic mutations from high-throughput sequencing data for a batch of viral isolates in Galaxy.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare analysis history and data

## Create a new history for this analysis

Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Prepare the Galaxy history</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md name="SARS-CoV-2 sequence data analysis" %}
>
{: .hands_on}

## Find, obtain and filter information about public sequencing data

First we need to find some good, public datasets to play with.

The [Sequence Read Archive (SRA)](https://www.ncbi.nlm.nih.gov/sra) is the primary archive of *unassembled reads*  operated by the [US National Institutes of Health (NIH)](https://www.ncbi.nlm.nih.gov/).
SRA is a great place to get the sequencing data that underlie publications and studies.
The study we are interested in for this tutorial is one on sequencing of SARS-CoV-2 viral isolates that were obtained in the Boston area in the first months of the COVID-19 pandemic ({% cite Lemieux2021 %}).

So lets see how we can find its raw data in the SRA, and how we can get some of that data into Galaxy to analyze it. The Acknowledgments section of the publication mentions that raw reads from the study have been deposited in the SRA under **BioProject PRJNA622837** so that's going to be our starting point.

> <hands-on-title>Find the example data in the SRA</hands-on-title>
>
> 1. Go to NCBI's SRA page by pointing your browser to [https://www.ncbi.nlm.nih.gov/sra](https://www.ncbi.nlm.nih.gov/sra)
> 2. In the search box enter `PRJNA622837[BioProject]`
>
>    Oh, this finds a *lot* of samples (more than 22,000 at the time of writing)!
>    This is because the BioProject ID we used is that of "SARS-CoV-2 Patient Sequencing from the Broad Institute", which has seen many more sample submissions since the published study.
>    Let's refine our search a bit.
> 3. Replace your previous search with `PRJNA622837[BioProject] AND RNA-Seq[Strategy]`
>
>    This returns only those samples from the BioProject, for which the sequencing strategy was RNA-Seq.
>
>    At the time of writing, 1908 hits are retained with this query, which is still more than the approximately 800 genomes mentioned in the publication, but we got a lot closer.
>
>    > <comment-title></comment-title>
>    >
>    > The vast majority of what has been excluded by adding `RNA-Seq[Strategy]` to the query are datasets for which the "Strategy" is *AMPLICON*.
>    > For these datasets, the viral RNA (or more precisely, the cDNA obtained from it) got amplified with a tiled-amplicon protocol.
>    > While tiled-amplicon sequencing has emerged as a standard way of sequencing SARS-CoV-2 samples over the course of the pandemic, the corresponding protocols/primer schemes were simply not available during the early phase of the pandemic.
>    >
>    {: .comment}
> 4. Instead of inspecting all 1908 hits one-by-one through the web interface, lets download the metadata for all samples in a single file
>    1. Click on the  **Send to:** dropdown at the top of the search results list
>    2. Under **Choose Destination**, select `Run Selector`
>    3. Click **Go**
>
>    ![SRA search results with the "Send to:" dropdown configured for sending results to the Run Selector](../../images/get_runinfo.png)
>
>    You will be taken to the SRA Run Selector page, where you can browse all of the metadata of all search hits rather conveniently.
>
> 5. In the **Select** table, click `Metadata` in the **Download** column
>
>    ![The SRA Run Selector page with the mouse hovering over the Metadata button](../../images/metadata_from_run_selector.png)
>
>    This will download the metadata for all retrieved sequencing runs as a comma-separated text file.
>
{: .hands_on}

Now that we have downloaded this file we can go to a Galaxy instance and start processing it.

> <comment-title></comment-title>
>
> Note that the file we just downloaded is **not** sequencing data itself. Rather, it is *metadata* describing the data itself. We will use a few Galaxy tools to filter this list down to just a few accessions that will be used in the remainder of this tutorial.
>
{: .comment}

> <hands-on-title>Upload the metadata file into Galaxy</hands-on-title>
>
> 1. In Galaxy, click {% icon galaxy-upload %} **Upload Data**
>
>    ![The Upload Data button in Galaxy](../../images/upload_button.png)
> 2. In the Upload dialog box, click **Choose local files**
>
>    ![The Choose local files button](../../images/choose_local_files_button.png)
> 3. Find and select the downloaded metadata file on your computer
> 4. Click **Start**
> 5. Close the dialog by pressing **Close**
> 6. Once the dataset has finished uploading to your history, you can {% icon galaxy-eye %} inspect its contents.
>
>    You will see that this file contains the same metadata that you could preview on the SRA Run Selector page.
>
{: .hands_on}

In Galaxy, it is rather easy to perform further filtering of the metadata records. For example, the publication mentions that the authors analyzed samples collected between 4 March and 9 May 2020.

If you inspect the metadata, you should see a column **Collection_Date**. That looks useful!

> <hands-on-title>Filter metadata lines by collection date range</hands-on-title>
>
> 1. {% tool [Filter data on any column using simple expressions](Filter1) %}
>
>    - {% icon param-file %} *"Filter"*: the uploaded metadata
>    - *"With following condition"*: `"2020-03-04" <= c11 <= "2020-05-09"`
>
>      Column 11 (c11) is the Collection_Date column in this case.
>      If your input has this column in a different position, you would need to adjust the filter condition above accordingly.
>    - *"Number of header lines to skip"*: `1`
>
{: .hands_on}

So now we are down to sequencing runs from the same date range as studied in the publication - though we still have more runs than samples discussed in the publication. Note, however, that the publication talks about assembled genomes, and it is possible that some of the sequencing runs listed in our metadata do not contain good enough data to assemble any SARS-CoV-2 genome information, or it could be that some viral samples got sequenced in several sequencing runs to obtain enough data.

We could explore this situation in more detail, and Galaxy could also process all the sequencing runs we are currently left with, but for the sake of simplicity of the tutorial, we want to select a *much* smaller subset of sequencing runs now anyway. In particular, from previous experience with this data we suggest to continue with just the following two interesting datasets: **SRR11954102** and **SRR12733957**. So let's pull their identifiers out of the metadata file.

> <comment-title></comment-title>
> The following steps may seem a bit silly given that we just revealed the two identifiers above.
> However, think of them as placeholders for extracting run identifiers based on some other criteria.
>
{: .comment}

> <hands-on-title>Extracting a subset of identifiers</hands-on-title>
>
> 1. {% tool [Select lines that match an expression](Grep1) %}
>
>    - {% icon param-file %} *"Select lines from"*: the metadata filter by collection date; output of **Filter** tool
>    - *"that"*: `Matching`
>    - *"the pattern"*: `SRR12733957|SRR11954102`
>
> 2. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0) %}
>
>    - {% icon param-file %} *"File to cut"*: the metadata with only two runs retained; output of **Select** tool
>    - Under *"Cut by"*
>      - *"List of Fields"*: `Column: 1`
>
>    {% snippet faqs/galaxy/analysis_cut.md %}
>
{: .hands_on}

At this point, you should have a dataset with just two lines and a single column with this content:

```
SRR12733957
SRR11954102
```

## Retrieve selected sequencing data from the SRA

Now that we have our identifiers of interest extracted, we are ready to download the actual sequencing data. Since this is a very common need Galaxy offers a dedicated tool for the purpose of downloading sequencing data from the SRA identified via its run accession.

> <hands-on-title>Retrieve sequencing data from SRA via run accessions</hands-on-title>
>
> 1. {% tool [Faster Download and Extract Reads in FASTQ](toolshed.g2.bx.psu.edu/repos/iuc/sra_tools/fasterq_dump/3.0.3+galaxy0) %} with the following parameters:
>    - *"select input type"*: `List of SRA accession, one per line`
>      - {% icon param-file %} *"sra accession list"*: the dataset with the two run accessions; output of **Cut**
>
{: .hands_on}

Several entries should have been created in your history panel when you submitted the previous job:
- **Pair-end data (fasterq-dump)**: Contains Paired-end datasets (if available)
- **Single-end data (fasterq-dump)** Contains Single-end datasets (if available)
- **Other data (fasterq-dump)** Contains Unpaired datasets (if available)
- **fasterq-dump log** Contains Information about the tool execution

The first three items are actually *collections* of datasets.

When the tool run is finished (all history items created by it are green / done), take a moment to explore the collections by first **clicking** on the collection name in the history panel.  This takes you inside the collection and shows you the datasets in it.  You can then navigate back to the outer level of your history.

You should find that only the **Pair-end data** collection contains any sequencing data, which makes sense as both samples have been paired-end sequenced on the Illumina platform.

## Get the reference genome data

Since our data analysis will involve mapping the sequenced reads just downloaded to the SARS-CoV-2 reference, we also need to obtain that reference now.

> <hands-on-title>Get the reference genome data</hands-on-title>
>
> 1. Import the following file into your history:
>
>    ```
>    https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/009/858/895/GCF_009858895.2_ASM985889v3/GCF_009858895.2_ASM985889v3_genomic.fna.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}


# Identification of mutations in the batch of SARS-Cov-2 sequencing data

As our actual analysis we will now perform all steps necessary to identify mutations present in the genomes of our minimal batch of two samples with respect to the SARS-CoV-2 Wuhan-Hu-1 reference sequence.

We will perform quality control and read trimming, map the reads from both samples to the reference, and perform variant calling. Finally we will annotate each sample's variants with their predicted functional impact and report our findings in a table.

We will also summarize different quality metrics obtained throughout the analysis in one nice summary report.

## Adapter trimming with **fastp**

Removing sequencing adapters improves alignments and variant calling. **fastp** {% icon tool %} can automatically detect widely used sequencing adapters.

> <hands-on-title>Task description</hands-on-title>
>
> 1. **fastp** {% icon tool %} with the following parameters:
>    - *"Single-end or paired reads"*: `Paired Collection`
>        - {% icon param-file %} *"Select paired collection(s)"*: `list_paired` (output of **Faster Download and Extract Reads in FASTQ** {% icon tool %})
>    - In *"Output Options"*:
>        - *"Output JSON report"*: `Yes`
{: .hands_on}


## Alignment with **Map with BWA-MEM**

**BWA-MEM** {% icon tool %} is a widely used sequence aligner for short-read sequencing datasets such as those we are analysing in this tutorial.

> <hands-on-title>Align sequencing reads to reference genome</hands-on-title>
>
> 1. **Map with BWA-MEM** {% icon tool %} with the following parameters:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>        - {% icon param-file %} *"Use the following dataset as the reference sequence"*: `output` (Input dataset)
>    - *"Single or Paired-end reads"*: `Paired Collection`
>        - {% icon param-file %} *"Select a paired collection"*: `output_paired_coll` (output of **fastp** {% icon tool %})
>    - *"Set read groups information?"*: `Do not set`
>    - *"Select analysis mode"*: `1.Simple Illumina mode`
>
{: .hands_on}

## Remove duplicates with **MarkDuplicates**

**MarkDuplicates** {% icon tool %} removes duplicate sequences originating from library preparation artifacts and sequencing artifacts. It is important to remove these artefactual sequences to avoid artificial overrepresentation of single molecule.

> <hands-on-title>Remove PCR duplicates</hands-on-title>
>
> 1. **MarkDuplicates** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Select SAM/BAM dataset or dataset collection"*: `bam_output` (output of **Map with BWA-MEM** {% icon tool %})
>    - *"If true do not write duplicates to the output file instead of writing them with appropriate flags set"*: `Yes`
>
{: .hands_on}

## Generate alignment statistics with **Samtools stats**

After the duplicate marking step above we can generate statistic about the alignment we have generated.

> <hands-on-title>Generate alignment statistics</hands-on-title>
>
> 1. **Samtools stats** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"BAM file"*: `outFile` (output of **MarkDuplicates** {% icon tool %})
>    - *"Set coverage distribution"*: `No`
>    - *"Output"*: `One single summary file`
>    - *"Filter by SAM flags"*: `Do not filter`
>    - *"Use a reference sequence"*: `No`
>    - *"Filter by regions"*: `No`
{: .hands_on}

## **Realign reads** with lofreq viterbi

**Realign reads** {% icon tool %} corrects misalignments around insertions and deletions. This is required in order to accurately detect variants.

> <hands-on-title>Realign reads around indels</hands-on-title>
>
> 1. **Realign reads** with lofreq {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Reads to realign"*: `outFile` (output of **MarkDuplicates** {% icon tool %})
>    - *"Choose the source for the reference genome"*: `History`
>        - {% icon param-file %} *"Reference"*: `output` (Input dataset)
>    - In *"Advanced options"*:
>        - *"How to handle base qualities of 2?"*: `Keep unchanged`
{: .hands_on}

## Add indel qualities with lofreq **Insert indel qualities**

This step adds indel qualities into our alignment file. This is necessary in order to call variants using **Call variants** with lofreq {% icon tool %}

> <hands-on-title>Add indel qualities</hands-on-title>
>
> 1. **Insert indel qualities** with lofreq {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Reads"*: `realigned` (output of **Realign reads** {% icon tool %})
>    - *"Indel calculation approach"*: `Dindel`
>        - *"Choose the source for the reference genome"*: `History`
>            - {% icon param-file %} *"Reference"*: `output` (Input dataset)
>
{: .hands_on}

## Call Variants using lofreq **Call variants**

We are now ready to call variants.

> <hands-on-title>Call variants</hands-on-title>
>
> 1. **Call variants** with lofreq {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Input reads in BAM format"*: `output` (output of **Insert indel qualities** {% icon tool %})
>    - *"Choose the source for the reference genome"*: `History`
>        - {% icon param-file %} *"Reference"*: `output` (Input dataset)
>    - *"Call variants across"*: `Whole reference`
>    - *"Types of variants to call"*: `SNVs and indels`
>    - *"Variant calling parameters"*: `Configure settings`
>        - In *"Coverage"*:
>            - *"Minimal coverage"*: `50`
>        - In *"Base-calling quality"*:
>            - *"Minimum baseQ"*: `30`
>            - *"Minimum baseQ for alternate bases"*: `30`
>        - In *"Mapping quality"*:
>            - *"Minimum mapping quality"*: `20`
>    - *"Variant filter parameters"*: `Preset filtering on QUAL score + coverage + strand bias (lofreq call default)`
{: .hands_on}

The output of this step is a collection of VCF files that can be visualized in a genome browser.

## Annotate variant effects with **SnpEff eff:**

We will now annotate the variants we called in the previous step with the effect they have on the SARS-CoV-2 genome.

> <hands-on-title>Annotate variant effects</hands-on-title>
>
> 1. **SnpEff eff:** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Sequence changes (SNPs, MNPs, InDels)"*: `variants` (output of **Call variants** {% icon tool %})
>    - *"Output format"*: `VCF (only if input is VCF)`
>    - *"Create CSV report, useful for downstream analysis (-csvStats)"*: `Yes`
>    - *"Annotation options"*: ``
>    - *"Filter output"*: ``
>    - *"Filter out specific Effects"*: `No`
>
{: .hands_on}

The output of this step is a VCF file with added variant effects.

## Create table of variants using **SnpSift Extract Fields**

We will now select various effects from the VCF and create a tabular file that is easier to understand for humans.

> <hands-on-title>Create table of variants</hands-on-title>
>
> 1. **SnpSift Extract Fields** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Variant input file in VCF format"*: `snpeff_output` (output of **SnpEff eff:** {% icon tool %})
>    - *"Fields to extract"*: `CHROM POS REF ALT QUAL DP AF SB DP4 EFF[*].IMPACT EFF[*].FUNCLASS EFF[*].EFFECT EFF[*].GENE EFF[*].CODON`
>    - *"multiple field separator"*: `,`
>    - *"empty field text"*: `.`
>
{: .hands_on}


## Summarize data with **MultiQC**

We will now summarize our analysis with MultiQC, which generates a beautiful report for our data.

> <hands-on-title>Summarize data</hands-on-title>
>
> 1. **MultiQC** {% icon tool %} with the following parameters:
>    - In *"Results"*:
>        - {% icon param-repeat %} *"Insert Results"*
>            - *"Which tool was used generate logs?"*: `fastp`
>                - {% icon param-file %} *"Output of fastp"*: `report_json` (output of **fastp** {% icon tool %})
>        - {% icon param-repeat %} *"Insert Results"*
>            - *"Which tool was used generate logs?"*: `Samtools`
>                - In *"Samtools output"*:
>                    - {% icon param-repeat %} *"Insert Samtools output"*
>                        - *"Type of Samtools output?"*: `stats`
>                            - {% icon param-file %} *"Samtools stats output"*: `output` (output of **Samtools stats** {% icon tool %})
>        - {% icon param-repeat %} *"Insert Results"*
>            - *"Which tool was used generate logs?"*: `Picard`
>                - In *"Picard output"*:
>                    - {% icon param-repeat %} *"Insert Picard output"*
>                        - *"Type of Picard output?"*: `Markdups`
>                        - {% icon param-file %} *"Picard output"*: `metrics_file` (output of **MarkDuplicates** {% icon tool %})
>        - {% icon param-repeat %} *"Insert Results"*
>            - *"Which tool was used generate logs?"*: `SnpEff`
>                - {% icon param-file %} *"Output of SnpEff"*: `csvFile` (output of **SnpEff eff:** {% icon tool %})
{: .hands_on}


# Conclusion

You have seen now how to import public sequencing data into Galaxy, and you have performed the basic steps of variant calling on a batch of RNA-Seq SARS-CoV-2 data.

Galaxy allows you to do a lot more in terms of viral sequence data analysis though. Specifically, the advanced tutorial [Mutation calling, viral genome reconstruction and lineage/clade assignment from SARS-CoV-2 sequencing data]({% link topics/variant-analysis/tutorials/sars-cov-2-variant-discovery/tutorial.md %}) is highly recommended if you want to learn how to perform a complete and more automated analysis of SARS-CoV-2 sequencing data that does not stop at lists of identified mutations but also produces consensus sequences and lineage assignments for all samples in a batch. This tutorial also shows how to deal with input data other than Illumina paired-end-sequenced RNA-Seq data.


topic: variant-analysis
tutorial: trio-analysis
title: Trio Analysis using Synthetic Datasets from RD-Connect GPAP
difficulty: advanced
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: ["cyoa"]
content_hash: 4036c7e9ed0c59789482e84f1b8fcd91
last_modified: 2026-02-26T21:12:18.650962
zenodo_link: https://doi.org/10.5281/zenodo.6483454
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/trio-analysis/tutorial.html

content:
To discover causal mutations of inherited diseases it’s common practice to do a trio analysis. In a trio analysis DNA is sequenced of both the patient and parents. Using this method, it’s possible to identify multiple inheritance patterns. Some examples of these patterns are autosomal recessive, autosomal dominant, and  de-novo variants, which are represented in the figure below. To elaborate, the most left tree shows an autosomal dominant inhertitance pattern where the offspring inherits a faulty copy of the gene from one of the parents. The center subfigure represents an autosomal recessive disease, here the offspring inherited a faulty copy of the same gene from both parents. In the right subfigure a de-novo mutation is shown, which is caused by a mutation during the offspring’s lifetime.

![Image of three family trees representing a different inheritance pattern each, from left to right the trees have a title on top with: autosomal dominant, autosomal recessive, and De-Novo. The families consists of a father, mother, and son. Under the trees there is a legend which shows, from left to right, a red diagonal line with the text 'Affected', a white square with a black border with the text 'Male Variant Absent', a white circle with a black border with the text 'Female Variant Absent', a half-blackened white square with a black border with the text 'Male Variant Present', and a half-blackened circle with a black border with the text 'Female Variant Present'.](../../images/trio-analysis/pedigree.svg "Three family trees representing autosomal dominant (left), autosomal recessive (center), and a de-novo inheritence pattern (right) from parents to son.")

To discover these mutations either whole exome sequencing (WES) or whole genome sequencing (WGS) can be used. With these technologies it is possible to uncover the DNA of the parents and offspring to find (shared) mutations in the DNA. These mutations can include insertions/deletions (indels), loss of heterozygosity (LOH), single nucleotide variants (SNVs), copy number variations (CNVs), and fusion genes.

In this tutorial we will also make use of the HTSGET protocol, which is a program to download our data securely and savely. This protocol has been implemented in the {% tool [EGA Download Client](toolshed.g2.bx.psu.edu/repos/iuc/ega_download_client/pyega3/5.0.2+galaxy0) %} tool, so we don't have to leave Galaxy to retrieve our data.

We will not start our analysis from scratch, since the main goal of this tutorial is to use the HTSGET protocol to download variant information from an online archive and to find the causative variant from those variants. If you want to learn how to do the analysis from scratch, using the raw reads, you can have a look at the [Exome sequencing data analysis for diagnosing a genetic disease]({% link topics/variant-analysis/tutorials/exome-seq/tutorial.md %}) tutorial.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Data preperation
In this tutorial we will use case 5 from the RD-Connect GPAP synthetic datasets. The dataset that we will use consists of WGS VCFs from a real healthy family trio, which originates from the Illumina Platinum initiative {% cite Eberle2017 %} and was made available by the [HapMap project](https://www.genome.gov/10001688/international-hapmap-project). In our dataset a real causative variant was manually spiked-in that should cause breast cancer. The spike-in has been synthetically introduced in the mother and daughter. Here our goal is to identify the genetic variation that is responsible for the disease.

We offer two ways to download the files. Firstly, you can download the files directly from the EGA-archive by requesting DAC access. This will take only 1 workday and gives you access to all of the RD-Connect GPAP synthetic datasets. However if you don't have the time you can also download the data from zenodo.

{% include _includes/cyoa-choices.html option1="EGA-Archive" option2="Zenodo" default="EGA-Archive"
       text="Here you can choose to either follow the data preperation for the data from the EGA-archive or Zenodo." %}

<div class="Zenodo" markdown="1">
I see, you can't wait to get DAC access. To download the data from zenodo for this tutorial you can follow the step below.

> <hands-on-title>Retrieve data from zenodo</hands-on-title>
>
> 1. Import the 3 VCFs from [Zenodo](https://zenodo.org/record/6483454) to Galaxy **as a collection**.
>    ```
>    https://zenodo.org/record/6483454/files/Case5_F.17.g.vcf.gz
>    https://zenodo.org/record/6483454/files/Case5_IC.17.g.vcf.gz
>    https://zenodo.org/record/6483454/files/Case5_M.17.g.vcf.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md collection=true %}
>
> 2. Set the datatype to **vcf**.
>
> 3. Click on **Start**
>
> 4. If you forgot to select the *"Collections"* tab during upload, please put the file in a collection now.
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
{: .hands_on}

</div>

<div class="EGA-Archive" markdown="1">

## Getting DAC access
Our test data is stored in EGA, which can be easily accessed using the EGA Download Client. Our specific EGA dataset accession ID is: "EGAD00001008392". However, before you can access this data you need to request DAC access to this dataset. This can be requested by emailing to <helpdesk@ega-archive.org>, don’t forget to mention the dataset ID! When the EGA grants you access it will create an account for you, if you don't have it already. Next, you should link your account to your Galaxy account by going to the homepage on Galaxy and at the top bar click **User > Preferences > Manage Information**. Now add your email and password of your (new) EGA account under: **Your EGA (european Genome Archive) account**. After that you can check if you can log in and see if you have access to the dataset.

> <hands-on-title>Check log-in and authorized datasets</hands-on-title>
>
> 1. {% tool [EGA Download Client](toolshed.g2.bx.psu.edu/repos/iuc/ega_download_client/pyega3/5.0.2+galaxy0) %} with the following parameters:
>    - *"What would you like to do?"*: `List my authorized datasets`
>
> > <comment-title>Check if the dataset is listed.</comment-title>
> >
> > Check if your dataset is listed in the output of the tool. If not you can look at the header of the output to find out why it is not listed. When the header does not provide any information you could have a look at the error message by clicking on the **eye** {% icon galaxy-eye %} of the output dataset and then click on the icon **view details** {% icon details %}. The error should be listed at **Tool Standard Error**.
> {: .comment}
>
{: .hands_on}

## Download list of files
When you have access to the EGA dataset, you can download all the needed files. However, the EGA dataset contains many different filetypes and cases, but we are only interested in the VCFs from case 5 and, to reduce execution time, the variants on chromosome 17. To be able to donwload these files we first need to request the list of files from which we can download. Make sure to use **version 4+** of the {% tool [EGA Download Client](toolshed.g2.bx.psu.edu/repos/iuc/ega_download_client/pyega3/5.0.2+galaxy0) %}.

> <hands-on-title>Request list of files in the dataset</hands-on-title>
>
> 1. {% tool [EGA Download Client](toolshed.g2.bx.psu.edu/repos/iuc/ega_download_client/pyega3/5.0.2+galaxy0) %} **version 4+** with the following parameters:
>    - *"What would you like to do?"*: `List files in a datasets`
>    - *"EGA Dataset Accession ID?"*: `EGAD00001008392`
>
> {% snippet faqs/galaxy/tools_change_version.md %}
>
> {: .comment}
{: .hands_on}


## Filter list of files
Now that we have listed all the files, we need to filter out the files we actually need. We can do this by using a simple regular expression or regex. With regex it will be easy to find or replace patterns within a textfile.

> <hands-on-title>Filter out VCFs from list of files</hands-on-title>
>
> 1. {% tool [Search in textfiles](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_grep_tool/1.1.1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `List of EGA datasets` (output of **EGA Download Client** {% icon tool %})
>    - *"Type of regex"*: `Extended (egrep)`
>    - *"Regular Expression"*: `Case5.+17.+vcf.gz$` The regex might seem a bit complicated if you never worked with it, however it is quite simple. We simply search for lines which contain our sequence of characters, which are lines that contain Case5 then any character for any length, denoted by the ".+", until (chromosome) 17 is found and then again any character for any length until the sequence ends with, denoted by the dollar sign, "vcf.gz".
>    - *"Match type"*: `case sensitive`
>
{: .hands_on}

## Download files
After the filtering you should have a tabular file with 3 lines each containing the ID of a VCF file from case 5, as shown below.
```
EGAF00005573839	1	33859685	53277d993a780e0c295079bf1346ee9f	Case5_IC.17.g.vcf.gz
EGAF00005573866	1	33350163	7e612852ee0824be458fbf9aeecaa61a	Case5_M.17.g.vcf.gz
EGAF00005573882	1	42856357	14b53924d1492e28ad6078ceb8cfdbc7	Case5_F.17.g.vcf.gz
```

> <hands-on-title>Download listed VCFs</hands-on-title>
>
> 1. {% tool [EGA Download Client](toolshed.g2.bx.psu.edu/repos/iuc/ega_download_client/pyega3/5.0.2+galaxy0) %} with the following parameters:
>    - *"What would you like to do?"*: `Download multiple files (based on a file with IDs)`
>        - {% icon param-file %} *"Table with IDs to download"*: `Filtered list of files` (output of **Search in textfiles** {% icon tool %})
>        - *"Column containing the file IDs"*: `Column: 1`
>
>
> After the download you should have a collection with one file for each family member, i.e. mother (M), father (F), and case (IC). Make sure the files are recognized as the **vcf_bgzip** format.
>
> {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

## Decompress VCFs
Finally, we need to decompress our bgzipped VCFs, since we will use a text manipulation tool as a next step to process the VCFs. To decompress the vcf we will use a built-in tool from Galaxy, which can be accessed by manipulating the file itself in a similair fashion as changing its detected type.

> <hands-on-title>Convert compressed vcf to uncompressed.</hands-on-title>
>
> Open the collection of VCFs and execute the following steps for each VCF.
>
> 1. Click on the **pencil** {% icon galaxy-pencil %} of the vcf you want to convert.
>
> 2. Click on the **convert tab** {% icon galaxy-gear %}.
>
> 3. Under **Target datatype** select `vcf (using 'Convert compressed file to uncompressed.')`
>
> 4. Click {% icon exchange %} **Create Dataset**.
>
> After transforming all the VCFs you need to combine the converted VCFs into a colllection again.
>
> {% snippet faqs/galaxy/collections_build_list.md datasets_description="the 3 vcf files" n="3" %}
>
{: .hands_on}

</div>

# Pre-Processing
Before starting the analysis, the VCF files have to be pre-processed in order to meet input requirements of the tools which we will use for the downstream analysis.

## Add chromosome prefix to vcf
Firstly, our next tool has some assumptions about our input VCFs. The tool expects the chromosome numbers to start with a prefix `chr`. Our VCFs only use the chromosome numbers however, the VCFs just use the chromosome numbers. This is due to a difference in reference genome used when creating the VCFs. To change the prefix in the VCFs we will use regex again.

> <hands-on-title>Add chr prefix using regex</hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.1) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `VCFs` (output of **Convert compressed file to uncompressed.** {% icon tool %})
>    - *"using column"*: `Column: 1`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `^([0-9MYX])`
>            - *"Replacement"*: `chr\1`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `^(##contig=<.*ID=)([0-9MYX].+)`
>            - *"Replacement"*: `\1chr\2`
>
> > <comment-title>Explaining the regex.</comment-title>
> >
> > The two regex patterns might look complicated but they are quite simple if you break them down in components.
> > - The first check `^([0-9MYX])` > `chr\1` adds the `chr` prefix to all the non-header line. The check can be broken down into the following elements:
> >      - `^` means that the pattern has to start at the beginning of the line not somewhere randomly in the line.
> >      - `[0-9MYX]` means that at this position there should be a character from the list `[]`, namely either a number from `0-9` or the character `M`, `Y`, or `X`.
> >      - The replacement pattern `chr\1` means that the prefix `chr` has to be inserted before the first match `\1`. Here the first match refers to the pattern in the first brackets `()` around the list `[0-9MXY]`.
> > - The second check `(##contig=<.*ID=)([0-9MYX].+)` > `\1chr\2` adds the `chr` prefix to the contig ids in the header lines. The check can be broken down into the following elements:
> >     - The first match, the pattern in the first brackets `##contig=<.*ID=`, matches the contig header lines, which start with `##contig=<`. It is followed by a match anything `.` for zero-or-more times `*` until it finds `ID=`.
> >     - The second match, the pattern in the second brackets `[0-9MYX].+`, matches the chromosome numbers and characters `[0-9MYX]` followed by a matching anything `.` for one-or-more times `+`.
> >     - The replacement pattern `\1chr\2` means that the prefix `chr` has to be inserted between the first match `\1` or `##contig=<.*ID=` and the second match `\2` or `[0-9MYX].+`.
> {: .comment}
>
{: .hands_on}

## Normalizing VCF
After adding the prefixes to the VCFs we need to normalize the variants in the VCF to standardize how the variants are represented within the VCF. This is a very important step, since the variants in the mother and daughter might be represented differently, which would mean that the causative variant might be overlooked!

One of the normalization steps is splitting multiallelic variants, 2 variants detected on the same position but on a different allele. Splitting these records will put the 2 variants on a separate line, that way the impact of the individual mutations can be evaluated. In addition, indels need to be left-aligned and normalized because that’s how they are stored in the annotation databases. An indel is left-aligned and normalized, according to {% cite Tan2015 %}, "if and only if it is no longer possible to shift its position to the left while keeping the length of all its alleles constant" and "if it is represented in as few nucleotides as possible".

![Image of different representations of the same variant. On the top the variant is shown in the reference sequence GGGCACACACAGGG and the alternate sequence GGGCACACAGGG. Underneath the image is divided into two panels, the left panel aligns each allele to the reference genome, and the right panel represents the variants in a VCF with the columns POS (position), REF (reference), and ALT (alternate). From top to bottom it shows that A is not left-aligned with the reference CAC and the alternate C  at position 6, B is neither left-aligned nor parsimonious with reference GCACA and alternate GCA at position 3, C is not parsimonious with reference GGCA and alternate GG at position 2, and D is normalized with reference GCA and the alternate position G at position 3.](../../images/trio-analysis/variant_normalization.png "The different ways to represent INDELs. (Source: https://academic.oup.com/bioinformatics/article/31/13/2202/196142)")

> <hands-on-title>Normalize VCF</hands-on-title>
>
> 1. {% tool [bcftools norm](toolshed.g2.bx.psu.edu/repos/iuc/bcftools_norm/bcftools_norm/1.9+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"VCF/BCF Data"*: `VCFs with chr prefix` (output of **Text reformatting** {% icon tool %})
>    - *"Choose the source for the reference genome"*: `Use a built-in genome`
>        - *"Reference genome"*: `Human (Homo sapiens): hg19`
>    - *"Left-align and normalize indels?"*: `Yes`
>    - *"~multiallelics"*: `split multiallelic sites into biallelic records (-)`
>    - In *"Restrict all operations to"*:
>        - *"Regions"*: `Do not restrict to Regions`
>        - *"Targets"*: `Do not restrict to Targets`
>    - *"output_type"*: `uncompressed VCF`
>
> > <tip-title>Check the normalization summary</tip-title>
> >
> > You can have a look at the summary to check what changes were made. First, expand the output of bcftools norm (by clicking on the box) and it should be listed in the box. If not you can find it by clicking on the icon view details {% icon details %} and look at the output of the ToolStandard Error.
> >
> {: .tip}
>
{: .hands_on}

## Filter NON_REF sites
After normalizing the VCFs we will filter out the variants with a NON_REF tag in the ALT column, the column which represents the mutated nucleotide(s). According to the header, these sites correspond to: "any possible alternative allele at this location". So these sites are a sort of placeholders for potential variants. However we are not interested in this kind of variants and they slow our analysis down quite a lot, so we will filter them out.

> <hands-on-title>Filter out NON_REF sites</hands-on-title>
>
> 1. {% tool [Filter](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `Normalized VCFs` (output of **bcftools norm** {% icon tool %})
>    - *"With following condition"*: `c5!='<NON_REF>'`
>    - *"Number of header lines to skip"*: `142` This has to be set manually since the tool skips lines starting with '#' automatically.
>
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Why didn't we filter out the NON_REF site directly after filtering? Have a look at the VCFs before and after normalization.
> 2. Could we have filtered out the NON_REF sites earlier with a different program?
>
> > <solution-title></solution-title>
> > 1. The `<NON_REF>` sites were sometimes also represented as a multi allelic variant e.g., `chr17	302	.	T	TA,<NON_REF>` which would not be filtered out with the {% tool [Filter](Filter1) %} tool.
> > 2. Possibly, if the `<NON_REF>` variants were filtered out and if the `<NON_REF>` tag was removed from multi-allelic variants. However, the last task (splitting multi-allelic sites) can already be handled by {% tool [bcftools norm](toolshed.g2.bx.psu.edu/repos/iuc/bcftools_norm/bcftools_norm/1.9+galaxy1) %} and it might involve more sophisticated steps to properly split these variants.
> {: .solution}
>
{: .question}

## Merge VCF collection into one dataset
Finally, we can merge the 3 separate files of the parents and patient into a single VCF. This will put overlapping variants on the same line by aligning the samples format column. If a sample misses a certain variant then on the line of that variant the sample's format information will look like this: `./.:.:.:.:.:.`. This makes it easier to find shared and missing variants between the parents and ofspring. A tool which can do this is the {% tool [bcftools merge](toolshed.g2.bx.psu.edu/repos/iuc/bcftools_merge/bcftools_merge/1.10) %} tool.

> <hands-on-title>Merge VCFs</hands-on-title>
>
> 1. {% tool [bcftools merge](toolshed.g2.bx.psu.edu/repos/iuc/bcftools_merge/bcftools_merge/1.10) %} with the following parameters:
>    - {% icon param-file %} *"Other VCF/BCF Datasets"*: `<NON_REF> filtered VCFs` (output of **Text reformatting** {% icon tool %})
>    - In *"Restrict to"*:
>        - *"Regions"*: `Do not restrict to Regions`
>    - In *"Merge Options"*:
>        - *"Merge"*: `none - no new multiallelics, output multiple records instead`
>    - *"output_type"*: `uncompressed VCF`
>
>    > <comment-title>Checking the merged VCF.</comment-title>
>    >
>    > Check the merged VCF, now each line should contain 3 sample columns, namely `Case6F`, `Case6M`, and `Case6C`. These columns represent the presence of the variant for the mother, father, and offspring.
>    {: .comment}
>
{: .hands_on}

# Annotation
To understand what the effect of our variants are, we need to annotate our variants. We will use the tool {% tool [SnpEff eff](toolshed.g2.bx.psu.edu/repos/iuc/snpeff/snpEff/4.3+T.galaxy1) %}, which will compare our variants to a database of variants with known effects.


## Annotate with SNPeff
Running SnpEff will produce the annotated VCF and an HTML summary file. The annotations are added to the INFO column in the VCF and the added INFO IDs (`ANN`, `LOF`, and `NMD`) are explained in the header. The summary files include the HTML stats file which contains general metrics, such as the number of annotated variants, the impact of all the variants, and much more.

> <hands-on-title>Annotation with SnpEff</hands-on-title>
>
> 1. {% tool [SnpEff eff:](toolshed.g2.bx.psu.edu/repos/iuc/snpeff/snpEff/4.3+T.galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Sequence changes (SNPs, MNPs, InDels)"*: `Merged VCF` (output of **bcftools merge** {% icon tool %})
>    - *"Output format"*: `VCF (only if input is VCF)`
>    - *"Genome source"*: `Locally installed snpEff database`
>        - *"Genome"*: `Homo sapiens : hg19`
>    - *"spliceRegion Settings"*: `Use Defaults`
>    - *"Filter out specific Effects"*: `No`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many variants got annotated?
> 2. Is there something notable you can see from the HTML file?
>
> > <solution-title></solution-title>
> > 1. The number of variants processed is 209,143.
> > 2. Most variants are found in the intronic regions. However, this is to be expected since the mutations in intonic regions generally do not affect the gene.
> {: .solution}
>
{: .question}

## Compress VCF
Now that we have finished running all the tools which require the VCF format as input, they can be converted back to the **vcf_bgzip** format. To compress the VCF we will use a built-in tool from Galaxy, which can be accessed by manipulating the file itself in a similair fashion as changing its detected type.

> <hands-on-title> Compress **vcf** back to **vcf_bgzip** </hands-on-title>
>
> Execute the following steps for the SnpEff VCF.
>
> 1. Click on the **pencil** {% icon galaxy-pencil %} of the SnpEff VCF.
>
> 2. Click on the **convert tab** {% icon galaxy-gear %}.
>
> 3. Under **Target datatype** select `vcf_bgzip (using 'Convert uncompressed file to compressed.')`
>
> 4. Click {% icon exchange %} **Create Dataset**.
>
{: .hands_on}

# Trio Analysis

## GEMINI analyses
Next, we will transform our VCF into a GEMINI database which makes it easier to query the VCFs and to determine different inheritence patterns between the mother, father, and offspring. In addition, GEMINI will add even more annotations to the variants. This allows us to filter the variants even more which gets us to closer to the real causative variant.
All these steps are performed by the {% tool [GEMINI load](toolshed.g2.bx.psu.edu/repos/iuc/gemini_load/gemini_load/0.20.1+galaxy2) %} tool. However, before we can load the VCF we also need to define a pedigree file.

### Create a pedigree file describing the family trio
A pedigree file is a file that informs GEMINI which family members are affected by the disease and which sample name corresponds to what individual. This information is saved as a table containing information about the phenotype of the family.

1. **#family_id**: Which family a row belongs to.
2. **name**: The name of the sample, note this sample name has to overlap with the sample name in the VCF, see the last 3 columns of the VCF. However, for some reason the VCF contains the sample name from case 6 and not case 5. This was probably just a typo in the VCF, however here we just copy the sample name to our pedigree file.
3. **paternal_id**: The sample name of the father or 0 for missing.
4. **maternal_id**: The sample name of the mother or 0 for missing.
5. **sex**: The sex of the person.
6. **phenotype**: Wether or not the person is affected by the disease.

> <hands-on-title>Creating the PED file</hands-on-title>
>
> 1. Upload the pedigree file from below.
> ```
> #family_id	name	paternal_id	maternal_id	sex	phenotype
> FAM0001822	Case6M	0	0	2	2
> FAM0001822	Case6F	0	0	1	1
> FAM0001822	Case6C	Case6F	Case6M	2	2
> ```
>
> {% snippet faqs/galaxy/datasets_create_new_file.md %}
>
{: .hands_on}

For more information on the PED file you can read the help section of the {% tool [GEMINI load](toolshed.g2.bx.psu.edu/repos/iuc/gemini_load/gemini_load/0.20.1+galaxy2) %} tool in the description, which can be found at the bottom of the page when clicking on the tool.

### Load GEMINI database
Now we can transform the subsampled VCF and PED file into a GEMINI database. Note that this can take a very long time depending on the size of the VCF. In our case it should take around 30-40 minutes.

> <hands-on-title>Transform VCF and PED files into a GEMINI database</hands-on-title>
>
> 1. {% tool [GEMINI load](toolshed.g2.bx.psu.edu/repos/iuc/gemini_load/gemini_load/0.20.1+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"VCF dataset to be loaded in the GEMINI database"*: `SnpEff Annotated vcf_bgzip` (output of **Convert uncompressed file to compressed.** {% icon tool %})
>    - {% icon param-file %} *"Sample and family information in PED format"*: the pedigree file prepared above
>
>
{: .hands_on}


### Find inheritance pattern
With the GEMINI database it is now possible to identify the causative variant that could explain the breast cancer in the mother and daughter. The inheritance information makes it a bit easier to determine which tool to run to find the causative variant, instead of finding it by trying all the different inheritance patterns.

> <question-title></question-title>
>
> Which inheritance pattern could have occurred in this family trio?
>
> > <tip-title>Possible patterns</tip-title>
> > The available inheritance patterns can be found in the {% tool [GEMINI inheritance pattern](toolshed.g2.bx.psu.edu/repos/iuc/gemini_inheritance/gemini_inheritance/0.20.1) %} tool.
> {: .tip}
>
> > <solution-title></solution-title>
> >
> >    - Since both the daughter and mother are affected, and the father likely unaffected[^1], the mutation is most probably dominant. The disease could still be recessive if the father has one copy of the faulty gene, however this is less likely.
> >    - The mutation is most likely not de-novo, since both the daughter and mother are affected.
> >    - The mutation could be X-linked, however since the VCF only contain mutations on chromosome 17 it would practically be impossible.
> >    - The disease could be compound heterozygous, however in that case the disease should be recessive which is less likely.
> >    - A loss of heterozygosity (LOH) is also possible, since it is a common occurrence in cancer so this could be a viable inheritance pattern.
> >
> {: .solution}
>
{: .question}

[^1]:  It is unlikely that the father has breast cancer, "For men, the lifetime risk of getting breast cancer is about 1 in 833" [via cancer.org](https://www.cancer.org/cancer/breast-cancer-in-men/about/key-statistics.html)

Based on these findings it would make sense to start looking for inherited autosomal dominant variants as a first step. If there are no convincing candidate mutations it would always be possible to look at the other less likely inheritance patterns, namely de-novo, compound heterozygous, and LOH events.

To find the most plausible causative variant we will use the {% tool [GEMINI inheritance pattern](toolshed.g2.bx.psu.edu/repos/iuc/gemini_inheritance/gemini_inheritance/0.20.1) %} tool. This tool allows us to select the most likely inheritance pattern (autosomal dominant). Below it is explained how to run the tool for this specific pattern, but you can always try other inheritence patterns if you are curious.

> <hands-on-title>Run GEMINI autosomal dominant inhertiance pattern</hands-on-title>
>
> 1. {% tool [GEMINI inheritance pattern](toolshed.g2.bx.psu.edu/repos/iuc/gemini_inheritance/gemini_inheritance/0.20.1) %} with the following parameters:
>    - {% icon param-file %} *"GEMINI database"*: `GEMINI database` (output of **GEMINI load** {% icon tool %})
>    - *"Your assumption about the inheritance pattern of the phenotype of interest"*: `Autosomal dominant`
>        - In *"Additional constraints on variants"*:
>            - {% icon param-repeat %} *"Insert Additional constraints on variants"*
>                - *"Additional constraints expressed in SQL syntax"*: `impact_severity != 'LOW'` To filter variants on their functional genomic impact we will use the impact_severity feature. Here a low severity means a variant with no impact on protein function, such as silent mutations.
>    - In *"Family-wise criteria for variant selection"*:
>        - *"Specify additional criteria to exclude families on a per-variant basis"*: `No, analyze all variants from all included families`
>    - In *"Output - included information"*:
>        - *"Set of columns to include in the variant report table"*: `Custom (report user-specified columns)`
>            - *"Additional columns (comma-separated)"*: `chrom, start, ref, alt, impact, gene, clinvar_sig, clinvar_disease_name, clinvar_gene_phenotype, rs_ids`
>
{: .hands_on}


> <question-title></question-title>
>
> Did you find the causative variant in the output of the GEMINI inheritance pattern tool?
>
> > <solution-title></solution-title>
> > The only pathogenic variant related to breast cancer in the output, according to clinvar, is a SNP at chr17 at position 41215919 on the BRCA1 gene which transforms a G into a T, which is shown below.
> > ```
> > chr17	41215919	G	T	missense_variant	BRCA1	pathogenic,other
> > ```
> > This missense mutation transforms an alanine amino acid into a glutamine amino acid. Even though this variant has an unknown clinical significance in BRCA1 it was found to be among the top 10 SNPs which likely leads to breast cancer, according to {% cite Easton2007 %}. You can find more info on this mutation by googling it's rs_ID rs28897696.
> {: .solution}
>
{: .question}


## Gene.iobio Analyses
Alternatively, the trio analysis can be done using gene.iobio {% cite Di_Sera_2021 %}. The {% tool [gene.iobio visualisation](toolshed.g2.bx.psu.edu/repos/iuc/geneiobio/gene_iobio_display_generation_iframe/4.7.1) %} tool generates an html file which sends the VCFs to the iobio site using public URLs. Therefore, your history has to be shared, i.e. **everyone with a link can access your history!**


> <hands-on-title> Make files accessible for gene.iobio </hands-on-title>
>
> 1. Open the **History Options menu** {% icon galaxy-dropdown %} at the topright corner of your history panel.
> 2. select **Share or Publish** {% icon workflow %}.
> 3. Then toggle the {% icon galaxy-toggle %} **Make History accessible**.
>
{: .hands_on}

For this tool it is not required to set any filtering steps or determine the most plausible inheritence pattern. Since gene.iobio runs multiple default filtering steps for autosomal dominant, recessive, *De novo*, compound heterozygous, and X-linked recessive inheritance patterns. In addition, it prioritizes variants based on gene-disease association algorithms, which significantly reduces processing time.

> <hands-on-title> Generate gene.iobio html </hands-on-title>
>
> 1. {% tool [gene.iobio visualisation](toolshed.g2.bx.psu.edu/repos/iuc/geneiobio/gene_iobio_display_generation_iframe/4.7.1) %} with the following parameters:
>    - {% icon param-file %} *"Proband VCF file"*: `SnpEff Annotated vcf_bgzip` (output of **Convert compressed file to uncompressed.** {% icon tool %})
>    - *"Proband sex"*: `Female`
>    - *"Proband sample name"*: `Case6C`
>    - *"Single/Trio analysis"*: `Trio`
>        - In *"Father input"*:
>            - {% icon param-file %} *"Father VCF file"*: `SnpEff Annotated vcf_bgzip` (output of **Convert compressed file to uncompressed.** {% icon tool %})
>            - *"Father sample name"*: `Case6F`
>        - In *"Mother input"*:
>            - {% icon param-file %} *"Mother VCF file"*: `SnpEff Annotated vcf_bgzip` (output of **Convert compressed file to uncompressed.** {% icon tool %})
>            - *"Is the mother affected?"*: `Yes`
>            - *"Mother sample name"*: `Case6M`
>    - *"Select reference genome version"*: `GRCh37`
>
>    > <tip-title> Using gene.iobio </tip-title>
>    >
>    > To open the generated html file click on the **eye** {% icon galaxy-eye %}, which will show the gene.iobio interface. Now a phenotype has to be added to generate a list of genes which likely contain causative variants, in this case it is breast cancer. Next, click on the **Hamburger icon** in the topleft corner of the gene.iobio interface. This will show 20 genes including the BRCA1 gene with the causative variant under the **Genes** tab. Click on the BRCA1 gene and gene.iobio starts analyzing the gene for causative variants. Then under the **Variants** tab you will find the causative variant listed. You can click on the variant block to see more information on it. A report will show with information on the type, location, and quality of the variant, the associated phenotype, the pathogenicity, the population frequency, conservation of the location of the variant, and the inheritence pattern (Autosomal dominant).
>    {: .tip}
>
>    > <tip-title> Increase genes to analyze </tip-title>
>    >
>    > More genes can be analyzed in three ways 1) the generated list of genes associated with a phenotypm can be increased. 2) The list of genes associated with another phenotype can be combined with the current list of genes. 3) A gene can manually be added.
>    >
>    > 1. To increase the number of genes in the generated list of genes click on the **gear icon** {% icon galaxy-gear %} in gene.iobio and click on the drop down box under **Keep top n genes from phenotype search**.
>    > 2. To add more genes to the analysis type and select another phenotype in the top bar. After finding your phenotype of interest press enter and click **Combine genes with current list** in the pop-up screen.
>    > 3. To add a single gene to the analysis type and select a gene name in the top bar. After finding your gene of interest press enter.
>    {: .tip}
>
{: .hands_on}

# Conclusion
In this tutorial we have illustrated how to easily download a dataset of interest savely and securely from the EGA using the HTSGET protocol. In addition, we performed a trio analysis to find the causative variant in an autosomal dominant inherited disease. We pre-processed and annotated our VCFs using the SnpEff and found the causative variant using both the GEMINI and gene.iobio tool. With this workflow you can now easily analyse and find the causative variant(s) in many different family trios from any database which HTSGET can connect to.

# Workflow
Here is the final layout of the workflow. For more details you can download the workflow from the overview at the top of the page.

![Image showing the whole workflow of the tutorial. Each seperate step is represented by a rectangular block with the tool name in the upper part in blue and the lower part shows the output in white. In the top left the workflow starts with the EGA Download Client which ouputs the authorized datasets (txt), below that again the EGA Download Client is shown which outputs the list of files in the ega dataset, which is connected to the search in textfiles tool. That tool is connected again to an EGA Download Client step where the VCFs are downloaded using the list of files from the previous tool. This step is connected to a decompress step to change the bgzipped VCFs to regular VCFs. That step is connected to a Filter step which outputs a VCF. The next connected step is the bcftools norm tool which outputs a normalized VCF. Then the normalized VCFs are processed using the Text reformatting tool. Using the output from that step the bcftools megre step is used to output a single VCF and connect it to the SnpEff eff tool. This tool annotates the VCF and that step is connected to the GEMINI load tool which is also connected to step block with the title pedigree. Then the GEMINI load tool connects to the GEMINI inheritance pattern block which outputs an tabular file called GEMINI autosomal_dominant pattern](../../images/trio-analysis/workflow.png "The workflow to download the VCFs and find the causative variant. If you skipped the EGA download step then the workflow starts at the second column. Here each column represents a seperate step, from left to right the steps are: Data preparation, Pre-Processing, Annotation, and GEMINI analysis.")


topic: variant-analysis
tutorial: exome-seq
title: Exome sequencing data analysis for diagnosing a genetic disease
difficulty: intermediate
time_estimation: 5h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 65d40891600110c1cf9f51ee0e88dd58
last_modified: 2026-02-26T21:12:18.617618
zenodo_link: https://doi.org/10.5281/zenodo.3054169
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/variant-analysis/tutorials/exome-seq/tutorial.html

content:
Exome sequencing is a method that enables the selective sequencing of the
exonic regions of a genome - that is the transcribed parts of the genome present
in mature mRNA, including protein-coding sequences, but also untranslated
regions (UTRs).

In humans, there are about 180,000 exons with a combined length of ~ 30 million
base pairs (30 Mb). Thus, the exome represents only 1% of the human genome, but
has been estimated to harbor up to 85% of all disease-causing variants ([Choi
et al., 2009](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2768590/)).

Exome sequencing, thus, offers an affordable alternative to whole-genome
sequencing in the diagnosis of genetic disease, while still covering far more
potential disease-causing variant sites than genotyping arrays. This is of
special relevance in the case of rare genetic diseases, for which the causative
variants may occur at too low a frequency in the human population to be
included on genotyping arrays.

Of note, a recent study focusing on the area of clinical pediatric neurology
indicates that the costs of exome sequencing may actually not be higher even
today than the costs of conventional genetic testing ([Vissers et al., 2017](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5589982/)).

> <details-title>Exome sequencing <i>vs</i> whole-genome sequencing</details-title>
>
> In principle, the steps illustrated in this tutorial are suitable also for
> the analysis of whole-genome sequencing (WGS) data. At comparable mean
> coverage, however, WGS datasets will be much larger than exome sequencing
> ones and their analysis will take correspondingly more time.
>
> The obvious benefit of WGS compared to exome-sequencing, of course, is that
> it will allow variant detection in even more regions of the genome.
> As a less apparent advantage, the more complete information of WGS data can
> make it easier to detect *copy number variation (CNV)* and
> *structural variants* such as translocations and inversions (although such
> detection will require more sophisticated analysis steps, which are not
> covered by this tutorial).
>
> Very generally, one could argue that exome-sequencing captures most of the
> information that can be analyzed with standard bioinformatical tools today at
> reasonable costs. WGS, on the other hand, captures as much information as
> today's sequencing technology can provide, and it may be possible to
> reanalyze such data with more powerful bioinformatical software in the
> future to exploit aspects of the information that were not amenable to
> analysis at the time of data acquisition.
{: .details}

The identification of causative variants underlying any particular genetic
disease is, as we will see in this tutorial, not just dependent on the
successful detection of variants in the genome of the patient, but also on
variant comparison between the patient and selected relatives. Most often
family trio data, consisting of the genome sequences of the patient and their
parents, is used for this purpose. With multisample data like this it becomes
possible to search for variants following any kind of Mendelian inheritance
scheme compatible with the observed inheritance pattern of the disease, or to
detect possibly causative *de-novo* mutations or *loss-of-heterozygosity* (LOH) events.

> <details-title>Related tutorials</details-title>
>
> This tutorial focuses on the practical aspects of analyzing real-world
> patient data. If you are more interested in the theoretical aspects of
> variant calling, you may want to have a look at the related tutorial on
> [Calling variants in diploid systems](../dip/tutorial.html).
>
> The tutorial on [Somatic variant calling](../somatic-variants/tutorial.html)
> follows an analysis workflow that is rather similar to the one here, but
> tries to identify tumor variants by comparing a tumor sample to healthy
> tissue from the same patient. Together, the two tutorials are intended to get
> you started with genomics medicine using Galaxy.
>
{: .details}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data Preparation

In this tutorial, we are going to analyze exome sequencing data from a family
trio, in which the boy child is affected by the disease
[osteopetrosis](https://ghr.nlm.nih.gov/condition/osteopetrosis/), while both
parents, who happen to be consanguineous, are unaffected. Our goal is to
identify the genetic variation that is responsible for the disease.

## Get data

{% icon tip %} This tutorial offers **two alternative entry points** allowing
you to

- conduct a full analysis starting from original sequenced reads in `fastq`
  format or to
- start your analysis with premapped reads in `bam` format that are (almost)
  ready for variant calling.

While the full analysis is probably closer to how you would analyze your own
data, the shortened analysis from premapped reads may suit your time frame
better, and will avoid redundancy if you have previously worked through other
tutorials demonstrating NGS quality control and read mapping, like the
dedicated
[Quality control]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %})
and [Mapping]({% link topics/sequence-analysis/tutorials/mapping/tutorial.md %})
tutorials.

The following hands-on section will guide you through obtaining the right
data for either analysis.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a meaningful name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Obtain the raw sequencing data
>
>    > <comment-title>Starting from raw sequencing data</comment-title>
>    > In this and the following steps you will obtain the original unmapped
>    > sequencing data and prepare for a full analysis including the mapping of
>    > the sequenced reads.
>    >
>    > If you prefer **to skip the mapping step** and start the analysis from
>    > premapped data **you should proceed directly to step 4** of this
>    > section.
>    {: .comment}
>
>    Import the original sequenced reads datasets of the family trio from
>    [Zenodo](https://zenodo.org/record/3054169):
>    ```
>    https://zenodo.org/record/3243160/files/father_R1.fq.gz
>    https://zenodo.org/record/3243160/files/father_R2.fq.gz
>    https://zenodo.org/record/3243160/files/mother_R1.fq.gz
>    https://zenodo.org/record/3243160/files/mother_R2.fq.gz
>    https://zenodo.org/record/3243160/files/proband_R1.fq.gz
>    https://zenodo.org/record/3243160/files/proband_R2.fq.gz
>    ```
>
>    Alternatively, the same files may be available on your Galaxy server
>    through a shared data library (your instructor may tell you so), in
>    which case you may prefer to import the data directly from there.
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="fastqsanger.gz" %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Check that the newly created datasets in your history have their
>    datatypes assigned correctly to `fastqsanger.gz`, and fix any missing or
>    wrong datatype assignment
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="fastqsanger.gz" %}
>
>    {% icon trophy %} Congratulations for obtaining the datasets required for
>    an analysis including reads mapping. You should now
>    **proceed with Step 7** below.
>
> 4. Obtain the premapped sequencing data
>    > <comment-title>Alternative entry point: Premapped data</comment-title>
>    > Skip this and the following two steps if you already obtained and
>    > prepared the original unmapped seuencing data and are planning to
>    > perform the mapping step yourself.
>    {: .comment}
>
>    Import the premapped reads datasets of the family trio from
>    [Zenodo](https://zenodo.org/record/3054169):
>    ```
>    https://zenodo.org/record/3243160/files/mapped_reads_father.bam
>    https://zenodo.org/record/3243160/files/mapped_reads_mother.bam
>    https://zenodo.org/record/3243160/files/mapped_reads_proband.bam
>    ```
>
>    Alternatively, the same files may be available on your Galaxy server
>    through a shared data library (your instructor may tell you so), in
>    which case you may prefer to import the data directly from there.
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="bam" %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 5. Check that the newly created datasets in your history have their
>    datatypes assigned correctly to `bam`, and fix any missing or wrong
>    datatype assignment
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="bam" %}
>
> 6. Specify the genome version that was used for mapping
>
>    Change the database/build (dbkey) for each of your bam datasets
>    to `hg19`.
>
>    {% snippet faqs/galaxy/datasets_change_dbkey.md dbkey="Human Feb. 2009 (GRCh37/hg19) (hg19)" %}
>
>    > <details-title>Why specify genome versions</details-title>
>    > When you are starting with sequencing data that has already been mapped
>    > to a particular genome version (human hg19 in this case), it is good
>    > practice to attach this information as metadata to the datasets.
>    >
>    > Doing so helps prevent accidental use of a different version of the
>    > reference genome in later steps like variant calling, which would
>    > typically lead to nonsensical results because of base position changes
>    > between the different versions.
>    {: .details}
>
>    {% icon trophy %} Congratulations for obtaining the premapped sequencing
>    datasets. Now, **follow the remaining steps** to set everything up for a
>    successful analysis.
>
> 7. Rename the datasets
>
>    For datasets that you upload via a link, Galaxy will pick the link
>    address as the dataset name, which you will likely want to shorten to
>    just the file names.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 8. Add #father/#mother/#child tags to the datasets
>
>    Parts of the analysis in this tutorial will consist of identical steps
>    performed on the data of each family member.
>
>    To make it easier to keep track of which dataset represents which step in
>    the analysis of which sample, Galaxy supports dataset tags. In particular,
>    if you attach a tag starting with `#` to any dataset, that tag will
>    automatically propagate to any new dataset derived from the tagged
>    dataset.
>
>    Tags are supposed to help you identify the origin of datasets quickly,
>    but you can choose them as you like.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
> 9. Obtain the reference genome
>
>    > <comment-title>Shortcut</comment-title>
>    > You can skip this step if the Galaxy server you are working on offers
>    > a `hg19` version of the human reference genome with prebuilt indexes for
>    > *bwa-mem* (only necessary if starting from unmapped original sequencing
>    >reads) and *freebayes*. Ask your instructor, or check the tools
>    > **Map with BWA-MEM** {% icon tool %} and **FreeBayes** {% icon tool %}
>    > if they list a `hg19` version as an option under
>    > *"Select a / Using reference genome"*).
>    {: .comment}
>
>    Import the `hg19` version of the human chromosome 8 sequence:
>    ```
>    https://zenodo.org/record/3243160/files/hg19_chr8.fa.gz
>    ```
>
>    In the upload dialog, make sure you specify:
>
>    - **Type**: `fasta`
>    - **Genome**: `Human Feb. 2009 (GRCh37/hg19) (hg19)`
>
>    Alternatively, load the dataset from a shared data library.
>
> 10. Rename the reference genome
>
>     The reference genome you have imported above came as a compressed
>     file, but got unpacked by Galaxy to plain `fasta` format according to
>     your datatype selection. At a minimum, you may now wish to remove the
>     `.gz` suffix from the dataset name to avoid confusion.
>
{: .hands_on}

{% icon trophy %} **Congratulations!**
You are all set for starting the analysis now.

If you have chosen to follow the complete analysis from the original sequenced
data, just proceed with the next section.

If, on the other hand, you have prepared to start from the premapped data, skip
the sections on *Quality control* and *Read mapping*, and conitnue with
**Mapped reads postprocessing**.

# Quality control

This step serves the purpose of identifying possible issues with the raw
sequenced reads input data before embarking on any "real" analysis steps.

Some of the typical problems with NGS data can be mitigated by preprocessing
affected sequencing reads before trying to map them to the reference genome.
Detecting some other, more severe problems early on may at least save you a lot
of time spent on analyzing low-quality data that is not worth the effort.

Here, we will perform a standard quality check on our input data and only point
out a few interesting aspects about that data. For a more thorough explanation
of NGS data quality control, you may want to have a look at the dedicated
tutorial on [Quality control]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).

> <hands-on-title>Quality control of the input datasets</hands-on-title>
> 1. Run {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy0) %} on each of your six fastq datasets
>       - {% icon param-files %} *"Short read data from your current history"*: all 6 FASTQ  datasets selected with **Multiple datasets**
>
>    {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>
>    When you start this job, twelve new datasets (one with the calculated raw
>    data, another one with an html report of the findings for each input
>    dataset) will get added to your history.
>
> 2. Use {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy1) %} to aggregate the raw **FastQC** data of all input datasets into one report
>      - In *"Results"*
>        - *"Which tool was used generate logs?"*: `FastQC`
>        - In *"FastQC output"*
>           - *"Type of FastQC output?"*: `Raw data`
>           - {% icon param-files %} *"FastQC output"*: all six *RawData*
>             outputs of **FastQC** {% icon tool %})
>
> 3. Inspect the *Webpage* output produced by the tool
>
>    > <question-title></question-title>
>    >
>    > 1. Based on the report, do you think preprocessing of the reads
>    >    (trimming and/or filtering) will be necessary before mapping?
>    > 2. Why do all samples show a non-normal GC content distribution, and
>    >    should you be worried?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Sequence quality is quite good overall. If anything you might
>    > >    consider trimming the 3' ends of reads (base qualities decline
>    > >    slightly towards the 3' ends) or to filter out the small fraction
>    > >    of reads with a mean base quality < 5.
>    > >    Feel free to run, *e.g.*, {% tool [Trimmomatic](toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.38.1) %} on the
>    > >    fastq datasets if you want to, but don't expect this to have a big
>    > >    effect on the analysis given the high overall quality of the data
>    > >    of all samples.
>    > > 2. In whole-genome sequencing, a non-normal distribution of the GC
>    > >    content of the reads from a sample is typically considered to hint
>    > >    at possible contamination.
>    > >    Here, however, we are dealing with sequencing data from captured
>    > >    exomes, *i.e*, the reads are not representing random sequences from
>    > >    a genome, but rather a biased selection.
>    > >
>    > >    A bimodal GC content distribution, like for the samples at hand, is
>    > >    a characteristic feature of many exome capture methods and has also
>    > >    been observed with an Illumina Nextera Rapid Capture exome kit
>    > >    before (compare Fig. 3D in
>    > >    [Shigemizu et al., 2015](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4522667/)
>    > >    ).
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

# Read mapping

Now that you confirmed that the quality of the input data is good enough to
warrant further analysis, it is time to map the sequenced reads to the
reference genome.

We assume here that you are at least vaguely familiar with the concept of read
mapping and only illustrate the concrete steps necessary  to map our specific
NGS reads datasets to the human reference genome. We recommend you to follow
the dedicated [Mapping tutorial]({% link topics/sequence-analysis/tutorials/mapping/tutorial.md %}), if you need a general introduction to read mapping.

> <hands-on-title>Read Mapping</hands-on-title>
> 1. {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2) %} to map the reads from the **father** sample to the reference genome
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a built-in genome index`
>        - *"Using reference genome"*: `Human: hg19` (or a similarly named option)
>
>      > <comment-title>Using the imported `hg19` sequence</comment-title>
>      > If you have imported the `hg19` chr8 sequence as a fasta dataset into
>      > your history instead:
>      >   - *"Will you select a reference genome from your history or use a
>      >     built-in index?"*: `Use a genome from history and build index`
>      >      - {% icon param-file %} *"Use the following dataset as the reference sequence"*: your imported `hg19` fasta dataset.
>      {: .comment}
>
>    - *"Single or Paired-end reads"*: `Paired`
>       - {% icon param-file %} *"Select first set of reads"*: the
>         forward reads (R1) dataset of the **father** sample
>       - {% icon param-file %} *"Select second set of reads"*: the
>         reverse reads (R2) dataset of the **father** sample
>
>      > <tip-title>No FASTQ datasets selectable?</tip-title>
>      > Please confirm that the problematic datasets declare *format*:
>      > `fastqsanger.gz`.
>      >
>      > Most Galaxy tools that accept FASTQ input expect the data to be
>      > formatted as *FASTQ with Sanger-scaled quality values*, the most
>      > widely spread version of the FASTQ format. To make this requirement
>      > explicit (instead of generating possibly wrong results) these tools
>      > require you to set the dataset type to `fastqsanger` (`fastqsanger.gz`
>      > for data compressed with gzip). You can do so either on data upload
>      > or later from the *Edit dataset attributes* view (which you can reach
>      > by clicking on the {% icon galaxy-pencil %} pencil icon.
>      {: .tip}
>
>    - *"Set read groups information?"*: `Set read groups (SAM/BAM specification)`
>      - *"Auto-assign"*: `No`
>        - *"Read group identifier (ID)"*: `000`
>      - *"Auto-assign"*: `No`
>        - *"Read group sample name (SM)"*: `father`
>
>    > <warning-title>Read group IDs and sample names - choose, but choose wisely</warning-title>
>    > In general, you are free to choose ID and SM values to your liking, but
>    > ...
>    >
>    > The **ID** should **unambiguously identify** the sequencing run that
>    > produced the reads. At the very least, no two input datasets in any
>    > given analysis should define the same ID twice, or tools like
>    > *FreeBayes*, which we are going to use in the next step, will refuse
>    > to work with the data.
>    >
>    > The **SM** value, on the other hand, should identify the biological
>    > sample represented by the data and is used by many tools (like *GEMINI*
>    > which we will use later) to let you refer to one specifc sample in a
>    > multisample analysis. Choose descriptive, but short and easy to
>    > remember sample names since you will have to type them in again!
>    >
>    {: .warning}
>
> 2. {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2) %} to map the reads from the **mother** sample to the reference genome **using the same parameters as before** except
>
>    - *"Single or Paired-end reads"*: `Paired`
>       - {% icon param-file %} *"Select first set of reads"*: the
>         forward reads (R1) dataset of the **mother** sample
>       - {% icon param-file %} *"Select second set of reads"*: the
>         reverse reads (R2) dataset of the **mother** sample
>
>    - *"Set read groups information?"*: `Set read groups (SAM/BAM specification)`
>      - *"Auto-assign"*: `No`
>        - *"Read group identifier (ID)"*: `001`
>      - *"Auto-assign"*: `No`
>        - *"Read group sample name (SM)"*: `mother`
>
> 3. {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2) %} to map the reads from the **child** sample to the reference genome **using the same parameters as before** except
>
>    - *"Single or Paired-end reads"*: `Paired`
>       - {% icon param-file %} *"Select first set of reads"*: the
>         forward reads (R1) dataset of the **child** sample
>       - {% icon param-file %} *"Select second set of reads"*: the
>         reverse reads (R2) dataset of the **child** sample
>
>    - *"Set read groups information?"*: `Set read groups (SAM/BAM specification)`
>      - *"Auto-assign"*: `No`
>        - *"Read group identifier (ID)"*: `002`
>      - *"Auto-assign"*: `No`
>        - *"Read group sample name (SM)"*: `proband`
>
{: .hands_on}

# Mapped reads postprocessing

At this point in the analysis you should have obtained three mapped reads
datasets in `bam` format. Each of these datasets should:

- have its *database* set to the key `hg19`

  Please correct any missing (`?`) or wrong keys now!

  {% snippet faqs/galaxy/datasets_change_dbkey.md dbkey="Human Feb. 2009 (GRCh37/hg19) (hg19)" %}

- ideally, carry one of the `#father`, `#mother` or `#child` tags
  for quick identification of the samples they provide data for.

In principle, you could use these datasets directly for variant calling, and in
many cases, including this one, this would be sufficient to identify the
sought-after variants.

To obtain an accurate picture of the variant spectrum found in your samples it
is good practice though to perform various postprocessing steps on the mapped
reads before passing them to a variant caller.

> <comment-title>Shortcut</comment-title>
> If you are in a hurry, you may skip the postprocessing steps and continue
> with your mapped reads datasets and the *Variant Calling* section. Make sure
> though you come back here, or read about mapped reads postprocessing
> elsewhere, before attempting to perform any variant analysis on your own data
> that may require high accuracy/sensitivity.
{: .comment}

The optimal set of postprocessing steps required depends on the variant calling
software used at the next step. The **FreeBayes** variant caller that we are
going to use in this tutorial is particularly well suited for use with minimal
mapped reads postprocessing pipelines, so all we are going to do here is:

- filter the paired-end reads of all samples to retain only those read pairs,
  for which both the forward and the reverse read have been mapped to the
  reference successfully

  For such pairs of reads, we can be extra confident that they don't
  come from some non-human contaminant DNA or represent a sequencing artefact
  of some sort.

- deduplicate reads

  Duplicate reads, which typically arise from PCR-overamplification of genomic
  fragments during sequencing library preparation, can, to some extent, lead to
  wrong genotype assignments at variant sites (if, for example, a sample is
  heterozygous for a variant, but fragments with one of the two alleles get
  amplified more efficiently than the others).

> <comment-title>More postprocessing steps</comment-title>
> The [Somatic variant calling tutorial](../somatic-variants/tutorial.html)
> provides an example of a much more complex mapped reads postprocessing
> pipeline that may be required for optimal performance of other variant
> callers.
{: .comment}

## Filtering on mapped reads properties

To produce new filtered BAM datasets with only mapped reads the mate of which is also mapped:

> <hands-on-title>Filtering for read pair mapping status</hands-on-title>
>
> 1. {% tool [Samtools view](toolshed.g2.bx.psu.edu/repos/iuc/samtools_view/samtools_view/1.15.1+galaxy0) %} with the following
> parameters (leaving non-mentioned ones at their defaults):
>   - {% icon param-files %} *"SAM/BAM/CRAM data set"*: all 3 mapped reads
>     datasets of the family trio, outputs of **Map with BWA-MEM**
>     {% icon tool %}
>   - *"What would you like to look at?"*: `A filtered/subsampled selection of reads`
>     - In *"Configure filters"*:
>       - *"Exclude reads with any of the following flags set"*:
>         `Read is unmapped` **and** `Mate is unmapped`
>
{: .hands_on}

This will result in three new datasets, one for each sample in the analysis.

> <details-title>More than one way to filter</details-title>
> Instead of the above filter conditions we could also have exploited the
> *Read is mapped in a proper pair* flag bit.
>
> For a read to be flagged as being mapped in a proper pair its mate needs to
> be mapped, but the mapped pair also needs to meet additional,
> aligner-specific criteria. These may include (and do so for **BWA-MEM**):
>
> - both reads need to map to the same chomosome
> - the two reads need to map to the reference in the orientation expected by
>   the aligner
> - the two read pairs need to map to the reference within an
>   aligner-determined distance
>
> Thus, filtering based on the flag has two consequences:
>
> - filtering will be stricter than with just the *Read is unmapped* and *Mate
>   is unmapped* flags above
> - you will eliminate read pairs that could be informative with regard to
>   chromosomal rearrangements and insertion/deletion events
>
>   => Do not filter for properly paired reads if you plan to detect such
>   structural variants!
>
> In addition, the *proper pair* flag is considered undefined if the read
> itself is unmapped, so a *proper pair* filter should eliminate unmapped reads
> explicitly to be on the safe side.
>
> Thus, if you would like to use proper pair filtering (we have no intention to
> detect structural variants in this tutorial) instead of just filtering for
> mapped reads with a mapped mate, you could run the alternative:
>
> > <hands-on-title></hands-on-title>
> >
> > 1. {% tool [Samtools view](toolshed.g2.bx.psu.edu/repos/iuc/samtools_view/samtools_view/1.15.1+galaxy0) %}:
> >   - {% icon param-files %} *"SAM/BAM/CRAM data set"*: all 3 mapped reads
> >     datasets of the family trio, outputs of **Map with BWA-MEM**
> >     {% icon tool %}
> >   - *"What would you like to look at?"*: `A filtered/subsampled selection of reads`
> >     - In *"Configure filters"*:
> >       - *"Require that these flags are set"*: `Read is mapped in a proper pair`
> >       - *"Exclude reads with any of the following flags set"*: `Read is unmapped`
> >
> {: .hands_on}
>
{: .details}

## Removing duplicate reads

> <hands-on-title>Remove duplicates</hands-on-title>
>
> 1. {% tool [RmDup](toolshed.g2.bx.psu.edu/repos/devteam/samtools_rmdup/samtools_rmdup/2.0.1) %} with the following parameters:
>   - {% icon param-files %} *"BAM file"*: all 3 filtered reads datasets; the
>     outputs of **Samtools view**
>   - *"Is this paired-end or single end data"*: `BAM is paired-end`
>     - *"Treat as single-end"*: `No`
>
{: .hands_on}

Again, this will produce three new datasets, one for each member of the family
trio.

# Variant calling

With the sequenced reads of all samples mapped and postprocessed, we can start
looking for evidence of sequence deviations, *i.e.* variants, between the
sequenced genomic samples and the reference genome.

This task has been automated and optimized continuously over the last decade,
and modern variant calling software hides much of the complexity involved in
it. At least a basic understanding of the underlying concepts is still highly
recommended though and, if you are new to variant calling, the tutorial on
[Calling variants in diploid systems](../dip/tutorial.html) may be a good
starting point for you.

## Generating FreeBayes calls

We will use **FreeBayes** to call our variants. **FreeBayes** is a Bayesian
genetic variant detector designed to find small polymorphisms, specifically
SNPs (single-nucleotide polymorphisms), indels (insertions and deletions), MNPs
(multi-nucleotide polymorphisms), and complex events (composite insertion and
substitution events) smaller than the length of a short-read sequencing
alignment.

> <hands-on-title>Generating FreeBayes calls</hands-on-title>
>
> 1. Run {% tool [FreeBayes](toolshed.g2.bx.psu.edu/repos/devteam/freebayes/freebayes/1.3.6+galaxy0) %}:
>    - *"Choose the source for the reference genome"*: `Locally cached`
>      - *"Run in batch mode?"*: `Merge output VCFs`
>        - {% icon param-files %} *"BAM dataset(s)"*: all three mapped reads
>          datasets of the family trio; the outputs of **RmDup**
>      - *"Using reference genome"*: `Human: hg19` (or a similarly named option)
>
>      > <comment-title>Using the imported `hg19` sequence</comment-title>
>      > If you have imported the `hg19` chr8 sequence as a fasta dataset
>      > into your history instead:
>      > - *"Choose the source for the reference genome"*: `History`
>      >   - *"Run in batch mode?"*: `Merge output VCFs`
>      >     - {% icon param-files %} *"BAM or CRAM dataset(s)"*: all three
>      >        mapped and fully post-processed reads datasets of the family trio;
>      >        outputs of **RmDup**
>      >   - {% icon param-file %} *"Use the following dataset as the
>      >     reference sequence"*: your imported `hg19` fasta dataset.
>      {: .comment}
>
>    - *"Limit variant calling to a set of regions?"*: `Do not limit`
>    - *"Choose parameter selection level"*: `1. Simple diploid calling`
>
{: .hands_on}

{% icon trophy %} **Congratulations!**
You have created you first multisample VCF file, one of the most complicated
file formats in bioinformatics. For every variant detected in at least one of
your samples, this tab-separated format uses a single line to store all
information about the variant. This includes but is not limited to:

- the position of the variant in the genome (with respect to the reference
  genome used for the analysis)
- the nature of the variant (the actual sequence change associated with it)
- the detected genotype of every sample at the variant position
- measures of the reliability of the variant call and of all individual
  genotype calls

> <hands-on-title>Optional hands-on: Inspect the VCF output produced by FreeBayes</hands-on-title>
>
> 1. Display the VCF dataset:
>    - Click the {% icon galaxy-eye %} icon next to the VCF dataset generated
>      by FreeBayes to display its contents.
>
>      VCF is a tabular plain text format though its information density makes
>      it complicated to  understand.
>
>    > <question-title></question-title>
>    >
>    > Can you locate at least some of the above-listed information in the
>    > dataset?
>    >
>    > Hints:
>    > - Lines starting with `##` are comment lines explaining the content of
>    >   the file.
>    > - Diploid genotypes at biallelic sites are encoded using `0/0`, `0/1`
>    >   and `1/1` to represent homozygous reference, heterozygous and
>    >   homozygous variant states, respectively.
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > - The position of each variant with respect to the reference genome is
>    > >   stored in the first two columns (labeled `CHROM` and `POS`).
>    > >
>    > > - The sequence change associated with the variant is provided by
>    > >   columns 4 and 5 (`REF` and `ALT`).
>    > >
>    > >   The first of these lists the sequence found at the variant site in
>    > >   the reference genome, the second the variant sequence.
>    > >
>    > > - The genotypes of the samples are stored as part of columns 10 and
>    > >   up.
>    > >
>    > >   Column 9 (`FORMAT`) provides the legend for understanding the
>    > >   sample-specific columns following it and tells us that the first
>    > >   element (before the first `:` separator) of each sample column holds
>    > >   the sample genotype (`GT`).
>    > >
>    > > - The variant call quality (a measure of the reliability of the
>    > >   variant existing in any of the samples, *i.e.* it being real) is
>    > >   stored in column 6 (`QUAL`).
>    > >   The values are [Phred-scaled](https://en.wikipedia.org/wiki/Phred_quality_score)
>    > >   like sequenced reads base qualities.
>    > >
>    > >   The (log10-scaled) likelihoods of individual genotype calls are
>    > >   stored as the last element (`GL`) in every sample column.
>    > >
>    > > If you are eager to learn all the details about the format, you could
>    > > consult the excellent (but also long) official
>    > > [VCF specification](http://samtools.github.io/hts-specs/VCFv4.3.pdf).
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

## Post-processing FreeBayes calls

Before starting to analyze the detected variants, we need to post-process the
VCF dataset to fix a few incompatibilities between Freebayes and downstream
analysis tools. In particular, we want to:

- Split multiallelic variant records, *i.e.*, records that list more than one
  alternate allele at a given genomic position, into separate record lines.

  This will allow us to annotate each record with information about the impact
  of one specific variant allele further on.

- Make sure that indels are represented in left-aligned and normalized form
  because this is how known indels are stored in public annotation databases.

  > <details-title>Indel normalization</details-title>
  > Indel normalization is a surprisingly complex topic, which is explained
  > really well and in detail in
  > [Tan et al., 2015](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4481842/).
  > The following figure illustrates the various possibilities to represent
  > an indel, only one of which is the *normalized* form:
  > ![different ways to represent the same indel](../../images/indel_normalization.png "The many ways to represent an indel (from: https://genome.sph.umich.edu/wiki/Variant_Normalization").
  >
  {: .details}

A tool that can do this and also ensures that a VCF dataset conforms to
standards in some other, less important respects is **bcftools norm**.

> <hands-on-title>Post-processing FreeBayes calls</hands-on-title>
>
> 1. {% tool [bcftools norm](toolshed.g2.bx.psu.edu/repos/iuc/bcftools_norm/bcftools_norm/1.15.1+galaxy3) %} with the following parameters:
>    - *"VCF/BCF Data"*: the VCF output of **FreeBayes** {% icon tool %}
>    - *"Choose the source for the reference genome"*: `Use a built-in genome`
>      - *"Reference genome"*: `Human: hg19` (or a similarly named option)
>
>      > <comment-title>Using the imported `hg19` sequence</comment-title>
>      > If you have imported the `hg19` chr8 sequence as a fasta dataset
>      > into your history instead:
>      > - *"Choose the source for the reference genome"*:
>      >   `Use a genome from the history`
>      >   - *"Reference genome"*: your imported `hg19` fasta dataset
>      {: .comment}
>    - *"When any REF allele does not match the reference genome base"*:
>      `ignore the problem (-w)`
>    - *"Atomize"*: `No`
>    - *"Left-align and normalize indels?"*: `Yes`
>    - *"Perform deduplication for the folowing types of variant records"*:
>      `do not deduplicate any records`
>
>      Freebayes is not producing any duplicate calls.
>    - *"~multiallelics"*: `split multiallelic sites into biallelic records (-)`
>      - *"split the following variant types"*: `both`
>
>        We want to split both, multiallelic SNP and indel records.
>    - *"output_type"*: `uncompressed VCF`
>
>      We would like to keep the results human-readable. VCF is also what tools
>      like *SnpEff* and *GEMINI* expect as input. Compressed, binary BCF is
>      interesting for space-efficient long-term storage of large lists of
>      variants.
>
{: .hands_on}

You could try to look for the differences between the original and the
normalized VCF dataset, but for convenience **bcftools norm** reports a brief
summary of the actions it performed. Expand the dataset in the history (by
clicking on its name) to see this output listing the total number of variant
lines processed, along with the number of split, realigned and skipped records.

# Variant annotation and reporting

A list of variants detected in a set of samples is a start, but to discover
biologically or clinically relevant information in it is almost impossible
without some additional tools and data. In particular, the
variants in the list need to be:

- **prioritized** with respect to their potential relevance for the biological
  / clinical phenotype that is studied

  Even with exome sequencing, only a fraction of the detected variants will
  have a clear impact on the function of a protein (many variants will
  introduce silent mutations, or reside in intronic regions still covered by
  the exome-enriched sequencing data). Of these, many will have been observed
  before in healthy individuals arguing against them playing an important role
  in an adverse phenotype.

- **filtered** based on the inheritance pattern expected for a causative
  variant

  A multisample VCF file records the most likely genotypes of all samples at
  every variant site. Knowing which individuals (samples) are affected by a
  phenotype we can exclude variants with inheritance patterns that are
  incompatible with the observed inheritance of the phenotype.

- **reported** in a more human-friendly form

  While the VCF format can be used to encode all relevant information about any
  variant, it is hard for humans to parse that information.

  Ideally, one would like to generate simpler reports for any set of filtered
  and prioritized variants.


## Get data

Our workhorse for annotating and reporting variants and the genes affected by
them will be the **GEMINI** framework. GEMINI comes bundled with a wealth of
annotation data for human variants from many different sources. These can be
used to annotate any list of human variants conveniently, without the need for
separate downloads and conversion between different annotation data formats.

> <details-title>Custom annotations and GEMINI</details-title>
> Beyond its bundled annotation data, GEMINI also provides (limited) support
> for using custom annotations.
> The [Somatic variant calling tutorial](../somatic-variants/tutorial.html)
> demonstrates the use of **GEMINI annotate** {% icon tool %} for this purpose.
{: .details}

The only additional annotation tool we need, for the purpose of this
tutorial, is the tool **SnpEff**, which can annotate variants with their
calculated effects on known genomic features. Because SnpEff is a generic tool
that can be used on variants found in the genome of any organism we need to
provide it with a so-called **SnpEff genome file** that holds the annotated
features (genes, transcripts, translated regions, *etc.*) for our genome of
interest.

While annotated variants are all we need to *prioritize* them as described
above, *filtering based on inheritance patterns* requires a way to inform
GEMINI about the relationship between our samples and their observed
phenotypes. This is done through a so-called **pedigree file** in PED format,
which is rather simple to generate manually.

> <hands-on-title>Obtain SnpEff genome and GEMINI pedigree files</hands-on-title>
> 1. Download **SnpEff functional genomic annotations**
>
>     > <comment-title>Shortcut</comment-title>
>     > You can skip this step if the Galaxy server you are working on offers
>     > `Homo sapiens: hg19` as a locally installed snpEff database. You can
>     > check the **Genome source** select list of the
>     > {% tool [SnpEff eff](toolshed.g2.bx.psu.edu/repos/iuc/snpeff/snpEff/4.3+T.galaxy2) %}
>     > tool to see if this is the case.
>     {: .comment}
>
>     Use {% tool [SnpEff Download](toolshed.g2.bx.psu.edu/repos/iuc/snpeff/snpEff_download/4.3+T.galaxy2) %} to download genome annotation
>     database `hg19`.
> 2. Create a PED-formatted pedigree dataset describing our single-family sample trio:
>
>    ```
>    #family_id    name     paternal_id    maternal_id    sex    phenotype
>    FAM           father   0              0              1      1
>    FAM           mother   0              0              2      1
>    FAM           proband  father         mother         1      2
>    ```
>
>    and set its datatype to `tabular`.
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md format="tabular" %}
>
>    > <warning-title>Remember those sample names</warning-title>
>    >
>    > The above content of the pedigree dataset assumes you chose `father`,
>    > `mother`, `proband` as the sample names at the read mapping step
>    > (if you haven't mapped the reads yourself, but started with the
>    > premapped data, you can safely skip this warning section).
>    > By now, these sample names will have been propagated through *BWA-MEM*
>    > and *Freebayes* to the VCF dataset of variants. It is **important** that
>    > you use matching sample names in the pedigree and in the VCF dataset, or
>    > GEMINI will not be able to connect the information in them.
>    >
>    > If you have chosen different sample names before, you have to adjust
>    > the pedigree dataset accordingly!
>    >
>    {: .warning}
>
>    > <details-title>More on PED files</details-title>
>    >
>    > The PED format is explained in the help section of
>    > {% tool [GEMINI load](toolshed.g2.bx.psu.edu/repos/iuc/gemini_load/gemini_load/0.20.1+galaxy2) %}.
>    >
>    > Take a moment and try to understand the information that is encoded in
>    > the PED dataset we are using here.
>    {: .details}
{: .hands_on}

## Variant annotation with functional genomic effects

We need to start annotating our variants with SnpEff simply because Gemini
knows how to parse SnpEff-annotated VCFs, while GEMINI output cannot be used
with SnpEff.

> <hands-on-title>Adding annotations with SnpEff</hands-on-title>
> 1. {% tool [SnpEff eff](toolshed.g2.bx.psu.edu/repos/iuc/snpeff/snpEff/4.3+T.galaxy2) %}
>    - {% icon param-file %} *"Sequence changes (SNPs, MNPs, InDels)"*: the
>      output of **bcftools norm** {% icon tool %}
>    - *"Input format"*: `VCF`
>    - *"Output format"*: `VCF (only if input is VCF)`
>    - *"Genome source"*: `Locally installed reference genome`
>       - *"Genome"*: `Homo sapiens: hg19` (or a similarly named option)
>
>      > <comment-title>Using the imported `hg19` SnpEff genome database</comment-title>
>      > If you have imported the `hg19` SnpEff genome database into your
>      > history instead:
>      >   - *"Genome source"*: `Downloaded snpEff database in your history`
>      >      - {% icon param-file %} *"SnpEff4.3 Genome Data"*: your imported `hg19` SnpEff dataset.
>      {: .comment}
>
>    - *"Produce Summary Stats"*: `Yes`
>
{: .hands_on}

Running the above job will produce two datasets. One is a *Summary Stats* HTML
report, which contains some interesting general metrics such as a distribution
of variants across gene features. The other one is the main annotation result -
a VCF like the input, but with annotations of variant effects added to the INFO
column.

> <hands-on-title>Optional hands-on: Inspect the Summary Stats output produced by SnpEff</hands-on-title>
>
> 1. Display the dataset:
>    - Click the {% icon galaxy-eye %} icon next to the HTML dataset generated
>      by SnpEff to display its contents.
>
>    > <question-title></question-title>
>    >
>    > One section in the report is **Number of effects by type and region**.
>    > Given that you are analyzing exome data, what is the most surprising
>    > aspect in this section? Do you have an idea how to explain it?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > According to the report, intronic variants make up 50% of all
>    > > variants detected!
>    > >
>    > > Exome capture kits are designed to capture exons plus a bit of
>    > > surrounding sequence to ensure proper coverage of the exon ends
>    > > including splice junction sites.
>    > >
>    > > Thus, even though intronic sequences are underrepresented in exome
>    > > sequencing data, not all of them are eliminated. Since mutations
>    > > in most intron bases are neutral, they can accumulate at higher
>    > > frequency than most mutations in exons and, thus, still represent a
>    > > relevant fraction of all detected variants.
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

## Generating a GEMINI database of variants for further annotation and efficient variant queries

Next, we are going to use the SnpEff-annotated VCF as the basis for more
exhaustive annotation with GEMINI. Unlike SnpEff, GEMINI does not just add
annotations to a list of variants in VCF format. Instead the framework
extracts the variants from the VCF input and stores them, together with newly
added annotations, in an SQL database. It then lets you formulate queries for
retrieving and reporting subsets of variants. The combined variant
extraction/annotation/storage step is performed by the **GEMINI load** tool. In
addition, that same tool can be used to incorporate sample pedigree info into
the database.

> <hands-on-title>Creating a GEMINI database from a variants dataset</hands-on-title>
>
> 1. {% tool [GEMINI load](toolshed.g2.bx.psu.edu/repos/iuc/gemini_load/gemini_load/0.20.1+galaxy2) %} with
>    - {% icon param-file %} *"VCF dataset to be loaded in the GEMINI database"*:
>      the output of **SnpEff eff** {% icon tool %}
>    - *"The variants in this input are"*: `annotated with snpEff`
>    - *"This input comes with genotype calls for its samples"*: `Yes`
>
>      Sample genotypes were called by Freebayes for us.
>
>    - *"Choose a gemini annotation source"*: select the latest available annotations snapshot (most likely, there will be only one)
>    - *"Sample and family information in PED format"*: the pedigree file
>      prepared above
>    - *"Load the following optional content into the database"*
>      - {% icon param-check %} *"GERP scores"*
>      - {% icon param-check %} *"CADD scores"*
>      - {% icon param-check %} *"Gene tables"*
>      - {% icon param-check %} *"Sample genotypes"*
>      - {% icon param-check %} *"variant INFO field"*
>
>      Leave **unchecked** the following:
>      - *"Genotype likelihoods (sample PLs)"*
>
>         Freebayes does not generate these values
>
>      - *"only variants that passed all filters"*
>
>        This setting is irrelevant for our input because Freebayes did not
>        apply any variant filters.
{: .hands_on}

Running this job generates a GEMINI-specific database dataset, which can only
be processed with other GEMINI tools. The benefit, however, is that we now have
variants, rich annotations and pedigree info stored in a format that enables
flexible and highly efficient queries, which will greatly simplify our actual
task to identify the variant responsible for the child's disease!

> <details-title>The GEMINI suite of tools</details-title>
>
> For a beginner, the sheer number of GEMINI tools may be a bit daunting and
> give the impression that this framework adds a lot of complexity.
> In practice, however, you will likely only need a very limited number of
> these tools for any given analysis. To help you get an overview, here is a
> list of the most general-purpose tools and their function:
>
> - Every analysis will always start with **GEMINI load** {% icon tool %} to
>   build the GEMINI database from a variant list in VCF format and the
>   default annotations bundled with GEMINI.
> - **GEMINI amend** {% icon tool %} and **GEMINI annotate** {% icon tool %}
>   let you add additional information to a GEMINI database after it has been
>   created. Use **GEMINI amend** to overwrite the pedigree info in a database
>   with the one found in a given PED input, and **GEMINI annotate** to add
>   custom annotations based on a VCF or BED dataset.
> - **GEMINI database info** {% icon tool %} lets you peek into the content of
>   a GEMINI database to explore, *e.g*, what annotations are stored in it.
> - **GEMINI query** {% icon tool %} and **GEMINI gene-wise** {% icon tool %}
>   are the two generic and most flexible tools for querying, filtering and
>   reporting variants in a database. There is hardly anything you cannot do
>   with these tools, but you need to know or learn at least some SQL-like
>   syntax to use them.
> - **GEMINI inheritance pattern** {% icon tool %} simplifies variant queries
>   based on their inheritance pattern. While you could use the *query* and
>   *gene-wise* tools for the same purpose, this tool allows you to perform a
>   range of standard queries without typing any SQL. **This tool is what we
>   are going to use here to identify the causative variant.**
> - The remaining tools serve more specialized purposes, which are beyond the
>   scope of this tutorial.
>
> The [Somatic variant calling tutorial](../somatic-variants/tutorial.html)
> demonstrates the use of the **GEMINI annotate** and **GEMINI query** tools,
> and tries to introduce some essential bits of GEMINI's SQL-like syntax.
>
> For a thorough explanation of all tools and functionality you should consult
> the [GEMINI documentation](https://gemini.readthedocs.io).
{: .details}

## Candidate variant detection

Let us now try to identify variants that have the potential to explain the
boy child's osteopetrosis phenotype. Remember that the parents are
consanguineous, but both of them do not suffer from the disease. This
information allows us to make some assumptions about


topic: teaching
tutorial: motivation_demotivation
title: Motivation and Demotivation
difficulty: intermediate
time_estimation: 60m
tools_json: []
requirements_json: []
tags_json: ["elixir", "train-the-trainers"]
content_hash: 785a99e639e8036e2504b29d84cefbbd
last_modified: 2026-02-26T21:12:18.342314
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/teaching/tutorials/motivation_demotivation/tutorial.html

content:
In order to facilitate learning, we had to learn (a bit) how learning works. But motivation is key to learning. If we want to enhance motivation, we have to learn a bit how motivation works.
Motivation is personal and it is very difficult to work with demotivated learners. Often, teachers cannot do much with deep lack of motivation. Here, we want to see what can trainers do to enhance motivation.
Adult learners usually come motivated to courses and the point is to not demotivate them.

> <comment-title>Resources / Reading recommendation</comment-title>
>
> This tutorial is significantly based on the 3rd session of ELIXIR Train the Trainer curriculum, but also:
> - [Teaching Tech Together {% cite wilson2019teaching %}](https://teachtogether.tech)
> - [How Learning works, {% cite ambrose2010learning %}](https://www.academia.edu/6743627/How_Learning_Works_Seven_Research_Based_Principles_for_Smart_Teaching_Susan_A_Ambrose_Michael_W_Bridges_Michele_DiPietro_Marsha_C_Lovett_Marie_K_Norman_with_a_Foreword_by_Richard_E_Mayer)
> - Understanding how we learn {% cite weinstein2018understanding %}
> - [Instructor Training, The Carpentries](https://carpentries.github.io/instructor-training/08-motivation/index.html)
>
> To get ready to teach this tutorial
> - Read the chapter "What factors motivate Students to learn?" in [How Learning works, {% cite ambrose2010learning %}](https://www.academia.edu/6743627/How_Learning_Works_Seven_Research_Based_Principles_for_Smart_Teaching_Susan_A_Ambrose_Michael_W_Bridges_Michele_DiPietro_Marsha_C_Lovett_Marie_K_Norman_with_a_Foreword_by_Richard_E_Mayer)
> - Check ["Motivation and Demotivation" section](https://carpentries.github.io/instructor-training/08-motivation/index.html), Instructor Training, The Carpentries
>
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Motivation

What is motivation?
- Motivation refers to the **personal investment** that an individual has in reaching a desired state or outcome ({% cite maehr1997understanding %})
- **Learning** results from what the **student does and thinks** and **only** from what the student does and thinks. **The teacher** can advance learning only by influencing what the student **does** to learn (Herbert A. Simon - Nobel Laureate, one of the founders of Cognitive Science)

    This means that the responsibility of the learning is on the learners. If the learners are not motivated, they will not learn (or hardly learn anything).

- **Principle P3**: Student's motivation determines, directs and sustains what they do learn ({% cite ambrose2010learning %})

Motivation influences **direction**, **intensity**, **persistence** and **quality** of the learning behaviors.


> <hands-on-title>Recall a motivating learning experience - {% icon time %} 5 min - Silent reflection</hands-on-title>
>
> Write in the shared notes about a **motivating** experience in your life (as a learner) and how it impacted you
>
>
{: .hands_on}

<!--{% include _includes/quiz.html id="motivating_learning_experience.yaml" %}-->

> <tip-title>For trainer</tip-title>
>
> The trainer will comment learners' motivating experiences, trying to identify and highlight "patterns of motivation". Key features of motivation (value, expectancy, environment) could be introduced starting from learners' experiences.
>
> Example:
> - Learner's experience: "I felt very much motivated by a teacher saying that, based on my performance during classes, I would have been perfectly able to achieve all the course learning outcomes.""
> - Trainer takes the opportunity to introduce the concept of "expectancy""' as one of the three pillars of motivation.
>
{: .tip }

## Goals

To say that someone is motivated tells us little unless we say what the person is motivated to do. Thus, **goals** serve as the basic organising feature of motivated behaviour. They act as the compass that guides and directs a broad range of purposeful actions.

![2 main boxes: a grey one with "Goals" written in it, a green one with "Motivation" in it. 5 smaller boxes stacked on each other on the left of the "Goals" box with "Performance goals", "Social goals", "Learning goals", "Work-avoidant goals", "Affective goals" written inside. "Basic features of" is written between "Goals" and "Motivation"](images/motivation_concept_map_l2.png)

Researcher have identified different categories of goals whose the type influence behavior
- **Performance goal**:
    - protecting a desired self-image
    - projecting a positive reputation and public persona
    - passing exams
    - appear intelligent
    - acquire recognition and praise.
- **Learning goal**: gain competence and truly learn what an activity or task can teach them (adult learners)
- **Work-avoidant goals**: desire to finish work as quickly as possible with as little effort as possible
- **Social goals**: making friends
- **Affective goals**: engaging in stimulating activities, e.g. have fun

> <details-title>Performance goals vs Learning goals</details-title>
>
> When guided by performance goals, students concerned with normative standards try to do what is necessary to demonstrate competence to
> 1. appear intelligent
> 2. gain status
> 3. Acquire recognition and praise
>
> When guided by learning goals, students try to
> - Gain competence
> - Truly learn
>
{: .details}

Learners' goals for themselves may **differ** from teachers' goals for them. Powerful learning situation happen when goals of of learners and teachers are aligned, but also when more than one goal satisfied.

## Value

What is also playing an important role in motivation? The importance of the goals, the **subjective value** of the goals, i.e. the value a learner attributes to the goals. A lack of of perceived value leads to a lack of motivation-

![3 main boxes: a grey one with "Goals" written in it, a green one with "Motivation" in it,, an blue one with "Value. A goal's importance" written in it. 5 smaller boxes stacked on each other on the left of the "Goals" box with "Performance goals", "Social goals", "Learning goals", "Work-avoidant goals", "Affective goals" written inside. "Basic features of" written between "Goals" and "Motivation". 1 arrow from "Value" to "Motivation"](./images/motivation_concept_map_l3.png)

There are 3 main sources of values
- **Attainment value**: satisfaction that one gains from mastery and accomplishment of a goal

    Example: satisfaction from solving complex problems to demonstrate the ability to solve them

- **Intrinsic value**: satisfaction that one gains simply from doing the task; source of intrinsic motivation

    Example: spend hours writing a computer program

- **Instrumental value**: degree to which an activity helps one accomplish other important goals = extrinsic rewards

    Examples:
    - Praise
    - Public recognition
    - An interesting career
    - A good salary

The difference sources of value are not necessarily conflicting, but potentially reinforcing.

For example, by working hard in a course, a biology student may derive value from multiple sources
- Solving challenging problems (attainment value)
- Engaging their fascination with biological processes (intrinsic value)
- Advancing their chances of getting into good medical school (instrumental value)

## Expectancies

Although one must value a desired outcome in order to be motivated to pursue it, value alone is insufficient to motivate behavior. People are also motivated to pursue goals and outcomes that they believe they can successfully achieve.

Conversely, if they do not expect to successfully achieve a desired goal or outcome, they will not be motivated to engage in the behaviors necessary to achieve it. Motivational theorists refer to these expectations as **expectancies**.

**Expectancies**, or expectations for successful attainment of that goal, are the goals and outcomes that people believe they can achieve. It has to do with the feeling of achieving a goal. When you have goals, value and expectancy, you can achieve motivation:

![4 main boxes: a grey one with "Goals" written in it, a green one with "Motivation" in it, an blue one with "Value. A goal's importance" written in it, an orange one with "Expectancy / Efficacy" in it. 5 smaller boxes stacked on each other on the left of the "Goals" box with "Performance goals", "Social goals", "Learning goals", "Work-avoidant goals", "Affective goals" written inside. "Basic features of" written between "Goals" and "Motivation". 2 arrows from "Value" and "Expectancy" to "Motivation"](./images/motivation_concept_map_l4.png)

We describe two forms of expectancies that help inform our understanding of motivated behavior:
- **Outcome expectancies**: the belief that specific actions will bring about a desired outcome ({% cite carver1998self %})

    Holding positive outcome expectancies leads to motivation

    For example, "If I do all the assigned readings and participate in class discussion, I will be able to learn the material well enough to solve problems in the exam and achieve a passing grade"

- **Efficacy expectancies**: the belief that one is capable of identifying, organizing, initiating, and executing a course of action that will bring about a desired outcome ({% cite bandura1997self %})

    It is the belief in personal agency. In order to hold a positive expectancy for success, learners must not only believe that doing the assigned work can earn a passing grade, they must also believe that they are capable of doing the work necessary to earn a passing grade

Learners' expectation for success is determined by prior experience in similar context. For example, the thought: "I am not good at math"

Highest motivation, effort and persistence of it happen among students who attribute successful performance to a combination of ability and effort.


With goals, value and expectancy, motivation leads to a **goal directed behaviour** which is key to support **learning and performance**.

![6 main boxes: a grey one with "Goals" written in it, a green one with "Motivation" in it, an blue one with "Value. A goal's importance" written in it, an orange one with "Expectancy / Efficacy" in it, a yellow one with "Goal-directed behavior" in it, a light blue with "Learning and Performance" written. 5 smaller boxes stacked on each other on the left of the "Goals" box with "Performance goals", "Social goals", "Learning goals", "Work-avoidant goals", "Affective goals" written inside. "Basic features of" written between "Goals" and "Motivation". 2 arrows from "Value" and "Expectancy" to "Motivation". 1 arrow  from "Motivation" to "Goal-directed behavior" with "Leads to" written below. 1 arrow from "Goal-directed behavior" to "Learning and Performance" with "Supports" written below](images/motivation_concept_map_l6.png)

Value and Expectancy are then two key pillars of motivation but that they're are not sufficient to be fully motivated to learn.

## Learning environment

There is a third pillar of motivation, which is the learning context or environment.

![7 main boxes: a grey one with "Goals" written in it, a green one with "Motivation" in it, an blue one with "Value. A goal's importance" written in it, an orange one with "Expectancy / Efficacy" in it, a yellow one with "Goal-directed behavior" in it, a light blue with "Learning and Performance" written, a lila one with "Environment" written. 5 smaller boxes stacked on each other on the left of the "Goals" box with "Performance goals", "Social goals", "Learning goals", "Work-avoidant goals", "Affective goals" written inside. "Basic features of" written between "Goals" and "Motivation". 3 arrows from "Value", "Expectancy" and "Environment" to "Motivation". 1 arrow  from "Motivation" to "Goal-directed behavior" with "Leads to" written below. 1 arrow from "Goal-directed behavior" to "Learning and Performance" with "Supports" written below](images/motivation_concept_map_l7.png)

**Principle 6**: Students' current level of development interacts with the social, emotional, and intellectual climate of the course to impact learning ({% cite ambrose2010learning %})

Perception of the environment along a continuum from supportive to unsupportive has impact on motivation.

What do we mean about this? To better understand, we should consider the following **aspects for an unbiased and fair environment**:

- discrimination issues and inclusivity
- stereotypes

    > <details-title>Examples of stereotypes</details-title>
    >
    > Providing extra or less support to groups of students based on the belief that, in certain disciplines, some groups are not very gifted whereas others are innately talented (e.g. women are not strong in the STEMs or Asians are good at math) represents a type of bias very difficult to detect and it is a subtle form of racial stereotype (see e.g., https://theconversation.com/asians-are-good-at-math-why-dressing-up-racism-as-a-compliment-just-doesnt-add-up-128731, https://www.hepg.org/her-home/issues/harvard-educational-review-volume-89,-issue-4/herarticle/asians-are-good-at-math-is-not-a-compliment)
    >
    {: .details}

- complex dynamics of a class
- accessibility (including venue, materials and teacher accessibility)
- mindset (growth vs fixed mindset)

    > <details-title>About mindset (from Dweck 2006)</details-title>
    >
    > Someone with a **growth mindset** views intelligence, abilities, and talents as learnable and capable of improvement through effort. On the other hand, someone with a **fixed mindset** views those same traits as inherently stable and unchangeable over time.
    >
    > Mindset is an outgrowth of the age-old nature vs. nurture debate (that is, how much of our ability and personality are hardwired into us, and how much is the result of how we're raised?). Recent studies suggest nurture is more important than nature—that growing up in a safe, secure, and inspirational environment outweighs innate abilities and behaviors. {% cite dweck2006mindset %} agrees with this assessment: The crux of her argument in Mindset is that we can continue to improve throughout our lives by nurturing our growth, and that doing so far outweighs whatever natural talents we do or don't possess.
    >
    > We can just recommend you to have also a look at the [Book summary of  {% cite dweck2006mindset %}](https://www.shortform.com/summary/mindset-the-new-psychology-of-success-summary-carol-dweck?gclid=CjwKCAjwgr6TBhAGEiwA3aVuIfDQDR77RVaUfjyQVrmQQ5ELPCl5fjLlbnfSZ2RPbdwoObh6POrJzhoCkZkQAvD_BwE)
    >
    {: .details}

All these aspects will influence the climate of a classroom. For example, "The instructor is approachable and several of my classmates seem willing to help me if I run into troubles"

In order to have fully MOTIVATED learners, it is important that:
- Learners **see the value**
- Learner self-efficacy is **high** and
- The environment is **supportive**

![3 level matrix ](images/environment_perception.png "Interactive effects of environment, efficacy, and value on motivation. Source: {% cite ambrose2010learning %}")

> <details-title>About Figure "Interactive effects of environment, efficacy, and value on motivation" (from Ambrose 2010)</details-title>
> From {% cite ambrose2010learning %}:
> 1. If no value and no expectation, leaners will reject learning with disengagement and even anger
> 2. If high expectancies but no valuem: the task is doable but unimportant, learners will evade and have difficulty to pay attention
> 3. If value but low expectancyies, they lack of confidence in their ability.
>    - If the environment is NOT supportive, they are hopeless with no expectation of success
>    - If the environment is supportive, they are fragile; they want to succeed, try to protect their sense of self-esteem and making excuse
> 4. If value and high expectation
>    - If the environment is NOT supportive, they will be defiant: "I will show you"
>    - If the environment is supportive, they are motivated
>
{: .details}

# Demotivation

Motivation can go both ways: motivation and de-motivation are strictly connected.
Adult learners usually come motivated to courses and the point is to not demotivate them.
In many cases, to not de-motivate learners is an excellent way to support motivation.


> <hands-on-title>Recall a demotivating experience - {% icon time %} 5 min - Silent reflection</hands-on-title>
>
> Write in the shared notes about a **demotivating** experience in your life (as a learner) and how it impacted you
>
{: .hands_on}

<!--{% include _includes/quiz.html id="demotivating_learning_experience.yaml" %}-->


## Things you should not do as instructors

- Tell learners they are rubbish because they use a technology, etc
- Say negative things about a technology or applications
- Pretend to know more than what you do
- Deliver long unidirectional lectures
- Dive into complex or detailed technical discussions with one or two people
- Use diminishing language "just", "simply", "obviously", "don’t you know?"

    > <details-title>About dismissive language</details-title>
    >
    > Unaware usage of dismissive language (like "just", "easy", etc.) or of apparently "innocent" terms from the point of view of the "white heterosexual male" may be demotivating and / or offensive to certain minority groups (non white and / or non heterosexual and / or non male).
    >
    {: .details}

- Hinder autonomy

    > <details-title>Why taking over the learner's keyboard hinder autonomy?</details-title>
    >
    > It is rarely a good idea to type anything for your learners
    >
    > Doing so can be demotivating for the learner (as it implies you do not think they can do it themselves or that you do not want to wait for them)
    >
    > It also wastes a valuable opportunity for your learner to develop muscle memory and other skills that are essential for independent work.
    >
    {: .details}

- Feign surprise with "I cannot believe you do not know X" or "You have never heard of Y?"

    > <details-title>Why may feigning surprise trigger demotivation?</details-title>
    >
    > Saying things like "I cannot believe you do not know X" or "You have never heard of Y?" signals to the learner that they do not have some required pre-knowledge of the material you are teaching, that they do not belong at the workshop, and it may prevent them from asking questions in the future.
    >
    {: .details}

It can be difficult to avoid these demotivators entirely. Teaching yourself to avoid these types of comments takes practice, but is well worth the effort. No one likes to be made fun of.

> <details-title>Systemic and psychological demotivators</details-title>
>
> Other factors can contribute to demotivation. Some of which are either **systemic**, or **built into our psychological makeup** as human beings. We cannot always control these demotivators - often they come from outside the classroom - but we can be aware of them, and take certain actions to lessen their impact by thinking carefully about the language that we use and how we interact with our learners.
>
> - **Stereotype threat**: Reminding people of negative stereotypes, even in subtle ways, can make them anxious about the risk of confirming those stereotypes, in turn reducing their performance. Clearest examples in computing are gender-related
>
>   > <tip-title>What can you as instructor?</tip-title>
>   > - Avoid thinking in terms of a deficit model, use a systems approach
>   > - Not highlight people based on their identity with a minority group: draws attention to the stereotype
>   > - Ask people to sign up for workshops in small teams rather than as individuals when possible
>   {: .tip }
>
> - **Impostor syndrome**: Belief that one is not good enough for a job or position, and that one's achievements are due to luck rather than talent or skill ([Resources for teaching about and dealing with imposter syndrome](https://adainitiative.org/continue-our-work/impostor-syndrome-training/))
>
>   > <tip-title>What can you as instructor?</tip-title>
>   > - Sharing stories of mistakes that you have made or things you struggled to learn
>   > - Emphasize that you want questions
>   > - Remember, it is much more important to _be_ smart than to _look_ smart
>   {: .tip }
>
> - **Accessibility issue**: It helps everyone to provide proper captioning of images, for example, benefits people with no or limited vision by giving screen readers something to say, but it also makes the images more findable by exposing their content to search engines.
>
>   > <tip-title>What can you as instructor?</tip-title>
>   > - Know what you need to do
>   >   - [Posters of dos and do nots from UK home office](https://hodigital.blog.gov.uk/category/accessibility/)
>   >   - [W3C Accessibility Initiative’s checklist for presentations](http://www.w3.org/WAI/teach-advocate/accessible-presentations/)
>   >   - Liz Henry’s blog post about[ accessibility at conferences](https://modelviewculture.com/pieces/unlocking-the-invisible-elevator-accessibility-at-tech-conferences)
>   >   - [Interview with Chad Taylor](https://modelviewculture.com/pieces/qa-making-tech-events-accessible-to-the-deaf-community)
>   > - Know how well you are doing using sites like[ WebAIM](http://webaim.org/) to check how accessible your online materials
>   > - Involve people with disabilities in decision-making
>   {: .tip }
>
> - **Inclusivity**: Policy of including people who might otherwise be excluded or marginalized
>
>   > <tip-title>What can you as instructor?</tip-title>
>   > - Read Lee's paper ["What can I do today to create a more inclusive community in CS?"](https://docs.google.com/document/d/1hLivou9-_wmsZuzKI2pCGQu0KHVIgYfJSaYhvTgO0Wo/edit?usp=sharing)
>   > - Ask learners to email you before the workshop to explain how they believe the training could help them achieve their goals
>   > - Review notes to make sure they are free from gendered pronouns, that they include culturally diverse names, etc.;
>   > - Emphasize that what matters is the rate at which they are learning, not the level of knowledge they had when they started;
>   > - Encouraging pair programming
>   > - Mitigate actively behavior that some learners may find intimidating, e.g., use of jargon or "questions" that are actually asked to display knowledge.
>   {: .tip }
>
{: .details }

# Strategies to improve motivation and avoid demotivation

Motivation is personal and it is very difficult to work with demotivated learners. Often, teachers cannot do much with deep lack of motivation.
But this is not always true: **there are things a trainer can do to support learners' motivation and to NOT demotivate them**. For example, using stereotypes will demotivate minority-status individuals.

## What can we do to increase learners' motivation?

We can act to establish or reinforce the value of goals, we can help build positive expectancies and we can definitely create an environment that supports motivation.

{% cite ambrose2010learning %} provides a number of strategies that may help

1. **Establish value**: Increase the value that students place on the goals and activities identified and created for students

    > <details-title>Strategies to Establish value</details-title>
    >
    > 1. **Connect the material to learners' interests**: More motivated with material that interests them or has relevance for important aspects of their lives
    >    1. **Show relevance to learners' current academic/working lives**: When presenting knowledge and skills, tell students which ones will be particularly useful for taking other courses fruitfully, make explicit connection between content of different courses for a better understanding the value of each course as building blocks
    >    2. **Demonstrate the relevance of higher-level skills to learners' future     professional lives**: When presenting knowledge and skills, tell students which ones will be particularly useful for their professional lives by explaining how various skills can serve more broadly. Example: writing in lab reports for developing written communication skills
    >
    > 2. **Provide examples and tasks from real situations / problems**: Use examples, datasets, problems, contexts taken from real situations as much as possible. Best: tasks for your own work that could be useful to others
    >
    > 3. **Identify and reward what you value**: What are your learning goals? What values do you associate with achieving those goals? Is your goal to acquire new skills and you value them as they allow you to participate in interesting projects? Share it with the students (in the syllabus, through feedback, through modeling), talk with them about what you value (in learning) and reward through assessments
    >
    > 4. **Show your own passion and enthusiasm for the discipline**: it is powerful and contagious, and raise curiosity
    >
    > 5. **Ensure alignment of objectives, assessments, and instructional strategies**: Choose learning activities allowing the achievement of learning outcomes in a targeted way and design exams assessing whether students have achieved LOs.
    >
    {: .details}

    > <tip-title>The Carpentries approach: Teach most useful first</tip-title>
    >
    > Learners do something that _they_ think is useful in their daily work within 15 minutes of starting each lesson.
    >
    > Not only motivates them, it also helps build their confidence in trainers, so that if it takes longer to get to something they find useful in a later topic, they will persist with the lesson
    {: .tip}

2. **Build positive expectancies**: Strengthen students' expectancies

    > <details-title>Strategies to Build positive expectancies</details-title>
    >
    > 1. **Identify an appropriate level of challenge**: Propose activities that are not too simple or too complex for the level of the students, but always commensurate with the level reached. It may be difficult: you need to know who are learners, for example using pre-assessment to evaluate prior knowledge and future goals
    >
    > 2. **Promote growth mindset**: According to psychologist Carol Dweck, people with growth mindsets are able to embrace challenges and view failures as a stepping stone to their own personal growth, rather than proof of inadequacy. Those who adopt this mentality do not care so much about being intelligent, but rather believe that their talents can be developed through hard work, commitment and feedback. For this reason, failure is perceived as a possibility of future success. On the contrary, considering one's abilities as static, immutable and talent as something that one has or does not possess, is the demonstration of a static mindset, or fixed-mindset.
    >
    > 3. **Provide early success opportunities**: At the beginning of the course or when starting a new topic, create activities or tasks in which students can immediately see the usefulness of what you are teaching and can carry out the activities and complete tasks with "success" and in any case thanks to a job well done rather than extraordinary insights. Expectation for future performance influence by past experiences: early success gives a sense of efficacy.
    >
    > 4. **Articulate your expectations**: Use the Syllabus and first day of class to establish the course climate. From the beginning of the course, discuss the syllabus in detail and in particular what you expect students will be able to do if they achieve the learning outcomes and how they will be assessed. Clear goals to know the desired outcome and clear expectations in order to reach them help make the connection between a course of action and a desired outcome more concrete and tangible. It leads to more positive outcome expectancy. You can help setting realistic expectations by identifying areas with potential difficult and letting know what support expecting from you
    >
    > 5. **Provide frequent and targeted formative feedback**: Use any possible occasion to provide feedback to learners and collect feedback from learners. It has a powerful motivating effect. It is more effective when it is:
    >    - Timely: close enough in proximity to the performance
    >    - Constructive: strengths, weakness, suggestions for future actions
    >
    {: .details}

3. **Create an environment that supports motivation**

    > <details-title>Strategies to Create an environment that supports motivation</details-title>
    >
    > 1. **Be fair**: And find your way to show learners you are fair, to make them feel they are in a fair environment.
    >
    > 2. **Describe effective study strategies**: Recommend study strategies to students. For example, you can think about talking to them about the 6 effective learning strategies, explaining to them what the research says about what works in the learning process and what doesn't. It helps to adjust expectations about being able to successfully obtain their goals
    >
    > 3. **Give learners an opportunity to reflect**: Create moments, tasks, activities and assignments where students are encouraged to reflect, with specific questions. It helps to structure the process to support motivation
    >
    > 4. **Address discrimination issues (Code of Conduct)**: Explicitly discuss your position on discriminatory behaviours and what are the rules in your class.
    >
    > 5. **Avoid the use of stereotypes – genders, ethnicity**: In many cases, the use of stereotypes can be completely unaware. Even devoting more time to students who are thought to be "weaker" by stereotype (for example women in STEM) could be a form of unconscious use of stereotype and therefore even be counterproductive.
    >
    > 6. **Be aware of and remove accessibility limitations in your material, teaching set-up, environment**: In the case of accessibility to the teaching material, in addition to making it fully accessible to all students, it would be necessary to take into account those who suffer from dyslexia or who are colour-blind or visually impaired, which is a further level of accessibility. Accessibility to the classroom set-up concerns, for example, the possibility that from the last row one sees and feels as from the first. Accessibility to the classroom does not only concern the disabled. In fact, worrying about accessibility for the disabled sends a message of great inclusiveness to all students. Some even argue that teachers, as an exercise of empathy, should travel from the street to the classroom in a wheelchair to understand all the obstacles that a person who cannot walk would encounter.
    >
    > 6. **Be accessible, supportive and encouraging**: With all the students, but with particular attention to those who appear most disheartened and in difficulty.
    >
    > 7. **Don't leave anyone behind**: Using formative feedback, try to always be aware of what is going on in your classroom and realise if anyone is left behind or is missing out. As much as possible, make sure you take the class forward without leaving anyone behind.
    >
    > 8. **Provide learners with options and the ability to make choices (promote independence)**: Create activities where students can make choices and develop independence. It lends to a sense of control, which can contribute to a learner's expectation of success
    >
    > 9. **Mitigate actively behavior that some learners may find intimidating**: Pay attention to the dynamics of the class, for example inappropriate jokes made by students, or inappropriate comments that could make some other student uncomfortable.
    >
    {: .details}

> <hands-on-title>Strategies to improve motivation - {% icon time %} 15 min - Groups of 2 people</hands-on-title>
>
> - Pick 1 strategy associated with **establishing value**, **building positive expectancies** and/or **creating an environment that supports motivation value and expectancies**
>
>   > <details-title>Strategies</details-title>
>   >
>   > 1. **Establish value**
>   >    1. **Connect the material to learners' interests**: More motivated with material that interests them or has relevance for important aspects of their lives
>   >        1. **Show relevance to learners' current academic/working lives**: When presenting knowledge and skills, tell students which ones will be particularly useful for taking other courses fruitfully, make explicit connection between content of different courses for a better understanding the value of each course as building blocks
>   >        2. **Demonstrate the relevance of higher-level skills to learners' future     professional lives**: When presenting knowledge and skills, tell students which ones will be particularly useful for their professional lives by explaining how various skills can serve more broadly. Example: writing in lab reports for developing written communication skills
>   >
>   >    2. **Provide examples and tasks from real situations / problems**: Use examples, datasets, problems, contexts taken from real situations as much as possible. Best: tasks for your own work that could be useful to others
>   >
>   >    3. **Identify and reward what you value**: What are your learning goals? What values do you associate with achieving those goals? Is your goal to acquire new skills and you value them as they allow you to participate in interesting projects? Share it with the students (in the syllabus, through feedback, through modeling), talk with them about what you value (in learning) and reward through assessments
>   >
>   >    4. **Show your own passion and enthusiasm for the discipline**: it is powerful and contagious, and raise curiosity
>   >
>   >    5. **Ensure alignment of objectives, assessments, and instructional strategies**: Choose learning activities allowing the achievement of learning outcomes in a targeted way and design exams assessing whether students have achieved LOs.
>   >
>   > 2. **Build positive expectancies**
>   >
>   >    1. **Identify an appropriate level of challenge**: Propose activities that are not too simple or too complex for the level of the students, but always commensurate with the level reached. It may be difficult: you need to know who are learners, for example using pre-assessment to evaluate prior knowledge and future goals
>   >
>   >    2. **Promote growth mindset**: According to psychologist Carol Dweck, people with growth mindsets are able to embrace challenges and view failures as a stepping stone to their own personal growth, rather than proof of inadequacy. Those who adopt this mentality do not care so much about being intelligent, but rather believe that their talents can be developed through hard work, commitment and feedback. For this reason, failure is perceived as a possibility of future success. On the contrary, considering one's abilities as static, immutable and talent as something that one has or does not possess, is the demonstration of a static mindset, or fixed-mindset.
>   >
>   >    3. **Provide early success opportunities**: At the beginning of the course or when starting a new topic, create activities or tasks in which students can immediately see the usefulness of what you are teaching and can carry out the activities and complete tasks with "success" and in any case thanks to a job well done rather than extraordinary insights. Expectation for future performance influence by past experiences: early success gives a sense of efficacy.
>   >
>   >    4. **Articulate your expectations**: Use the Syllabus and first day of class to establish the course climate. From the beginning of the course, discuss the syllabus in detail and in particular what you expect students will be able to do if they achieve the learning outcomes and how they will be assessed. Clear goals to know the desired outcome and clear expectations in order to reach them help make the connection between a course of action and a desired outcome more concrete and tangible. It leads to more positive outcome expectancy. You can help setting realistic expectations by identifying areas with potential difficult and letting know what support expecting from you
>   >
>   >    5. **Provide frequent and targeted formative feedback**: Use any possible occasion to provide feedback to learners and collect feedback from learners. It has a powerful motivating effect. It is more effective when it is:
>   >    - Timely: close enough in proximity to the performance
>   >    - Constructive: strengths, weakness, suggestions for future actions
>   >
>   > 3. **Create an environment that supports motivation**
>   >
>   >    1. **Be fair**: And find your way to show learners you are fair, to make them feel they are in a fair environment.
>   >
>   >    2. **Describe effective study strategies**: Recommend study strategies to students. For example, you can think about talking to them about the 6 effective learning strategies, explaining to them what the research says about what works in the learning process and what doesn't. It helps to adjust expectations about being able to successfully obtain their goals
>   >
>   >    3. **Give learners an opportunity to reflect**: Create moments, tasks, activities and assignments where students are encouraged to reflect, with specific questions. It helps to structure the process to support motivation
>   >
>   >    4. **Address discrimination issues (Code of Conduct)**: Explicitly discuss your position on discriminatory behaviours and what are the rules in your class.
>   >
>   >    5. **Avoid the use of stereotypes – genders, ethnicity**: In many cases, the use of stereotypes can be completely unaware. Even devoting more time to students who are thought to be "weaker" by stereotype (for example women in STEM) could be a form of unconscious use of stereotype and therefore even be counterproductive.
>   >
>   >    6. **Be aware of and remove accessibility limitations in your material, teaching set-up, environment**: In the case of accessibility to the teaching material, in addition to making it fully accessible to all students, it would be necessary to take into account those who suffer from dyslexia or who are colour-blind or visually impaired, which is a further level of accessibility. Accessibility to the classroom set-up concerns, for example, the possibility that from the last row one sees and feels as from the first. Accessibility to the classroom does not only concern the disabled. In fact, worrying about accessibility for the disabled sends a message of great inclusiveness to all students. Some even argue that teachers, as an exercise of empathy, should travel from the street to the classroom in a wheelchair to understand all the obstacles that a person who cannot walk would encounter.
>   >
>   >    6. **Be accessible, supportive and encouraging**: With all the students, but with particular attention to those who appear most disheartened and in difficulty.
>   >
>   >    7. **Don't leave anyone behind**: Using formative feedback, try to always be aware of what is going on in your classroom and realise if anyone is left behind or is missing out. As much as possible, make sure you take the class forward without leaving anyone behind.
>   >
>   >    8. **Provide learners with options and the ability to make choices (promote independence)**: Create activities where students can make choices and develop independence. It lends to a sense of control, which can contribute to a learner's expectation of success
>   >
>   >    9. **Mitigate actively behavior that some learners may find intimidating**: Pay attention to the dynamics of the class, for example inappropriate jokes made by students, or inappropriate comments that could make some other student uncomfortable.
>   {: .details}
>
> - Think about a concrete example of what you could do / say in the classroom to implement that strategy.
>
{: .hands_on }


# Conclusion

As intructor, you should remember:
- Make sure that prerequisites for a course are made explicit – but levels of knowledge will spread widely
- Emphasize that what matters is the rate at which they are learning
- Share stories of mistakes that you have made or things you struggled to learn
- Be aware of the impostor syndrome
- Encourage pair programming and group work
- Remember, it is much more important to be smart than to *look* smart


topic: teaching
tutorial: workshop-final-exercise
title: Train-the-Trainer: putting it all together
difficulty: intermediate
time_estimation: 60m
tools_json: []
requirements_json: ["learning-principles", "design", "assessment", "motivation_demotivation"]
tags_json: ["elixir", "train-the-trainers"]
content_hash: 3d3f146961efe9a5f9b86b2e81afc938
last_modified: 2026-02-26T21:12:18.343832
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/teaching/tutorials/workshop-final-exercise/tutorial.html

content:
# Running a demo training

After going through all the lessons around teaching, you have already learned the following components:

1. How learning works
2. How to design effective trainings, based on well-defined Learning Outcomes (LOs) and concept maps
3. How to assess the effectiveness of the learning process
4. How to provide and receive constructive feedback

In this final exercise, you will be putting all those skills together, produce a mini-tutorial (5'), deliver it and receive feedback.

> <hands-on-title> Prepare your training  - *{% icon time %} 15 min*</hands-on-title>
>
> Prepare your 5 minutes training based on the concept map created during the [design session]({% link topics/contributing/tutorials/design/tutorial.md %}) (~10-15 seconds introduction, 2.5 min training, 2 min assessment, and ~30 seconds closing)
> 
{: .hands_on}

> <hands-on-title>Training delivery and feedback  - *{% icon time %} 30 min - Groups of 4*</hands-on-title>
>
> Each person does the 5 minutes delivery of the material they prepared in the previous session, and then everyone (including the person doing the delivery) provides feedback using the following structure:
> - One thing I liked
> - One thing that could be improved
> 
> Keep in mind, the feedback needs to be positive and be specific
> 
{: .hands_on}


topic: teaching
tutorial: teaching-online
title: Teaching online
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 997380f05a2de3cae98f2b800ef39c06
last_modified: 2026-02-26T21:12:18.343693
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/teaching/tutorials/teaching-online/tutorial.html

content:
The different ways of teaching online (synchronous vs asynchronous), recording videos or not, adding captions, mix of synchronous and asynchronous (what do you need to have, type of support, Q&A sessions)


topic: teaching
tutorial: setup-galaxy-for-training
title: Set up a Galaxy for Training
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: []
content_hash: fdd9e885a25f9eb59a0fc225bd1e7af9
last_modified: 2026-02-26T21:12:18.343259
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/teaching/tutorials/setup-galaxy-for-training/tutorial.html

content:
In this tutorial, you will learn how to provision your Galaxy instance to support training modules from the GTN repository.

Tutorials in this repository are all supplemented with files describing the technical requirements to run them. This makes it easy to automate installation of tutorial requirements.
  - `data-library.yaml`: describes the input datasets
  - `data-manager.yaml`: describes the reference data required by tools
  - `workflows` folder: contains one or more workflows with all steps in the tutorial
  - `tours` folder: contains one or more yaml files describing interactive tours

> <comment-title>Requirements</comment-title>
> For more information about how to create these files, please see our module on [specifying the technical requirements for your tutorial]({% link topics/contributing/tutorials/create-new-tutorial-technical/tutorial.md %}).
{: .comment}

For just the list of Ephemeris commands needed for installation, skip to the [Quickstart section](#quickstart) at the end of this tutorial.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare for installation

If you have a Galaxy server already running somewhere and would like to support one or more training modules, [ephemeris](https://ephemeris.readthedocs.io) can be used to easily install all the required tools, reference data, data libraries, tours and workflows.

For the purposes of this training, we will create a Galaxy instance running on `localhost:8080`, but these instructions can be adapted for use with any Galaxy instance by replacing this value with the URL or IP address of your Galaxy server.

## Start a local Galaxy

To setup a Galaxy server locally, we will first clone the Galaxy github repository, make a few small edits to the `galaxy.yaml` configuration file, and then start the server.


> <hands-on-title>Setup a local Galaxy instance</hands-on-title>
>
> 1. Clone the github repository
>    ```bash
>    git clone https://github.com/galaxyproject/galaxy.git
>    cd galaxy
>    ```
>
> 2. Add yourself as admin user in `config/galaxy.yaml`
>    ```bash
>    cp config/galaxy.yml.sample config/galaxy.yml
>    ```
>    open the `galaxy.yml` file with your favorite editor and edit the following line with your email address:
>    ```yaml
>    admin_users: user@example.org
>    ```
>
> 3. Start Galaxy
>    ```bash
>    sh run.sh
>    ```
>    Galaxy will now install all its requirements, which may take a few minutes,
>    when all is finished installing, you should see something like this in your screen:
>
>    ```
>    Starting server in PID 9560.
>    serving on http://localhost:8080
>    ```
>
> 4. Open Galaxy
>     - Open a web browser
>     - Navigate to `localhost:8080` to access Galaxy
>
{: .hands_on}


## Find your Galaxy API key

In order to install the tutorial requirements, we will need the API key of an admin user.

> <hands-on-title>Obtain Galaxy API key</hands-on-title>
>
> 1. Register an account on Galaxy using the email address you added to the `config/galaxy.yml` file
>    - Once logged in, verify that you have a menu item named `Admin` in your top menu bar.
> 2. Go to your `User -> Preferences` in the top menu bar, then click on `Manage API key`
> 3. Click on `Create a new key` to generate an API key
>    - Copy your API key to somewhere convenient, you will need it throughout this tutorial
>
{: .hands_on}


## Install Ephemeris

To install to training requirements to our Galaxy, we will use Ephemeris, let's install it now:


> <hands-on-title>Install Ephemeris</hands-on-title>
>
> ```bash
> # optional: create a virtual environment
> virtualenv .venv; . .venv/bin/activate
>
> # install ephemeris
> pip install ephemeris
> ```
{: .hands_on}


# Installing tutorial requirements

We have created a small bash script to automatically install all of a tutorial's requirements to an existing Galaxy. It's located in this repository under: [`bin/install_tutorial_requirements.sh`]({{ site.github_repository }}/tree/{{ site.github_repository_branch }}/bin/install_tutorial_requirements.sh)

In this example we will install the requirements for the [*Reference-based RNASeq*]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}) tutorial to the Galaxy instance running on localhost.


> <hands-on-title>Install a tutorial</hands-on-title>
>
> 1. If you have not done so yet, clone the training material github repo:
>    ```bash
>    git clone https://github.com/galaxyproject/training-material.git
>    cd training-material
>    ```
>
> 2. Run the script to install the RNASeq tutorial
>
>    ```bash
>    bin/install_tutorial_requirements.sh topics/transcriptomics/tutorials/ref-based http://localhost:8080 <api key>
>    ```
>
{: .hands_on}

Installation may take some time. This script will automatically install the tools, create a data library and populate it with the input datasets from Zenodo, install and publish the workflows, and run any data managers that might be required.

The only thing the script cannot currently automate is the installation of the interactive tours. We will now do this manually by copying the contents of the `tours` folder to our Galaxy instance, in the folder `$GALAXY_ROOT/config/plugins/tours`

> <hands-on-title>Install the interactive Tours</hands-on-title>
>
> 1. Copy the `tour.yaml` file from the training materials repo to Galaxy
>    ```bash
>    cp -r topics/transcriptomics/tutorials/ref-based/tours/ $GALAXY_ROOT/config/plugins/tours
>    ```
>
{: .hands_on}




## Installing an entire topic

If you would like to install all the requirements for every tutorial within an entire topic, you can use the script in [`bin/install_topic_requirements.sh`]({{ site.github_repository }}/tree/{{ site.github_repository_branch }}/bin/install_topic_requirements.sh)


## Installing a subset of components

If you would like to pick and choose what to install for each tutorial, below are descriptions of the commands used to install each of the components (tools, workflows, reference data, data libraries, tours) please see the [Quickstart section](#quickstart) for the individual commands used by the script

# Quickstart

Below is the list of commands used in this tutorial.

Using the scripts in this repository:

```bash
# Make sure you are in the root of the training-material repo
cd <training-materials repo root>

# install single tutorial
bin/install_tutorial_requirements.sh topics/<yourtopic>/tutorials/<yourtutorial> <Galaxy URL> <API key>

# install entire topic
bin/install_topic_requirements.sh topics/<yourtopic> <Galaxy URL> <API key>

```

Using ephemeris directly:

```
# install tools
shed-tools install -g <Galaxy URL> -a <API key> -t <(curl https://training.galaxyproject.org/training-material/api/topics/<topic>/tutorials/<tutorial>/tutorial.json | jq .admin_install_yaml -r)

# create data library with input datasets
setup-data-libraries -g <Galaxy URL> -a <API key> -i topics/<topic>/tutorials/<tutorial>/data-library.yaml

# install reference data
run-data-managers -g <Galaxy URL> -a <API key> --config topics/<topic>/tutorials/<tutorial>/data-manager.yaml

# install workflows
workflow-install --publish-workflows -g <Galaxy URL> -a <API key> -w topics/<topic>/tutorials/<tutorial>/workflows

# install tours
copy the contents of the "tours" directory for the tutorial to Galaxy's "config/plugins/tours/"
```

When data libraries are created, depending on the server settings, they may be created privately. You will need to fix their permissions either through the UI or the API if you have many datasets. UseGalaxy.eu uses [a script](https://github.com/usegalaxy-eu/shared-data/blob/fb9b4ed069dcd59a39b40e9c7ea08a068d8a9df2/run.sh) to correct the permissions.

# Conclusion


topic: teaching
tutorial: running-workshop
title: Running a workshop as an instructor
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 63d502be15a3905d4b15b621b18abb0e
last_modified: 2026-02-26T21:12:18.342951
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/teaching/tutorials/running-workshop/tutorial.html

content:
In this tutorial, you'll learn how to run a workshop as a Galaxy Instructor


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Tips for instructors

## Using the Training Material

1. Decide (individually or together) if you will:

     1. Run through the tutorial with them step by step, waiting for everyone at each step
     2. OR allow trainees to follow along at their own pace

   Each of these options has pros and cons:

   - Step by step

      - Pro: everyone is on the same step, helpers can see similar errors at similar times that they can point out to the group at once
      - Con: Some trainees will be faster, and some will be slower. If you wait for everyone, it may cause you to go at a slower pace than expected, depending on their skill levels

   - At their own pace:

      - Pro: Those who want to explore features have time to do so. If they are intrigued by the workflow editor, they can play around with it for some time before moving on.
      - Con: People will finish at wildly different times. Be prepared for some users to be done and maybe encourage them with a list of things they can do after they've completed the tutorial, or additional reading material, or just encourage them to try different parameters.

2. Will you present slides? Make sure these are prepared ahead of time. The Training Material offers introductory slides for many topics and many tutorials. See if you can contribute your slides back, if the topic does not currently have any.

## (Un)expected situations

### Students asking extremely specific questions

Occasionally students will either want to show off their knowledge on specific topics, or will just want special attention with their particular research question.
If you can answer the question briefly then this can be good. Otherwise, just ask them to bring it up with you during the next break.

### Interruptions

There are many kinds of interruptions that can occur:

Fire alarm / evacuate the classroom
:    There is not much you can do in this situation. If they bring their stuff with them (they shouldn't if it's a fire alarm), then consider continuing outside. However this can be unpleasant for participants due to weather, or too much sun on their laptops. Make a break, point out nearby attractions, and make a plan to meat back in 30 minutes.

Internet / Server Outage
:    Can you estimate how long this will last? If it is going to be a short interruption, this can be a good time to recap briefly and then ask questions of the students, or suggest students should ask their questions. If it will be a long outage, you can switch to your backup server, OR you can walk the students through the rest of the steps. Be sure to go slowly and be detailed with what would have happened during the steps that they will miss.

(Any others?)

### Unsocial participants

Not everyone is always outwardly enthusiastic about the course for any number of reasons; some students may be very introverted, maybe haven't had their coffee yet, were told to be there by their boss, or just not willing to be social. While you cannot force them everyone, you can encourage try and make activities cover a wide range of groups, from small to large. Some people will be more willing to join smaller groups, or will be more than willing to interact with you one-on-one. Ice breakers that involve one-on-one conversations can be good, a nice one is to have a 'bingo' chart with various aspects about researchers (has PhD, works with viruses, is from local city, is from outside of the country, etc.) and ask them to find people matching these attributes. This allows them to have more personal conversations.

# Checklists

## Before the workshop

1. Read the [teaching recommendations](https://carpentries.github.io/instructor-training/11-practice-teaching.html) of Software Carpentry
2. Read the [organiser recommendations]({% link  topics/teaching/tutorials/organize-workshop/tutorial.md %}) and see if there is anything you can help with
2. Decide on the order of the lessons and who will teach what with the organizers
3. Assist the host in recruiting helpers (people to walk around the classroom and help people if they get stuck)
4. Assist the host in ensuring the workshop location is accessible and everything is ready for the students
5. Share emergency contact information with the host in case of last-minute changes
6. Practice teaching the material
7. Remind the host of any special equipment you might need (e.g. Mac connector)
8. Confirm criteria for reimbursement (per diem or save receipts)
9. Create a recap with the different formats and workflows taught during the workshop and print them

### Prepare your Galaxy

1. Identify where you will run the training:

   1. A Public Galaxy?

      - UseGalaxy.org? UseGalaxy.eu? .org.au? (etc.)
      - If you run on UseGalaxy.eu, they provide free [Training Infrastructure]({% link topics/teaching/tutorials/setup-tiaas-for-training/tutorial.md %})
      - Work with the Galaxy administrator to ensure that the tools and datasets required are available

   2. A Private Galaxy?

      Check our tutorial on [how to set up a Galaxy for Training]({% link topics/teaching/tutorials/setup-galaxy-for-training/tutorial.md %})

      - Technical infrastructure
      - Install the needed tools and data
      - Check the scalability for the number of participants
      - Test the tutorials on the infrastructure

   3. **No matter what**: Have a backup plan!

2. Run the tutorials several times on your server and your backup server

3. Identify the time / memory limiting steps and plan accordingly

## Before the participants arrive

1. Check accessibility

    - Set up the computer and go to the back of the room. Is the font used in your slides big enough? Is the font for showing the training materials?
    - If your room has a mic, use it, and have a helper stand in the back of the room to confirm that you can be heard
    - If the participants are new to the environment, consider hanging up guiding signs to help them find the room.

## During the workshop

1. Review Code of Conduct with participants
2. Support host in collecting attendee names and emails
3. Remind learners to use sticky notes to give feedback

   Ask "Who is not yet finished?" and remind them to place the post-it notes so you can see quickly whether or not you can continue (assuming you have chosen step-by-step instructions). If you are using the at-your-own-pace instruction, you can ask them to place postit notes if they have questions (better than their arms getting tired.)

4. Make regular breaks (every 1-2 hours for 15-30 minutes)
5. Engage and interact with the participants

    You can use gamification and interactive examples to help participants learn. For example: use LEGO bricks to demonstrate how mapping works (TODO: Link to how to do this demo.)

    Find someone you haven't talked to yet today, and talk to them. Make sure all the helpers have the same goal.

6. Regularly ask questions of the trainees:

   - What data formats work here?
   - Where is the data coming from?
   - What data transformation is being made?
   - What were the data formats used today?
   - What were the main steps of today's workflow?

7. Regularly remind trainees of things you've taught them

   Repeat again and again, in different ways, to reinforce concepts

   Explain and re-explain file formats and group them for their uses (e.g. SAM/BAM/CRAM are sequence alignment formats, fasta is for genomic sequences, FastQ variants all store sequencing reads)

8. Draw the workflow, maybe interactively ask students to remind you of the next steps
9. Do a recap at the end of each day about what happened during the day
10. Do a recap each morning about what happened yesterday
11. Provide the students with a recap of the formats and workflows as a take-home

## After the workshop

1. Submit receipts for reimbursement
2. Report any training errors or suggest improvements on GitHub, Matrix or by email to the GTN
3. Update this document with anything that you did that wasn't listed.

# Conclusion

Workshop organization encompasses a range of different aspects to consider. This tutorial can help you keep track of the different tasks. Please add any missing points to continue improving this tutorial.


topic: teaching
tutorial: live-coding
title: Live Coding is a Skill
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 73cd3345ac1ababf032068fe78ea88f2
last_modified: 2026-02-26T21:12:18.335967
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/teaching/tutorials/live-coding/tutorial.html

content:
One of the most effective ways to teach new computational skills is through live coding {% cite Rubi2013 %} {% cite Haar2017 %} {% cite Raj2018 %}: *instructors do not use slides to teach the material*, but work through the lesson material, typing in the code or going through the instructions, with the workshop participants following along. This tutorial explains how it works, why we use it, and gives general tips for an effective participatory live coding presentation. We will finish this tutorial by practicing ourselves and providing feedback for each other.


## Why Participatory Live Coding?

We do not use slides in our lessons. Instead, instructors plug their laptop into the projector and work through the lesson, typing in the code, reformatting data, and talking as we go. This is called ["live coding"](https://en.wikipedia.org/wiki/Live_coding).

However, the instructor is not live coding in a vacuum. Importantly, learners are strongly encouraged to "code-along" with the instructor. We refer to the practice of having the instructor live code and the learners code along as "participatory live coding" or, less formally, 'code-along sessions'.

> <hands-on-title>Up and Down (5 min)</hands-on-title>
>
> List some advantages and challenges of participatory live coding from both a learner's and an instructor's point of view.
>
> > <solution-title></solution-title>
> > Some advantages are:
> > * Watching a program being written is more compelling than watching someone page through slides that present bits and pieces of the same code.
> > * It enables instructors to be more responsive to "what if?" questions. Where a slide deck is like a railway track, participatory live coding allows instructors to go off-road and follow their learners' interests.
> > * Lateral knowledge transfer: participatory live coding facilitates the transfer of [tacit knowledge](https://jonudell.net/udell/2006-09-19-screencasting-of-tacit-knowledge.html) -- people learn more than we realized we were teaching by watching *how* instructors do things.
> > * It slows the instructor down: if she has to type in the program as she goes along, she can only go twice as fast as her learners, rather than ten-fold faster as she could with slides.
> > * Learners get to see instructors' mistakes *and how to diagnose and correct them*. Novices are going to spend most of their time doing this, but it is left out of most textbooks.
> >
> > Some challenges are:
> > * It requires instructors to be able to improvise when things go wrong or when learners have questions not directly addressed in the text of the lesson.
> > It can be hard for learners to listen and type at the same time, due to the *split-attention effect* that was discussed in the [learning principles tutorial]({% link topics/contributing/tutorials/learning-principles/tutorial.md %}). This is why it is very important that instructors first explain what they are going to do, then say what they are typing as they type it, and then explain what they did again afterwards.
> > It may take a bit of practice for instructors to get used to thinking aloud while coding in front of an audience.
> >
> {: .solution}
{: .hands_on}

Live coding fits well into the practice-feedback model we have been discussing - by providing learners with continuous opportunities for practice (every time they type in a line of code) and continuous feedback (their code either works or fails with an error message). It is important to keep in mind, however, that feedback is not helpful if you cannot understand it.
Many error messages are obscure and not written with novices in mind. Continue to use the strategies for error framing that
[was discussed in the motivation tutorial]({% link topics/teaching/tutorials/learner_participation_engagement/tutorial.md %}) to make sure this feedback is useful to learners.

> <hands-on-title>Compare and Contrast (15 min)</hands-on-title>
>
> Watch this first participatory live coding demo video: [https://youtu.be/bXxBeNkKmJE](https://youtu.be/bXxBeNkKmJE) and this second demo video: [https://youtu.be/SkPmwe_WjeY](https://youtu.be/SkPmwe_WjeY) as a group and then summarize your feedback on both. Use the 2x2 rubric for feedback that was [discussed in the feedback tutorial]({% link topics/teaching/tutorials/assessment/tutorial.md %}).
>
> In the videos, the bash shell `for` loop is taught, and it is assumed learners are familiar with how to use a variable, the `head` command and the content of the `basilisk.dat unicorn.dat` files.
>
> Note: Sometime sounds in the room can be poor. Turning on closed captioning by pressing the cc button will improve the accessibility of these videos.List some advantages and challenges of participatory live coding from both a learner's and an instructor's point of view.
>
> > <solution-title></solution-title>
> > The main idea is for the Instructor to lead a discussion about the videos and your feedback on them, making sure that the points of the Top Ten Tips below have been made.
> > If this a self-taught session, please do reflect on your feedback by contrasting against the Top Ten Tips below.
> {: .solution}
>
{: .hands_on}

## Top Ten Tips for Participatory Live Coding in a Workshop
1. **Stand up and move around the room if possible.** This makes the experience more interactive and less monotonous. Use a microphone if one is available to make it easier for people with hearing difficulties to hear you.
2. **Go slowly.** For every command you type, every word of code you write, every menu item or website button you click, say out loud what you are doing while you do it.  Then point to the command and its output on the screen and go through it a second time.  This slows you down and allows learners to copy what you do, or to catch up.  Do not copy-paste code.
3. **Mirror your learner's environment.** Try to create an environment that is as similar as possible to what your learners have to reduce cognitive load. Avoid using keyboard shortcuts.
4. **Use your screen wisely.** Use a big font, and maximize the window.  A black font on a white background works better than a light font on a dark background.  When the bottom of the projector screen is at the same height, or below, the heads of the learners, people in the back will not be able to see the lower parts.  Draw up the bottom of your window(s) to compensate. Pay attention to the lighting (not too dark, no lights directly on/above the presenter's screen) and if needed, re-position the tables so all learners can see the screen, and helpers can easily reach all learners.
5. **Use illustrations** to help learners understand and organize the material. You can also generate the illustrations on the board as you progress through the material.  This allows you to build up diagrams, making them increasingly complex in parallel with the material you are teaching.  It helps learners understand the material, makes for a more lively workshop and gathers the learners' attention to you as well.
6. **Turn off notifications** on your laptop and phone.
7. **Stick to the lesson material.** The core Carpentries lessons are developed collaboratively by many instructors and tried and tested at many workshops.  This means they are very streamlined - which is great when you start teaching them for the first time.  It may be tempting to deviate from the material because you would like to show a neat trick, or demonstrate some alternative way of doing something.  Do not do this, since there is a fair chance you will run into something unexpected that you then have to explain.  If you really want to use something outside of the material, try it out thoroughly before the workshop: run through the lesson as you would during the actual teaching and test the effect of your modification.
Some instructors use printouts of the lesson material during teaching. Others use a second device (tablet or laptop) when teaching, on which they can view their notes and the Etherpad session.  This seems to be more reliable than displaying one virtual desktop while flipping back and forth to another.
8. **Leave no learner behind.** Use sticky notes, see below, to gauge learners' progress and understanding.
9. **Embrace mistakes.** No matter how well prepared you are, you will make mistakes. This is OK! Use these opportunities to do error framing ([as discussed in the motivation tutorial]({% link topics/teaching/tutorials/learner_participation_engagement/tutorial.md %})) and to help your learners learn the art of troubleshooting.
10. **Have fun!** It is OK to use humor and improvisation to liven up the workshop. This becomes easier when you are more familiar with the material, and more relaxed. Start small, even just saying 'that was fun' after something worked well is a good start.

Read more in [Ten quick tips for teaching with participatory live-coding] {% cite Nederbragt2020 %}, as well as the excellent section on "Live Coding" from the book "Teaching Tech Together" by Greg Wilson {% cite Wilson2019 %}.


> <hands-on-title>Practice Teaching (25 min)</hands-on-title>
>
> 1. Split into groups of three.
> 1. Assign roles, which will rotate: presenter, timekeeper, note-taker.
> 2. Have each group member teach 3 minutes of your chosen lesson episode using live coding. For this exercise, your peers will not "code-along." Before you begin, briefly describe what you will be teaching and what has been learned previously. Do not record this exercise.
> 3. After each person finishes, each group member should share feedback (starting with themselves) using the same 2x2 rubric as yesterday. The timekeeper should keep feedback discussion to about 1 minute per person; this may leave some time at the end for general discussion. The note-taker should record feedback in the Etherpad.
> 4. Trade off roles.
>
> > <solution-title></solution-title>
> > If this a self-taught session, one way to approach this exercise is by recording your teaching demonstration (e.g. through a cell-phone), and attempting to provide an objective self-feedback based on the 2x2 rubric.
> {: .solution}
{: .hands_on}




# Recap

##


topic: teaching
tutorial: organize-workshop
title: Organizing a workshop
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 84a9be9dd042d4e0a90b8985539ae6b2
last_modified: 2026-02-26T21:12:18.342644
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/teaching/tutorials/organize-workshop/tutorial.html

content:
These recommendations are highly inspired from the one by [The Carpentries](https://docs.carpentries.org/topic_folders/hosts_instructors/index.html)

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Groundwork

1. Decide on a general topic for the workshop
2. Identify possible dates for the workshop
3. Identify possible instructors, and contact them for their availability

## Before the workshop

1. Discuss with the instructors to decide on the curriculum
2. Set dates for the workshop
3. Find a good venue

    A good venue is crucial to establishing a positive learning environment. Some things to consider include:

    - Tables arranged so participants can watch the instructor, use their laptops, and talk with their peers

      Fold up desks are usually not a good choice.

      If a computer pool or computer room is available, these are usually a good choice since they come equipped with desktops and large monitors. If you allow students to bring their own laptops, they will sometimes bring very tiny ones and then struggle to read tutorials at the same time as following along.

    - Tables arranged so helpers and instructors can easily walk around the room and talk with individual participants
    - A room with a projector and screen that everyone can see, with HDMI, VGA, and Mac adapters to allow anyone to plug in their laptop
    - Hi speed internet access (WiFi if not in a computer room) that can support everyone using it simultaneously
    - High table or podium that instructor can stand behind while teaching

      A flat podium is important; it’s difficult to use a laptop on a slanted podium

    - Black/white boards can be useful for drawing diagrams. These work best in smaller rooms, in larger rooms people sometimes cannot see from the back, and slides can be better here

4. Ensure your location is accessible for those with different needs

    It's important to make sure the workshop space is accessible to all individuals. The registration form should ask whether instructors, helpers, or learners need any advance arrangements to ensure they can participate in the event. This may include considering whether or not:

    - Building and room are accessible to those who can not use stairs (ramps, elevators, etc.)
    - Restrooms are accessible
    - There is a microphone for instructors
    - The screen is large enough and bright enough to be easily read
    - The building and room can accommodate service animals
    - Lactation space is provided

    [Here is an additional (US-centric)](https://www.adaatyourservice.org/accessible-meetings-events-conferences-guide/book) guide book covering all aspects of event accessibility required under the ADA in the US. Inspiration can be taken for other non-US countries, as the list is quite extensive.

5. Create a registration form
    1. Decide if there will be a workshop fee

      Even a small fee can greatly reduce the no-show rate.

    2. Registration form: Google Forms or Eventbrite if fee

      Example of regular questions

    3. If there are social events planned, ask if students will participate in them
6. Create a small webpage with all the information about the workshop

    Ideally, such a page should include:

    1. General presentation
    2. Registration information
    2. Schedule with topics, presenters and helpers, start / end, links to material (latter) for each day
    3. Location of the venue with extra information
      1. Map
      2. Lunch options around
      3. Accomodation options around
    4. Organizers

    It can be easily done on the [Galaxy Community Hub](https://galaxyproject.org/) or [European Galaxy](https://github.com/usegalaxy-eu/website/) website, depending on the audience
7. Advertise your workshop on the Galaxy Hub and usegalaxy.\*

    PR / Issue template for training events?

8. Prepare for visiting instructors
   - Make travel arrangements for them or communicate the reimbursement process
   - Make accommodation arrangements for them or maybe point out some common choices using your local knowledge
   - If you will give instructors a thank-you gift, find something that is inclusive regardless of their background (so wine / chocolate / food are bad choices.)
9. Collect emergency contact information for instructors in case of last minute changes
10. Arrange required materials and equipment
    - Power outlets for all participants

      This may include setting up extension cords or power strips

    - Sticky notes in two contrasting colors
    - Name tags to help people get to know each other
    - Extra pens and paper in case people want to take notes by hand
    - Sign in sheets
    - Pens for black/white board
11. Arrange for beverages, snacks, and meals
    - We all learn better with fuel!
    - Catering can be expensive. If you choose to have participants find their own lunch, then you should add nearby lunch location ideas to the workshop webpage
    - If your workshop is a multi-day workshop, you should include dinner options on the workshop webpage
12. Test the projector with a different computers
13. Contact participants to remind them of workshop webpage where you have written location/directions, start/end times, and contact information
14. Contact instructors and help to remind them of workshop webpage
15. Make a reservation for a social dinner
16. Prepare the participation certificates
17. Test the training materials that will be taught from the perspective of a student in the room you have booked

    In one case the training material was taught in English, but the computers were in German. Some participants could not find the "Copy link location" that was mentioned in the tutorial. If you have sections that instruct users to do things like this, make sure to check how they will see it!

18. Name badges for the instructors, usually with a different colour border so attendees can identify people who can help them with any issues they encounter.
19. Name badges for attendees are important for participants as well. At busy venues it can help attendees to find each other and remember each others' names.
20. Ensure that your catering service will label the food for contents (e.g. for vegetarian/vegan/allergens like gluten, etc.) This is extremely important for attendees to be able to enjoy themselves and find food that will not hurt them.
21. Identify places you can promote your event [Galaxy Hub](https://github.com/galaxyproject/galaxy-hub) is the primary place to do this. You should do this 3-6 months in advance of your event (if you can)

## During the workshop

### Before the participants arrive

1. Ensure that the room is set up with all necessary materials and equipment
2. Make coffee/tea

### After it begins

1. Introduce the workshop
    1. Explain the emergency plan
    2. Explain the schedule
    4. Organize a icebreaker (bingo sheet, for example) and ask for the data needs

        Everything is about networking!
2. Remind them of the resources on the webpage
    1. The schedule
    2. Point out the lunch options
3. Periodically check in on the class to be sure things are running smoothly


### In the mornings

1. Remind the participants what they have learned on the previous day. Show a quick slideshow covering the major themes
2. Organise an ice breaker to help them make new friends they didn't meet last time

### In the evenings

1. Before they leave in the evening, recap what was learned that day, just the highlights
2. Sometime later in the evening, contact the participants to remind them:
  - it's happening
  - what the schedule is
  - what's going on the next day
3. Prepare the recap slides for the next morning, if they do not exist already
4. Update the workshop webpage with links to each training material that was presented that day


### Before they leave

1. Hand out participation certificates

## After the workshop

1. Collect workshop attendance
2. Reimburse the instructors, thank them for their participation
3. Collect feedback form and discuss the results with the instructors
   - Ask what went right? What went wrong?
   - This is a good opportunity to identify things that are missing from this list
4. Contact the participants with the links to the materials

# Conclusion


