# tutorials

topic: microbiome
tutorial: beer-data-analysis
title: Identification of the micro-organisms in a beer using Nanopore sequencing
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: ["nanopore", "beer", "citizen science", "metagenomics", "microgalaxy"]
content_hash: e54c7e58446a450d1cd5a056aa630d89
last_modified: 2026-05-15T22:28:13.936597
zenodo_link: https://doi.org/10.5281/zenodo.7093173
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/beer-data-analysis/tutorial.html

content:
What is a microbiome? It is a collection of small living creatures.
These small creatures are called **micro-organisms** and they are **everywhere**. In our gut,
in the soil, on vending machines, and even inside the beer. Most of these micro-organisms are
actually very good for us, but some can make us very ill.

Micro-organisms come in different shapes and sizes, but they have the same components.
One crucial component is the **DNA**, the blueprint of life. The DNA encodes the
shape and size and many other characteristics unique to a species. Because DNA is so species-specific,
reading the DNA can be used to identify what kind of micro-organism
the it is from. Therefore, within a metagenomic specimen, *e.g.* a sample form soil, gut,
or beer, one can identify what kind of species are inside the sample.

In this tutorial, we will use data of beer microbiome generated via the
[BeerDEcoded project](https://streetscience.community/projects/beerdecoded/).

> <details-title>The BeerDEcoded project</details-title>
>
> The BeerDEcoded Project is a series of workshops organized with and for schools as well as the general
> audience, aiming to introduce biology and genomic science. People learn in an interactive way
> about DNA, sequencing technologies, bioinformatics, open science, how these technologies
> and concepts are applied and how they are impacting their daily life.
>
> Beer is alive and contains many microorganisms. It can be found in many places
> and there are many of them. It is a fun media to bring the people to the
> contact of molecular biology, data-analysis, and open science.
>
> A BeerDEcoded workshop includes the following steps:
> 1. Extract yeasts and their DNA from beer bottle,
> 2. Sequence the extracted DNA using a MinION sequencer to obtain the sequence of bases/nucleotides (A, T, C and G) for each DNA fragment in the sample,
> 3. Analyze the sequenced data in order to know which organisms this DNA is from
>
> ![The image represents a BeerDEcoded workshop. On the left, there is a beer glass. An arrow goes from the bottle to DNA with "Extraction" written on the below. An arrow goes from DNA to DNA sequences with "Sequencing" written on the below. An arrow goes from the DNA sequences to Yeasts with "Data analysis" written on the below](./images/beerprocess.png)
>
{: .details}

> <comment-title>Beer microbiome</comment-title>
>
> Beer is alive! It contains microorganisms, in particular **yeasts**.
>
> Indeed, grain and water create a sugary liquid (called wort). The beer brewer
> adds yeasts to it. By eating the sugar, yeast creates alcohol,
> and other compounds (esters, phenols, etc.) that give beer its particular
> flavor.
>
> Yeasts are microorganisms, more precisely **unicellular fungi**. The majority
> of beers use a yeast genus called ***Saccharomyces***, which in Greek means
> "sugar fungus". Within that genus, two specific species of *Saccharomyces*
> are the most commonly used:
>
> - ***Saccharomyces cerevisiae***: a top-fermenting (*i.e.* yeast which rise
> up to the top of the beer as it metabolizes sugars, delivering alcohol as a by-product),
> ale yeast responsible for a huge range of beer styles like witbiers, stouts, ambers,
> tripels, saisons, IPAs, and many more. It is most likely the yeast that the early brewers were
> inadvertently brewing with over 3,000 years ago.
> - ***Saccharomyces pastorianus***: a bottom-fermenting (*i.e.* it sits on the
> bottom of the tank as it ferments) lager yeast, responsible for beer styles
> like Pilsners, lagers, märzens, bocks, and more. This yeast was originally
> found, and cultivated, by Bavarian brewers a little over 200 years ago. It is
> the most commonly used yeast in terms of the raw amount of beer produced around the world.
>
> Since yeast is all around us, we can actually brew spontaneously
> fermented beer by using wild yeasts and souring microbiota floating through the air.
>
{: .comment}

During one BeerDEcoded workshop, we extracted yeasts out of a bottle of
[Chimay](https://en.wikipedia.org/wiki/Chimay_Brewery). We then
extracted the DNA of these yeasts and sequenced it using a MinION to obtain the DNA
sequences. Now, we would like to **identify the yeast species** sequenced there, and
thereby **outline the diversity of microorganisms** (the microbiome community) in the beer
sample.

To get this information, we need to process the sequenced data in a few steps:
1. Check the quality of the data
2. Assign a taxonomic label, *i.e.* assign 'species' to each sequence
3. Visualize the distribution of the different species

This type of data analysis requires running several bioinformatics tools and
usually requires a computer science background. [Galaxy](https://galaxyproject.org/) is
an open-source platform for data analysis that enables anyone to use bioinformatics
tools through its graphical web interface, accessible via any Web browser.

So, in this tutorial, we will use Galaxy to extract and visualize the community
of yeasts from a bottle of beer.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Prepare Galaxy and data

First of all, this tutorial will get you hands on with some basic Galaxy tasks, including creating a history and importing data.

## Get familiar with Galaxy

> <hands-on-title>Open Galaxy</hands-on-title>
>
> 1. Open your favorite browser (Works on Chrome, Firefox, Safari but not Internet Explorer!)
> 2. Create a Galaxy account if you do not have one
>
>    {% snippet faqs/galaxy/account_create.md %}
>
{: .hands_on}

The Galaxy homepage is divided into three panels:
* Tools on the left
* Viewing panel in the middle
* History of analysis and files on the right

![Galaxy interface screenshot showing history panel on the right, tools panel on the left, and main panel at the center]({% link topics/introduction/images/galaxy_interface.png %} "The Galaxy interface")

The first time you use Galaxy, there will be no files in your history panel.

Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Prepare the Galaxy history</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

## Get data

Before we can begin any Galaxy analysis, we need to upload the input data: FASTQ files

> <hands-on-title>Upload your dataset</hands-on-title>
>
> 1. Import the sequenced data including fastq in the name
>
>    - Option 1 [{% icon video %}](https://youtu.be/FFCDx1rMGAQ): Your own local data using **Upload Data** (recommended for 1-10 datasets).
>
>      {% snippet faqs/galaxy/datasets_upload.md %}
>
>    - Option 2: From Zenodo, an external server, via URL
>
>      ```text
>      https://zenodo.org/record/7093173/files/ABJ044_c38189e89895cdde6770a18635db438c8a00641b.fastq
>      ```
>
>      {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    Your uploaded file is now in your current history. When the file is fully uploaded to Galaxy, it will turn green. But, what is this file?
>
> 2. Click on the {% icon galaxy-eye %} (eye) icon next to the dataset name to look at the file contents.
>
{: .hands_on}

The contents of the file will be displayed in the central Galaxy panel.

This file contains the sequences, also called **reads**, of DNA, *i.e.* succession of nucleotides, for all fragments from the yeasts in the beer, in FASTQ format.

{% snippet topics/sequence-analysis/faqs/fastq.md %}

# Data quality

## Assess data quality

Before starting to work on our data, it is necessary to assess its quality. This is an essential step if we aim to obtain a **meaningful downstream analysis**.

**FastQC** is one of the most widely used tools to **check the quality** of data generated by High Throughput Sequencing (HTS) technologies.

> <hands-on-title>Quality check</hands-on-title>
>
> 1. {% tool [FASTQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0) %} with the following parameters
>    - {% icon param-file %} *"Raw read data from your current history"*: `Reads`
>
> 2. Inspect the generated HTML file
>
{: .hands_on}

> <question-title></question-title>
>
> Given the Basic Statistics table on the top of the page:
>
> ![Screenshot of the FastQC Basic Statistics with filename, file type (Conventional base calls), encoding (Sanger / Illumina 1.9), total Sequences (1876), sequences flagged as poor quality (0), sequence length (130-2327) and %GC (29)](./images/fastqc_top_table.png)
>
> 1. How many sequences are in the FASTQ file?
> 2. How long are the sequences?
>
> > <solution-title></solution-title>
> >
> > 1. There are 1876 sequences.
> > 2. The sequences range from 130 nucleotides to 2327 nucleotides. Not all sequences have then the same length.
> >
> {: .solution}
>
{: .question}

**FastQC** provides information on various parameters, such as the range of quality values across all bases at each position:

![FastQC Per base sequence quality with scores below 20](./images/fastqc_1.png "Per base sequence quality. X-axis: position in the reads (in base pair). Y-axis: quality score, between 0 and 40 - the higher the score, the better the base call. For each position, a boxplot is drawn with: the median value, represented by the central red line;the inter-quartile range (25-75%), represented by the yellow box; the 10% and 90% values in the upper and lower whiskers; and the mean quality, represented by the blue line. The background of the graph divides the y-axis into very good quality scores (green), scores of reasonable quality (orange), and reads of poor quality (red).")

We can see that the quality of our sequencing data grows after the first few bases, stays around a score of 18 and then decreases again at the end of the sequences. MinION and Oxford Nanopore Technologies (ONT) are known to have a higher error rate compared to other sequencing techniques and platforms ({% cite delahaye2021sequencing %}).

For more detailed information about the other plots in the FASTQC report, check out our [dedicated tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).

## Improve the dataset quality

In order to improve the quality of our data, we will use two tools:
- **porechop** ({% cite rrwick2017 %}) to remove adapters that were added for sequencing and chimera (contaminant)
- **fastp** ({% cite Chen_2018 %}) to filter sequences with low quality scores (below 10)

> <hands-on-title>Improve the dataset quality</hands-on-title>
>
> 1. {% tool [Porechop](toolshed.g2.bx.psu.edu/repos/iuc/porechop/porechop/0.2.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input FASTA/FASTQ"*:  `Reads`
>    - *"Output format for the reads"*: `fastq`
>
> 2. {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.23.2+galaxy0) %} with the following parameters:
>    - *"Single-end or paired reads"*: `Single-end`
>      - {% icon param-file %} *"Input 1"*: output of **Porechop**
>      - In *"Adapter Trimming Options"*:
>        - *"Disable adapter trimming"*: `Yes`
>    - In *"Filter Options"*:
>        - In *"Quality filtering options"*:
>            - *"Qualified quality phred"*: `10`
>    - In *"Read Modification Options"*:
>        - *"PolyG tail trimming"*: `Disable polyG tail trimming`
>
> 3. Inspect the HTML report of **fastp** to see how the quality has been improved
{: .hands_on}

> <question-title></question-title>
>
> 1. How many sequences are there before filtering? Is it the same number as in FASTQC report?
> 2. How many sequences are there after filtering? How many sequences have then been removed by filtering?
> 3. What is the mean length before filtering? And after filtering?
>
> > <solution-title></solution-title>
> >
> > 1. There are 1,869 reads before filtering. The number is lower than in the FASTQC report. Some reads may have been discarded via Porechop
> > 2. There are 1,350 reads after filtering. So the filtering step has removed $$1869-1350 = 519$$ sequences.
> > 3. The mean length is 314 nucleotide before filtering and 316bp after filtering.
> {: .solution}
>
{: .question}

# Assign taxonomic classification

One of the main aims in microbiome data analysis is to identify the organisms sequenced. For that we try to **identify the taxon** to which each individual read belong.

{% snippet topics/microbiome/faqs/taxon.md %}

> <question-title></question-title>
>
> 1. Which microorganisms do we expect to identify in our data?
> 2. What is the taxonomy of the main expected microorganism?
>
> > <solution-title></solution-title>
> >
> > 1. The sequences are supposed to be yeasts extracted from a bottle of beer. The majority of beers contain a yeast genus called ***Saccharomyces*** and 2 species in that genus: *Saccharomyces cerevisiae* (ale yeast) and *Saccharomyces pastorianus* (lager yeast). The used beer is an ale beer, so we expect to find ***Saccharomyces cerevisiae***. But other yeasts can also have been used and then found. We could also have some DNA left from other beer components, but also contaminations by other microorganisms and even human DNA from people who manipulated the beer or did the extraction.
> >
> >
> > 2. The main expected microorganism is ***Saccharomyces cerevisiae*** with its taxonomy:
> >
> >    Level | Classification
> >    --- | ---
> >    Domain | Eukaryota
> >    Kingdom | Fungi
> >    Phylum | Ascomycota
> >    Class | Saccharomycetes
> >    Order | Saccharomycetales
> >    Family | Saccharomycetaceae
> >    Genus | *Saccharomyces*
> >    Species | *S. cerevisiae*
> >
> {: .solution}
>
{: .question}

Taxonomic assignment or classification is the process of assigning an **Operational Taxonomic Unit** (OTUs, that is, groups of related individuals / taxon) to sequences. To assign an OTU to a sequence it is compared against a database, but this comparison can be done in different ways, with different bioinformatics tools. Here we will use **Kraken2** ({% cite wood2019improved %}).

{% snippet topics/microbiome/faqs/kraken.md %}

> <hands-on-title>Kraken2</hands-on-title>
>
> 1. {% tool [Kraken2](toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.1.1+galaxy1) %} with the following parameters:
>    - *"Single or paired reads"*: `Single`
>        - {% icon param-file %} *"Input sequences"*: Output of **fastp**
>    - *"Print scientific names instead of just taxids"*: `Yes`
>    - In *"Create Report"*:
>        - *"Print a report with aggregrate counts/clade to file"*: `Yes`
>    - *"Select a Kraken2 database"*: `Prebuilt Refseq indexes: PlusPF`
>
>      The database here contains reference sequences and taxonomies. We need to
>      be sure it contains yeasts, i.e. fungi.
>
> 2. Inspect the report file
{: .hands_on}

The Kraken report is a tabular files with one line per taxon and 6 columns or fields:

1. Percentage of fragments covered by the clade rooted at this taxon
2. Number of fragments covered by the clade rooted at this taxon
3. Number of fragments assigned directly to this taxon
4. A rank code, indicating
    - (U)nclassified
    - (R)oot
    - (D)omain
    - (K)ingdom
    - (P)hylum
    - (C)lass
    - (O)rder
    - (F)amily
    - (G)enus, or
    - (S)pecies

    Taxa that are not at any of these 10 ranks have a rank code that is formed by using the rank code of the closest ancestor rank with a number indicating the distance from that rank. E.g., `G2` is a rank code indicating a taxon is between genus and species and the grandparent taxon is at the genus rank.

5. NCBI taxonomic ID number
6. Indented scientific name


```
Column 1	Column 2	Column 3	Column 4	Column 5	Column 6
38.00 	513 	513 	U 	0 	unclassified
62.00 	837 	1 	R 	1 	root
61.93 	836 	27 	R1 	131567 	cellular organisms
56.00 	756 	3 	D 	2759 	Eukaryota
55.33 	747 	3 	D1 	33154 	Opisthokonta
29.78 	402 	0 	K 	33208 	Metazoa
29.78 	402 	0 	K1 	6072 	Eumetazoa
29.78 	402 	0 	K2 	33213 	Bilateria
29.78 	402 	0 	K3 	33511 	Deuterostomia
29.78 	402 	0 	P 	7711 	Chordata
29.78 	402 	0 	P1 	89593 	Craniata
29.78 	402 	0 	P2 	7742 	Vertebrata
29.78 	402 	0 	P3 	7776 	Gnathostomata
29.78 	402 	0 	P4 	117570 	Teleostomi
29.78 	402 	0 	P5 	117571 	Euteleostomi
29.78 	402 	0 	P6 	8287 	Sarcopterygii
29.78 	402 	0 	P7 	1338369 	Dipnotetrapodomorpha
29.78 	402 	0 	P8 	32523 	Tetrapoda
29.78 	402 	0 	P9 	32524 	Amniota
29.78 	402 	0 	C 	40674 	Mammalia 
```

> <question-title></question-title>
>
> 1. How many taxons have been identified?
> 2. How much reads have been classified?
> 3. Which domains were found and with how many reads?
> 4. How much reads have been assigned by fungi Kingdom?
>
> > <solution-title></solution-title>
> >
> > 1. The file contains 300 lines (*information visible when expanding the report dataset in the history panel*). So 300-2 = 298 taxons have been identified.
> >
> > 2. On the 1350 sequences in the input, 837 (62%) were classified (or identified as a taxon) and 513 unclassified (38%). *Information visible when expanding the report dataset in the history panel, and scrolling in the small box starting with "Loading database information" below the format information, but also on the top of the report.*
> >
> > 3. The domains are identified by a `D` in colum 4. 
> >
> >    To get the domains (or other taxonomic level), we can use {% tool [Filter](Filter1) %} with the following parameters:
> >    - {% icon param-file %} *"Filter"*: report outpout of **Kraken2**
> >    - *"With following condition"*: `c4=='D'` 
> >
> >    The 3 domains were found:
> >    - Eukaryota with 756 (56%) reads assigned to it
> >    - Bacteria with 51 (3.78%) reads
> >    - Archaea with 2 (0.15%) reads
> >
> > 4. 342 (25.33%) reads are assigned to fungi.
> {: .solution}
>
{: .question}

Other taxons than yeast have been identified. They could be contamination or misidentification of reads. Indeed, many taxons have less than 5 reads assigned. We will filter these reads out to get a better view of the possible contaminations.

> <hands-on-title>Filter taxons with low assignements</hands-on-title>
>
> 1. {% tool [Filter](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: report outpout of **Kraken2**
>    - *"With following condition"*: `c2>5`
>
>      We want to keep only taxons with more than 5 reads assigned, *i.e.* the value in the 2nd column, is higher than 5.
>
> 2. Inspect the output
{: .hands_on}

> <question-title></question-title>
>
> 1. How many taxons have been removed? How many were kept?
> 2. What are the possible contaminations?
>
> > <solution-title></solution-title>
> > 1. 59 lines are now in the file so $$300 - 59 = 241$$ taxons have been removed because low assignment rates.
> >
> > 2. Most of the reads (402) were assigned to humans (*Homo sapiens*). This is likely a contamination either during the beer production or more likely during DNA extraction.
> >
> >    Bacteria were also found: Firmicutes, Proteobacteria and Bacteroidetes. But the identified taxons are not really precise (not below order level). So difficult to identify the possible source of contamination.
> {: .solution}
{: .question}

# Visualize the community

Once we have assigned the corresponding taxa to the sequences, the next step is to properly visualize the data: visualize the diversity of taxons at different levels.

To do that, we will use the tool **Krona** ({% cite Ondov_2011 %}). But before that, we need to adjust the output from Kraken2 to the requirements of **Krona**. Indeed, **Krona** expects as input a table with the first column containing a count and the remaining columns describing the hierarchy. Currently, we have a report tabular file with the first column containing the taxonomy and the second column the number of reads. We will now use another tool, which also provides taxonomic classification, but it produces the exact formatting Krona needs.

> <hands-on-title>Prepare dataset for Krona</hands-on-title>
>
> 1. {% tool [Krakentools: Convert kraken report file](toolshed.g2.bx.psu.edu/repos/iuc/krakentools_kreport2krona/krakentools_kreport2krona/1.2+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Kraken report file"*: **Report** output of **Kraken**
>
> 2. Inspect the output file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > Column 1	Column 2	Column 3	Column 4	Column 5	Column 6	Column 7	Column 8
>    > 513 	Unclassified 						
>    > 7 	k__Eukaryota 						
>    > 0 	k__Eukaryota 	p__Chordata 					
>    > 0 	k__Eukaryota 	p__Chordata 	c__Mammalia 				
>    > 0 	k__Eukaryota 	p__Chordata 	c__Mammalia 	o__Primates 			
>    > 0 	k__Eukaryota 	p__Chordata 	c__Mammalia 	o__Primates 	f__Hominidae 		
>    > 0 	k__Eukaryota 	p__Chordata 	c__Mammalia 	o__Primates 	f__Hominidae 	g__Homo 	
>    > 402 	k__Eukaryota 	p__Chordata 	c__Mammalia 	o__Primates 	f__Hominidae 	g__Homo 	s__Homo_sapiens					
>    > ```
>    >
>    > 1. What are the columns in the file?
>    >
>    > > <solution-title></solution-title>
>    > > 1. 8 columns: one with the number of reads and one for each of the 7 levels of taxonomy.
>    > {: .solution}
>    {: .question}
{: .hands_on}

We can now run **Krona**. This tool creates an interactive report that allows hierarchical data (like taxonomy) to be explored with zooming, as multi-layered pie charts. With this tool, we can easily visualize the composition of a microbiome community.

> <hands-on-title>Krona pie chart</hands-on-title>
>
> 1. {% tool [Krona pie chart](toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1+galaxy0) %} with the following parameters:
>    - *"What is the type of your input data"*: `Tabular`
>        - {% icon param-file %} *"Input file"*:  output of **Krakentools** {% icon tool %}
>
> 2. Inspect the generated file
{: .hands_on}

Let's take a look at the result.

<iframe id="krona" src="krona.html" frameBorder="0" width="100%" height="900px"> ![Krona chart with multi-layered pie chart representing the community profile with in the center the higher taxonomy levels (i.e. domain) and on the exterior the more detailed ones (i.e. species)](./images/krona.png) </iframe>

> <question-title></question-title>
>
> 1. What is the percentage of reads assigned to *Homo sapiens*? 
> 2. To Archaea?
>
> > <solution-title></solution-title>
> >
> > 1. 30% of reads are assigned to *Homo sapiens*
> > 2. 0.08% of Archaea
> {: .solution}
>
{: .question}

# Investigate the beer microbiome

Let's come back to our original question: characterization of the beer microbiome, specially looking at the yeasts.

Yeasts do not form a single taxonomic group ({% cite kurtzman1994molecular %}). They are parts of the fungi kingdom but belong two separate phyla: the Ascomycota and the Basidiomycota. [But the "true yeasts" are classified in the order Saccharomycetales](https://web.archive.org/web/20090226151906/http://www.yeastgenome.org/VL-what_are_yeast.html).

> <question-title></question-title>
> 1. Click on `o__Saccharomycetales` in the graph (Krona pie chart). Which yeast species have been identified? Are they the expected in beer?
> 2. Click on *Saccharomyces* in the graph. What are the percentages of identified reads assigned to *Saccharomyces* for different levels?
> 3. Click a second time on *Saccharomyces* in the graph. What is the repartition between the different *Saccharomyces* species?
>
> > <solution-title></solution-title>
> >
> > 1. 6 species from the Saccharomycetales order have been identified:
> >    - Saccharomycetaceae family
> >      - *Saccharomyces* genus
> >        - *Saccharomyces cerevisiae* species, the most abundant identified yeast species with 293 reads and the one expected given the type of beers
> >        - *Saccharomyces paradoxus* species, a wild yeast and the closest known species to *Saccharomyces cerevisiae*
> >
> >          These reads might have been misidentified to *Saccharomyces paradoxus* instead of *Saccharomyces cerevisiae* because of some errors in the sequences, as *Saccharomyces cerevisiae* and *Saccharomyces paradoxus* are close species and should share then a lot of similarity in their sequences.
> >
> >        - *Saccharomyces eubayanus* species, most likely the parent of the lager brewing yeast, *Saccharomyces pastorianus* ({% cite sampaio2018microbe %})
> >
> >          Similar to *Saccharomyces paradoxus*, these reads might have been misassigned.
> >
> >      - *Kluyveromyces* genus - *Kluyveromyces marxianus* species: only 1 read
> >
> >    - Trichomonascaceae family - *Sugiyamaella* genus - *Sugiyamaella lignohabitans* species: only 1 read
> >    - Debaryomycetaceae family - *Candida* genus - *Candida dubliniensis* species: only 1 read
> >
> >    Everything except *Saccharomyces cerevisiae* are probably misindentified reads.
> >
> > 2. Reads are assigned to *Saccharomyces*
> >    - 25% out of total reads (root)
> >    - 44% out of identified reads for Eukaryota domain
> >    - 96% out of identified reads for Ascomycota phylum
> >    - 98% out of identified reads for Saccharomycetales order
> >    - 99% out of identified reads for Saccharomycetaceae family
> >
> >    ![Krona chart with multi-layered pie chart representing the community profile with in the center the higher taxonomy levels (i.e. domain) and on the exterior the more detailed ones (i.e. species). *Saccharomyces* is highligthed with on the right the percentages of identified reads assigned to *Saccharomyces* for the different taxonomic levels](./images/krona_saccharomyces_1.png)
> >
> > 3. 92% of *Saccharomyces* reads are assigned to *Saccharomyces cerevisiae*, 5% to *Saccharomyces paradoxus* and 3% to *Saccharomyces eubayanus*,.
> >
> >    ![Krona chart with the community profile for *Saccharomyces* with the different species](./images/krona_saccharomyces_2.png)
> {: .solution}
>
{: .question}

Microbiome of several beers, including Chimay beers, have been previously investigated by targeting specifically the fungi, in which we can find yeasts ({% cite sobel2017beerdecoded %}):

![Heatmap with rows being type of beers and columns fungi species. For the rows, the name of beers are on the right and clustering tree on the left side of the heatmap. For the columns, the name of the species are on the bottom and the clustering tree on the top. The fungi species are sorted from the least abundant on the left to the most abundant on the right.](./images/beerdecoded.gif "Heatmap of the number of reads per internal transcribed spacer (ITS) sequencing of fungal species per beer. Source {% cite sobel2017beerdecoded %}")

The species identified for Chimay beers are (from the most abundant to the least one):
- *Saccharomyces cerevisiae*
- *Saccharomyces mikatea*: a species generally used in winemaking ({% cite bellon2013introducing %})
- *Kazachstania martiniae*: *Kazachstania* is a genus from the family Saccharomycetaceaethe.
- *Saccharomyces kudriavzevii*
- *Brettanomyces bruxellensis*

    *Brettanomyces* is a non-spore forming genus of yeast in the family Saccharomycetaceae, and is important to both the brewing and wine industries due to the sensory compounds it produces. 

    *Brettanomyces bruxellensis* is typically used for the production of the Belgian beers.

- *Saccharomyces paradoxus*: a wild yeast species closely related to *Saccharomyces cerevisiae*
- *Kazachstania kunashirensis*
- *Saccharomyces cariocanus*: a wild yeast species closely related to *Saccharomyces cerevisiae*
- *Filobasidium magnum*
- *Malasseria restricta*
- *Pichia kudriavzevii*
- *Aureobasidium pullulans*
- *Sporidiobolus metaroseus*

In a structured way:

Phylum | Class | Order | Family | Genus | Species
--- | --- | --- | --- | --- | ---
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces cerevisiae*
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces mikatea*
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces kudriavzevii*
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces paradoxus*
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces cariocanus*
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Kazachstania | *Kazachstania martiniae*
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Kazachstania | *Kazachstania kunashirensis*
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Brettanomyces | *Brettanomyces bruxellensis*
Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae |  | *Pichia kudriavzevii*
Ascomycota | Dothideomycetes | Dothideales | Dothioraceae | Aureobasidium | *Aureobasidium pullulans*
Basidiomycota | Tremellomycetes | Filobasidiales | Filobasidiaceae | Filobasidium | *Filobasidium magnum*
Basidiomycota | Malasseziomycetes | Malasseziales | Malasseziaceae | Malassezia | *Malasseria restricta*
Basidiomycota | Sporidiobolales | Sporidiobolales | Sporidiobolaceae | Sporidiobolus | *Sporidiobolus metaroseus*

> <question-title></question-title>
>
> By looking at the output of **Krakentools**, which fungi species identified for the Chimay beers in {% cite sobel2017beerdecoded %} are also identified in our data? And vice versa?
>
> > <solution-title></solution-title>
> >
> > Phylum | Class | Order | Family | Genus | Species | {% cite sobel2017beerdecoded %} | Our data 
> > --- | --- | --- | --- | --- | --- | --- | ---
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces cerevisiae* | X | X
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces mikatea* | X | 
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces kudriavzevii* | X | 
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces paradoxus* | X | X
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces cariocanus* | X | 
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Saccharomyces | *Saccharomyces eubayanus* | | X
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Kazachstania | *Kazachstania martiniae* | X | 
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Kazachstania | *Kazachstania kunashirensis* | X | 
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Kluyveromyces | *Kluyveromyces marxianus* | | X
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae | Brettanomyces | *Brettanomyces bruxellensis* | X | 
> > Ascomycota | Saccharomycetes | Saccharomycetales | Saccharomycetaceae |  | *Pichia kudriavzevii* | X | 
> > Ascomycota | Saccharomycetes | Saccharomycetales | Debaryomycetaceae  | Candida | *Candida dubliniensis* | | X 
> > Ascomycota | Dothideomycetes | Dothideales | Dothioraceae | Aureobasidium | *Aureobasidium pullulans* | X | 
> > Ascomycota | Sordariomycetes | Sordariales | Sordariaceae | Neurospora | *Neurospora crassa* | | X 
> > Basidiomycota | Tremellomycetes | Filobasidiales | Filobasidiaceae | Filobasidium | *Filobasidium magnum* | X | 
> > Basidiomycota | Malasseziomycetes | Malasseziales | Malasseziaceae | Malassezia | *Malasseria restricta* | X | 
> > Basidiomycota | Sporidiobolales | Sporidiobolales | Sporidiobolaceae | Sporidiobolus | *Sporidiobolus metaroseus* | X | 
> {: .solution}
{: .question}

Some interesting yeast have been found in  {% cite sobel2017beerdecoded %} and not in our data (e.g. *Brettanomyces bruxellensis*), and vice versa.

# (Optional) Sharing your history

One of the most important features of Galaxy comes at the end of an analysis: sharing your histories with others so they can review them.

{% snippet faqs/galaxy/histories_sharing.md %}

# Conclusion

tools_json:
["FASTQC", "toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0", "Porechop", "toolshed.g2.bx.psu.edu/repos/iuc/porechop/porechop/0.2.4+galaxy0", "fastp", "toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.23.2+galaxy0", "Kraken2", "toolshed.g2.bx.psu.edu/repos/iuc/kraken2/kraken2/2.1.1+galaxy1", "Filter", "Filter1", "Krakentools: Convert kraken report file", "toolshed.g2.bx.psu.edu/repos/iuc/krakentools_kreport2krona/krakentools_kreport2krona/1.2+galaxy0", "Krona pie chart", "toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1+galaxy0"]


topic: microbiome
tutorial: lotus2-identifying-fungi
title: Identifying Mycorrhizal Fungi from ITS2 sequencing using LotuS2
difficulty: intermediate
time_estimation: 3H
tools_json: ["LotuS2", "toolshed.g2.bx.psu.edu/repos/earlhaminst/lotus2/lotus2/2.32+galaxy0"]
requirements_json: []
tags_json: ["fungi", "ecology"]
content_hash: daba4cb90030d92e65a6735f17ab4cc0
last_modified: 2026-05-15T22:28:13.979799
zenodo_link: https://zenodo.org/records/13710591
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/lotus2-identifying-fungi/tutorial.html

content:
tutorial-example-run
    date: 2024-09-20
---

# Introduction

This tutorial is for you if you are a mycologist using metabarcoding data to understand the fungal composition of soil samples. In particular, this tutorial and its downstream applications will be of most interest to mycorrhizal scientists. [{SPUN}](https://www.spun.earth) is a research and advocacy organisation that uses Illumina sequencing of environmental {eDNA} from soil samples to identify mycorrhizal fungi. At {SPUN}, we use both {ITS2} and {SSU} amplicon sequencing to identify ectomycorrhizal and arbuscular mycorrhizal fungi, respectively. This tutorial focuses on the {ITS2} sequencing used to identify {MF}.

We do this by using a specific primer pair that amplifies just the {ITS2} region of the ribosomal DNA present in a soil sample. This tutorial covers data for an {ITS2} amplicon produced using the ITS3F and ITS4R primer set and sequenced using Illumina sequencing, which generates paired-end data. In this case, the example data has been generated using a NextSeq 2000, using 2x300bp chemistry.

This tutorial concentrates on the bioinformatics part of the process, i.e. the steps we need to do **after** we get data back from the sequencer. Starting from some example FASTQ files, we demonstrate how to upload data, run LotuS2, and examine the output files. We end with a brief description of what we can do with these output files.

We use LotuS2 at {SPUN} because we have found that it performs best out of all the tools we tried for identifying {MF} from sequencing data. The LotuS2 publication also includes a benchmark where it outperformed most amplicon tools using default parameters (see {% cite ozkurt2022 %}).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Pre-requisites for this tutorial

If you have never used Galaxy before, we highly recommend doing their [interactive tour](https://usegalaxy.eu/tours/core.galaxy_ui) first (takes a few minutes). Or, if you have more time (1h 30 min recommended), you can do the [Galaxy Basics for Everyone]({% link topics/introduction/tutorials/galaxy-intro-101-everyone/tutorial.md %}) tutorial.

We recommend signing up for a user account at [https://usegalaxy.eu](https://usegalaxy.eu). Although other UseGalaxy servers are available (see the [first part of the Galaxy Basics tutorial]({% link topics/introduction/tutorials/galaxy-intro-101-everyone/tutorial.md %})), LotuS2 is currently fully functional only on this Galaxy server.

> <hands-on-title>Launch galaxy</hands-on-title>
>
> 1. Open your favorite browser (preferably Chrome/Chromium or Firefox. Internet Explorer/Edge and Safari sometimes have problems!)
> 2. Browse to [https://usegalaxy.eu](https://usegalaxy.eu)
> 3. We recommend keeping this tutorial open in a separate browser window side-by-side if you have space on your desktop. If you only have one screen, you can also launch tutorial mode on the UseGalaxy server.
>
> {% snippet faqs/galaxy/tutorial_mode.md %}
>
{: .hands_on}

# Understanding the files needed for running LotuS2

At {SPUN}, we run LotuS2 to identify mycorrhizal fungi in a set of samples, using the following input files:

1. DNA sequence files for each sample in **gzipped FASTQ** format from the Illumina MiSeq sequencer
2. A mapping file in a **tab-separated-values** format which specifies which FASTQ files correspond to which samples. The file can also provide additional metadata for each sample, such as the primers used for each sample, the location where the sample was collected, and so on.
3. A {sdm} file in **text** format which specifies how the sequence FASTQ files should be quality filtered and demultiplexed.

For this tutorial, we have already provided a few example files at {{ page.zenodo_link }}. You can click on this link to see which files are available (there is no need to download them).

These files correspond to the three types of input files above, as shown:

- DNA sequence files:
    1. C_ITS2_S160_R1_001.fastq.gz
    1. C_ITS2_S160_R2_001.fastq.gz
    1. N5_ITS2_S140_R1_001.fastq.gz
    1. N5_ITS2_S140_R2_001.fastq.gz
    1. Pcov3_ITS2_S151_R1_001.fastq.gz
    1. Pcov3_ITS2_S151_R2_001.fastq.gz
- A mapping file:
    1. Colombia_ITS2_Mapping.tsv
- An {sdm} file
    1. sdm_miSeq_ITS.txt

For this tutorial, you do not need to download these files. Galaxy allows you to fetch data from a remote location directly into a tool or workflow without first downloading the files.

In the next section, we will get the data into Galaxy, and after that, we will look at the files to see what they look like.

# Get Data

This section describes two options that will allow you to download and use {SPUN}'s example data files. Select one option and follow the instructions.

> <hands-on-title>Option 1: Data upload - Add to history</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the 6 sequencing read files, 1 mapping file, and 1 sdm parameters file from [Zenodo]({{ page.zenodo_link }}) by pasting the following URLs into the upload window under "Paste/Fetch Data"
>
>    ```
>    {{ page.zenodo_link }}/files/C_ITS2_S160_R1_001.fastq.gz
>    {{ page.zenodo_link }}/files/C_ITS2_S160_R2_001.fastq.gz
>    {{ page.zenodo_link }}/files/N5_ITS2_S140_R1_001.fastq.gz
>    {{ page.zenodo_link }}/files/N5_ITS2_S140_R2_001.fastq.gz
>    {{ page.zenodo_link }}/files/Pcov3_ITS2_S151_R1_001.fastq.gz
>    {{ page.zenodo_link }}/files/Pcov3_ITS2_S151_R2_001.fastq.gz
>    {{ page.zenodo_link }}/files/Colombia_ITS2_Mapping.tsv
>    {{ page.zenodo_link }}/files/sdm_miSeq_ITS.txt
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. The history should be visible in a pane on the right of your window called ‘History’. Here, you can see the files and rename them.
> 4. **Rename** {% icon galaxy-pencil %} the history to your name of choice. This should be something that helps you to remember the project, such as "SPUN Colombia 24"
>
{: .hands_on}

> <hands-on-title>Option 2: Data upload - Upload from your own computer</hands-on-title>
>
> 1. This option will take more time, and should not be used if you have a slow internet connection.
> 2. However, this option will most closely mimic how you will use this tool in your real analysis, i.e. showing you how to upload data from your own computer
> 3. First, download all 8 files from [Zenodo]({{ page.zenodo_link }}) to your local computer. You can do this by scrolling down to the "Files" list section and clicking the "Download All" link at the top of this section. This will download a 55 {MB} zip file to your local computer
> 4. Unzip the file - the folder should have 8 files in it.
> 5. Create a new history in Galaxy
> 6. Upload these 8 files to the new history
>
>    {% snippet faqs/galaxy/datasets_upload.md %}
>
{: .hands_on}

## Examine the input files

We will briefly look at each type of file to see that it has been uploaded correctly. This doesn’t need to be done for every file each time you use LotuS2 if you have a large number of files, but it is good practice to check some files to see that they have been uploaded in the correct format.

> <hands-on-title>Inspect the FASTQ file</hands-on-title>
>
> * Click on the file name of one of the fastq.gz files in the history
>    - The filename should expand to show you some information such as the size of the file, and the type of the file
> * Click on the {% icon galaxy-eye %} (eye) icon next to a fastq.gz file in the history
> * You should see 4 lines for each sequence:
>     1. A header beginning with `>` followed by sequence identifiers
>     2. The DNA sequence of the read, made up of ATGCN
>     3. A line with just `+` on it (indicates that the next line has sequence quality values)
>     4. A line with sequence quality values for each nucleotide, in ASCII format
>
> You do not need to know more about FASTQ files for this tutorial, or about sequence quality in ASCII format, but if you want to learn more, you can do the [Sequence Analysis: Quality Control tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).
>
{: .hands_on}


> <hands-on-title>Inspect the mapping {TSV} file</hands-on-title>
>
> * Click on the {% icon galaxy-eye %} (eye) icon next to *Colombia_ITS2_Mapping.tsv* in your history
> * Each row is a sample
> * The columns tell you information about each sample:
>    - #SampleID is a short name/ID for the sample. Here, "C" stands for "Control" sample, and the other two are called samples "N5" and "Pcov3"
>    - fastqFile lists the sequence file names that correspond to each sample. We have two FASTQ files for paired-end sequencing: first, the forward file is listed, followed by a "," (comma), and then the reverse file name is listed, with no spaces between these.
>    - ForwardPrimer: Lists the forward primer sequence used during PCR to produce the sequenced amplicon (ITS3 forward primer here).
>    - ReversePrimer: Lists the forward primer sequence used during PCR to produce the sequenced amplicon (ITS4 reverse primer here).
> * The remaining columns (Latitude, Longitude, Country, Vegetation, etc) are metadata for each sample. LotuS2 does not **need** these columns, but it will copy them to the final R Phyloseq object that it creates. These columns will be needed for doing ecological analyses later. You can add metadata to the phyloseq object that is created through LotuS2 later in R, as long as the sample ID in your metadata sheet matches the sample ID in your mapping file.
>
{: .hands_on}

> <comment-title>Note</comment-title>
> 
> - In this case, the ForwardPrimer and ReversePrimer are the same across all samples. This is typical as we usually sequence the same region in all samples using the same primer pair.
> - LotuS2 allows you to specify the primer set as a parameter, and if this is done, the primer sequence does not need to be written in the mapping file. In this case, we are including primer sequences in the mapping file. When using different primer sets, if you are following these instructions for your data, you may need to change the primer sequences to reflect the primer set used for your sequencing.
> 
{: .comment}

> <hands-on-title>Inspect the {sdm} options file</hands-on-title>
>
> * Click on the {% icon galaxy-eye %} (eye) icon next to *sdm_miSeq_ITS.txt* in your history
> * This file specifies some of the parameters needed for processing ITS data with Illumina MiSeq paired-end sequencing using the software 'SDM' used by LotuS2. It can be also used for our files that were generated using an Illlumina NextSeq 2000.
> * The sdm options file is not needed if you are using default sdm settings, because LotuS2 provides an internal default version of the file. But in this case, specific parameters are required for ITS sequencing, so we provide this file.
>
{: .hands_on}

> <question-title>Check your understanding of the inputs</question-title>
>
> 1. What is the reverse primer for sample "N5"?
> 2. What is the DNA concentration in ng per µl for sample "Pcov3"?
> 3. When LotuS2 processes ITS paired-end sequences using these inputs, what is the minimum sequence length (minSeqLength) it will consider, below which it will discard the sequence?
> 4. Which of the 6 FASTQ files has the most data? (i.e., is the biggest in size?)
>
> > <solution-title></solution-title>
> >
> >1. In *Colombia_ITS2_Mapping.tsv*: **TCCTCCGCTTATTGATATGC**
> >2. In *Colombia_ITS2_Mapping.tsv*, under column *DNA_concentration_ng_ul*: **20.4**
> >3. In *sdm_miSeq_ITS.txt*, next to *minSeqLength*: **110**
> >4. Click on the names of each of the fastq.gz files in the history. The biggest is N5_ITS2_S140_R2_001.fastq.gz: **19.0 MB**
> >
> {: .solution}
>
{: .question}


## Create a "list of pairs" from the fastq.gz files

- This is a very important step when you first upload the data, because the LotuS2 tool needs our fastq.gz files in pairs.
- Galaxy has a very handy feature that automatically detects pairs of files from the filenames

> <hands-on-title>Create pairs of fastq files</hands-on-title>
>
> 1. Select ONLY the 6 fastq.gz files in your history (do not select `sdm_miSeq_ITS.txt` and `Colombia_ITS2_Mapping.tsv`)
> 2. Choose the "Build List of Dataset Pairs" option for these 6 files! [Screenshot showing how to select 6 out of 8 items, and then click the top right dropdown to choose the "Build List of Dataset Pairs" option](images/history-build-list-of-dataset-pairs.png)
> 3. Galaxy will automatically pair the files for you on the next screen
> 4. Rename the new collection as "Colombia ITS2 FASTQ pairs"
> 5. By default these two options are checked: "Hide original elements" and "Remove file extensions". You can leave them checked or unchecked. If you leave them checked, the 6 individual fastq.gz files will disappear from your history and be replaced by one collection with 3 pairs of FASTQ files with the new name "Colombia ITS2 FASTQ pairs"
>
> {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
{: .hands_on}

# Run LotuS2

When we run the LotuS2 tool on our data, it runs many steps in the background:

1. demultiplexing and filtering raw FASTQ sequences
2. denoising, removing chimeric sequences and clustering sequences into very high quality {OTU}s/{ASV}s
3. determining the taxonomic origin of each OTU using specialized and general-purpose databases and statistical algorithms
4. constructing OTU, genus, family, class, order and phylum abundance tables in .txt or .biom format
5. reconstructing the OTU phylogenetic tree
6. generating phyloseq objects for downstream analysis

The figure below with an overview of LotuS2 is taken from {% cite ozkurt2022 %}.

![Flowchart image showing the different steps done by LotuS2](https://media.springernature.com/full/springer-static/image/art%3A10.1186%2Fs40168-022-01365-1/MediaObjects/40168_2022_1365_Fig1_HTML.png "Workflow of the LotuS2 pipeline")

As LotuS2 is a very powerful, general-purpose tool used in many metabarcoding projects for bacteria, fungi, and eukaryotes, it provides many different parameters (options for running the software) specified for each special use case.

In the next subsection, we show how to run LotuS2 in Galaxy and how to set the parameters needed for a fungal dataset.

## Run LotuS2 with the example fungal dataset

> <hands-on-title>Run LotuS2</hands-on-title>
>
> In the panel on the left, select "Tools", search for "LotuS2" and then run the tool using the parameters below. Leave the rest of the parameters at their default settings. 
>
> Make sure you have the right version (2.32+galaxy0). You can check the version by clicking the {% icon tool-versions %} (blocks) icon.
>
> 1. {% tool [LotuS2](toolshed.g2.bx.psu.edu/repos/earlhaminst/lotus2/lotus2/2.32+galaxy0) %} with the following parameters:
>    - *"Single- or Paired-end data?"*: `Paired-end list`
>        - In *"List of paired reads"*: choose the paired-list you created in the previous section: `Colombia ITS2 FASTQ pairs` (or whatever name you gave to the collection)
>    - In *"Mapping file (optional)"*: `Colombia_ITS2_mapping.tsv`
>    - Forward (and Reverse) Primer: Leave blank
>    - *"Clustering algorithm"*: `VSEARCH`
>    - *"Taxonomy aligner for taxonomic profiling of OTUs"*: `Lambda, LCA against custom reference database`
>        - *"Taxonomy reference database"*: `Use a built-in taxonomy database`
>            - *"Using reference database"*: `ITS fungi specific (UNITE)`
>    - In *"Other Clustering Options"*:
>        - *"Minimum size of dereplicated raw reads (optional)"*: `10:1,5:2,3:3`
>    - In *"Other Taxonomy Options"*:
>        - *"Amplicon type"*: `ITS2`
>        - *"Tax group"*: `fungal 18S/23S/ITS annotation`
>    - In *"Other options"*:
>        - *"SDM option file"*: `sdm_miSeq_ITS.txt`
>
> 2. Once all the parameters above are entered, click the **"Run Tool"** button at the end.
> 3. You should see a green box on the next page saying "Started tool LotuS2 and successfully added 1 job to the queue". The box lists the 6 outputs that the tool produces.
> 4. This step can take 5-10 minutes to run or longer depending on how many other jobs are running on the Galaxy server. While we are waiting for it to finish, you can do the next step on "Creating your own mapping.tsv" file
> 5. You will know when the tool has finished because all the outputs in the history will turn green.
> 
> 
>  > <comment-title>Notes</comment-title>
>  >
>  > - Remember to choose _Paired-end list_ in the sequencing read data section. Galaxy will pick up the Paired-end list available in the History, which will have the name you gave it in the _Create a list of pairs_ step
>  > - In *"Forward (and Reverse) Primer"*: Leave blank, as we have already provided them in the mapping.tsv file. If you are carrying out an analysis using your own data, you may add here the primer sequences used in your analyses instead of in the mapping file, if you prefer.
>  > - In *"Other Clustering Options"*: *"Minimum size of dereplicated raw reads (optional)"*: we put `10:1,5:2,3:3`. Each "X:Y" pair means "A unique dereplicated read must be seen at least X times in at least Y samples". So, if a sequence read is only found in 1 sample, it must be present in 10 copies. If a sequence read is found in only 2 samples, it must be found in 5 copies in each, etc. This is a strict parameter, so that sequence errors are not taken as novel biological sequences, reducing the occurrences of false positive OTUs or ASVs. You may want to use more lenient values if you know your data has a high number of real singletons, i.e., sequences that occur only once.
>  >
>  {: .comment}
>
{: .hands_on}

> <question-title>Examine the parameter options</question-title>
>
> 1. What other clustering algorithms can you use inside LotuS2?
> 2. What other amplicon types can LotuS2 classify?
>
> > <solution-title></solution-title>
> >
> >1. In the "LotuS2" tool's parameter page, under *"Clustering Algorithms"*, click on the drop down. You should see these tools:
> >    - DADA2
> >    - swarm
> >    - CD-HIT
> >    - VSEARCH
> >
> >    SPUN uses VSEARCH for our {ITS2} amplicon for the identification of fungi from soils.
> >2. In *"Other Taxonomy Options"*, click on the drop-down under *"Amplicon type"*:
> >    - Default
> >    - LSU
> >    - SSU
> >    - ITS
> >    - ITS1
> >    - ITS2
> >
> >    Here, we were using general fungal primers that amplify the {ITS2} region. In another tutorial, we will see how to use the {SSU} region for specific amplification of {AMF}
> >
> {: .solution}
>
{: .question}

# Examine the outputs

The LotuS2 Galaxy tool creates 6 output files that you should see in your history, with names like this (the exact number after each data might be different):
1. LotuS2 on data 8, data 6, and others: main log file
2. LotuS2 on data 8, data 6, and others: mapping file
3. LotuS2 on data 8, data 6, and others: Newick-formatted phylogenetic trees between sequences
4. LotuS2 on data 8, data 6, and others: FASTA-formatted extended OTU seed sequences
5. LotuS2 on data 8, data 6, and others: OTU abundance matrix
6. LotuS2 on data 8, data 6, and others: Complete LotuS2 output

> <hands-on-title>Examine outputs</hands-on-title>
> Click on the little {% icon galaxy-eye %} (eye) icon next to each output in the history to examine it. An explanation of each file is given below.
>
> 1. **main log file**: If we had run LotuS on the command line, the contents of the `output/LotuSLogS/LotuS_run.log` would be this main LotuS2 run log file.
> 
>     It has information on all the parameters passed to the tool, and information on the start time of each step in the program, plus the output of each step.
>   
>     You should always check this file first to see if LotuS2 completed correctly, and approximately how many reads were used/classified. If this number is much lower than what you expected, that might indicate a problem with the run and the parameters.
> 
> 2. **mapping file**: You can ignore this file as it is a repeat of the mapping TSV file (tab-separated-values) that we used as an input.
> 
> 3. **Newick-formatted phylogenetic trees between sequences**: A phylogenetic tree created from the ITS2 sequences. If you want to see what the tree looks like, you can copy the contents of this file and paste it at this [online tree viewer](http://etetoolkit.org/)treeview/
> 
> 4. **FASTA-formatted extended OTU seed sequences**: OTU sequences created by the LotuS2 program after clustering near-identical reads
> 
> 5. **OTU abundance matrix**: A tabular file with rows and columns with tab-separated-values. Each row is an OTU. The first column has the OTU name, and the remaining columns have the OTU abundance (i.e. how many reads were seen for that OTU) in each sample
> 
> 6. **Complete LotuS2 output**: If you try to view this file you will see some unreadable binary characters on the screen. That's because this is a zip file with the complete LotuS2 output folder in one zip folder. You should download this zip file and unzip it on your local computer if you want to see everything inside this folder. ![screenshot of how to download the zip file](images/history-download-zip.png)
> 
>     One of the most useful files in this output folder is the `phyloseq.Rdata` file which you can load in R to do further ecological analysis. This is beyond the scope of the current tutorial, but we will cover this topic in future tutorials. If you want to get started on your own, you can look at the last part "Exploration with phyloseq" in the [Building an amplicon sequence variant (ASV) table from 16S data using DADA2]({% link topics/microbiome/tutorials/dada-16S/tutorial.md %}) tutorial.
{: .hands_on}

> <question-title>Test your understanding of the outputs</question-title>
>
> If for some reason, your LotuS2 run did not complete, you can look at this [example LotuS2 run](https://usegalaxy.eu/u/sujai_spun_earth/h/identifying-mf-from-its2-sequencing-using-lotus2---tutorial-example-run) that we generated using the steps above.
>
> 1. How many total reads were in the OTU abundance matrix? (hint: main log file)
> 2. What percentage of reads are assigned at the phylum level and at the genus level? (hint: main log file)
> 3. How long did the entire LotuS2 pipeline take to run? (hint: the main log has time stamps at the left of each step in the format hh:mm:ss)
> 4. Which OTU was most abundant in all samples? (hint: OTU abundance matrix). How many reads were present in each sample?
> 5. What was the sequence of the least abundant OTU? (hint: FASTA-formatted extended OTU seed sequences)  
>
> > <solution-title>Answers</solution-title>
> >
> > For the first 3 questions, scroll to near the bottom of the *"main log file"*. Under "Calculating Taxonomic Abundance Tables" the answers in our [example LotuS2 run](https://usegalaxy.eu/u/sujai_spun_earth/h/identifying-mf-from-its2-sequencing-using-lotus2---tutorial-example-run) were:
> >
> > 1. Total reads in matrix: 132123
> > 2. Phylum	95%, Genus	65%
> > 3. 00:02:34 (i.e. 2 minutes 34 seconds)
> > 4. In the *"OTU abundance matrix"*, the OTUs are arranged in order of greatest-to-least abundance so OTU1 was the most abundant with 0 reads in sample **C** (Control), 25,659 reads in sample **N5**, and 0 reads in sample **Pcov3**
> > 5. In the *"OTU abundance matrix"*, the two least abundant sequences with 6 reads total each were OTU256 and OTU257. We can look up the sequences of these files in the *"FASTA-formatted extended OTU seed sequences"* at the bottom of the file:
> >
> > ```
> > >OTU256
> > >OTU257
> > ```
> >
> {: .solution}
>
{: .question}


# Run LotuS2 on your own data

To run the LotuS2 workflow on your own fungal data, you only need the sequencing FASTQ files and some information about what primers were used.

In the example dataset, we used the ITS3/ITS4 primer pair to sequence the ITS2 region, so our Forward primer was `GCATCGATGAAGAACGCAGC` and the Reverse primer was `TCCTCCGCTTATTGATATGC`

When you want to process your own sequencing files, you can specify the Forward and Reverse primer in the Galaxy LotuS2 Parameters directly in the options if all your samples have the same forward and reverse primers:
  - *"Forward primer used to amplify DNA region (optional) - optional"*: `GCATCGATGAAGAACGCAGC`
  - *"Reverse primer used to amplify DNA region (optional) - optional"*: `TCCTCCGCTTATTGATATGC`

We advise adding metadata to the phyloseq object after running LotuS2, so that a large mapping file with all metadata does not have to be created. We will have future tutorials on adding metadata from a .csv file to a phyloseq object in R.

## Create your own mapping tsv file

Mapping files for LotuS2 have to be in tab-separated-values format. You can create this file in any spreadsheet software, eg: Microsoft Excel or Google Sheets. The important thing to note is that when you save or export the file, you should select the TSV format (called "Tab separated values" in Google Sheets or "Tab-delimited text .txt" in Excel)

LotuS2 allows many columns in the mapping tsv file according to the [specification](https://lotus2.earlham.ac.uk//main.php?site=documentation#mappingfile). However, for simplicity, we recommend this format for sequencing data using the SPUN protocol:

- Only 1 header row that begins with `#SampleID`
- One row per sample
- The only essential columns are `#SampleID`, `fastqFile`
- If you have the same ForwardPrimer and ReversePrimer in all your samples, then you can provide them in the LotuS2 Galaxy tool and delete the ForwardPrimer and ReversePrimer columns
- If you have any additional metadata variables that you want to analyse downstream using R, then you can provide them as columns
- We recommend using "SequencingRun" and labelling with the name of the sequencing run, which is arbitrary. Something like "Scripps run 1". This is because if samples were run across multiple different sequencing runs it is important for LotuS2 to factor this into analyses. If samples were all run on the same run, write the same for every sample.
- In the data files for this tutorial, and in most cases, the sequencing provider will separate the data from separate samples into separate pairs of files. This is known as *demultiplexing*. If all your samples are combined in one pair of files, then you will need to provide the demultiplexing barcodes in the mapping tsv file.

For more information, see the [LotuS2 Mapping File Format Documentation](https://lotus2.earlham.ac.uk/main.php?site=documentation#mappingfile)

In the exercise below you will create your own mapping tsv file for a new run.

> <hands-on-title>Create and use your own mapping tsv file</hands-on-title>
>
> Steps:
>
> 1. Import history from the [mapping example history](https://usegalaxy.eu/u/sujai_spun_earth/h/identifying-mf-from-its2-sequencing-using-lotus2--create-mapping)
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
> 2. Create a table in your favourite spreadsheet software (eg *Google Sheets* or *Microsoft Excel*) with the following columns (you can copy and paste from the example below. We have filled in the first two columns for you #SampleID and fastqFile):
>
>    | #SampleID | fastqFile | ForwardPrimer | ReversePrimer | Vegetation |
>    | N11 | N11_ITS2_S134_R1_001.fastq.gz,N11_ITS2_S134_R2_001.fastq.gz | | | |
>    | N16 | N16_ITS2_S138_R1_001.fastq.gz,N16_ITS2_S138_R2_001.fastq.gz | | | |
>
> 3. Complete the table:
>   - Add the correct ForwardPrimer `GCATCGATGAAGAACGCAGC` and the correct ReversePrimer `TCCTCCGCTTATTGATATGC` to each row
>   - Add the Vegetation `Trigonobalanus excelsa forest` to each row
>
> 4. Export or Save the spreadsheet as a new file in tab-separated-values format (Google Sheets: _File_: _Download_: _Tab separated values (.tsv)_; Excel: _File_: _Save as_: choose Tab delimited text in the format)
>
> 5. Name the file something like "mapping.tsv" or "mapping.txt"
>
> 6. Upload this file to your history as a local file
>
>    {% snippet faqs/galaxy/datasets_upload.md %}
>
> 7. Now you should have all the files you need in this new history to run LotuS2:
>    - 4 paired-end FASTQ files for 2 samples
>    - 1 mapping tsv file specifying sample IDs, fastqFiles, primers, and metadata for these 2 samples
>    - 1 sdm_miSeq_ITS.txt
>
> 8. Rerun LotuS2 following the [steps above](#run-lotus2)
>
> 9. If your LotuS2 run finishes - then CONGRATULATIONS! You now know how to create your own mapping tsv file and run LotuS2 on a new set of fastq.gz files
>
{: .hands_on}

# Conclusion

- LotuS2 is a powerful tool that can be used to identify species present in a DNA sequencing sample
- You have learnt how to run LotuS2 on example data
- You have learnt how to create a mapping tsv file for a new set of data


topic: microbiome
tutorial: mgnify-amplicon
title: MGnify v5.0 Amplicon Pipeline
difficulty: introductory
time_estimation: 2H
requirements_json: []
tags_json: []
content_hash: 3db490aeb379e2ea398da257c5fb294d
last_modified: 2026-05-27T09:01:49.371380
zenodo_link: https://zenodo.org/records/13710235
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/mgnify-amplicon/tutorial.html

content:
Amplicon sequencing is a powerful research method designed to explore and reveal the taxonomic composition of diverse microbial communities. By focusing on specific genetic regions, researchers can gain insights into the structure and function of these communities. Typically, ribosomal ribonucleic acid (rRNA) or ribosomal deoxyribonucleic acid (rDNA) is sequenced, with particular attention given to key marker genes. For bacteria and archaea, the 16S rRNA gene is commonly targeted; for eukaryotes, the 18S rRNA gene is used, while fungi are often studied through internal transcribed spacers (ITS) regions. These marker genes provide the genetic signatures necessary for identifying and categorizing the various organisms present within a sample.<br>

In this tutorial, we will dive into the MGnify amplicon pipeline v5.0 on Galaxy, a sophisticated toolset for microbial analysis. This pipeline is a ported version of the same well-established amplicon pipeline used by MGnify, a leading metagenomics platform dedicated to the analysis and archiving of microbiome sequences. MGnify has seen significant adoption and growth in recent years ({%cite Richardson2022%}); as of the time of writing, over 500k analyses have been conducted using this pipeline for amplicon data alone ({%cite ebi_metagenomics%}). One of the key advantages of this workflow is that it does not require users to know their target region in advance—it automatically processes all amplicons for ITS, SSU, and LSU, and can even handle mixed amplicon datasets. This workflow does not currently generate ASVs (Amplicon Sequence Variants). If ASV-level resolution is required, we recommend reading the [DADA2 tutorial]({% link topics/microbiome/tutorials/dada-16S/tutorial.md %}) or utilizing {% tool [LotuS2](toolshed.g2.bx.psu.edu/repos/earlhaminst/lotus2/lotus2/2.32+galaxy0) %}.

![Graphical overview of the full pipeline](./images/pipeline_v5.0_amplicon.png "<strong>The workflow of the MGnify amplicon pipeline v5.0</strong> ({%cite ebi_amplicon_pipeline%}). The figure illustrates the workflow of the MGnify amplicon pipeline, beginning with quality control to filter low-quality reads. The processed reads then proceed to the rRNA-prediction step, where SSU and LSU regions are classified and their taxonomic abundance is visualized. Following this, the pipeline handles the ITS regions, classifying them and visualizing their taxonomic abundance using pie charts. Each stage is clearly outlined, showcasing how data moves through the pipeline and is analyzed to provide comprehensive insights into microbial communities."){: style="width:45%; align-self:center"}

Our journey through this tutorial will provide a comprehensive understanding of the different sub-workflows that make up the pipeline and how each contributes to the overall analysis process. We will explore how these workflows are executed within the Galaxy platform. Additionally, we will explore which databases and tools are integrated into the pipeline, giving us a clearer picture of the resources that power these analyses. By the end of this tutorial you will have the knowledge to apply the MGnify amplicon pipeline effectively in your own research.<br>

Before diving into the tutorial, it is strongly recommended to review the Introduction to Microbiome Analysis slides or watch the corresponding video, which are linked in the requirements section above, especially if you are new to this field. These slides will provide essential background knowledge and help you better understand the concepts covered in this tutorial.

> <warning-title>Slight discrepancy between runs</warning-title>
> The outputs generated by this workflow may vary slightly between runs due to the non-deterministic behavior of the MAPseq version used. While this variation does not always occur, certain processing steps involve elements of randomness that can lead to minor differences in the results on different executions.
>
> ![Beta diversity benchmark](./images/beta_div_plot_mapseq2.1.1.png "<b>Beta diversity-based benchmark</b>. Dissimilarity between MAPseq runs. Executed 10 times using identical configuration and the same sample from analysis MGYA00578954 to demonstrate MAPseq inconsistency. The term ’All’ refers to all ranks combined. Species rank was excluded, due to taxa absence at this rank. Bars represent the average dissimilarity values, while individual data points represent the values for each of the ten different runs.")
>
{: .warning}

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

In this tutorial, we'll cover four different versions:
1. Executing the amplicon workflow using ENA data (default)
2. Executing the amplicon workflow using own reads, which requires some minor adjustments
2. Executing each subworkflow individually using ENA data
2. Executing each subworkflow individually using own reads

{% include _includes/cyoa-choices.html option1="Executing the full workflow using ENA data" option2="Executing subworkflows using ENA data" option3="Executing the full workflow using own data" option4="Executing subworkflows using own data" default="Executing the full workflow using ENA data" %}


# Import datasets

Let's begin with creating a history and giving it a suitable name.

> <hands-on-title>Create history</hands-on-title>
>
> 1. Create a new history
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the created history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}



<div class="Executing-the-full-workflow-using-ENA-data" markdown="1">

> <hands-on-title>Downloading datasets</hands-on-title>
> 1. Click on {% icon galaxy-upload %} **Upload Data** at the top of left panel
>    - Select {% icon galaxy-wf-edit %} **Paste/Fetch Data** at the bottom
>    - Copy {% icon copy %} and paste the following URLs into the text box:
>    - Click on **Start**
>
>      ```
>      https://zenodo.org/records/13710235/files/ribo.claninfo
>      https://zenodo.org/records/13710235/files/accessions.csv
>      ```
>
>    - **Close** the window
>    - **Note**: The file `ribo.claninfo` is required for the workflow to run properly. `ribo.claninfo` provides classification information for the rRNA models in ribo.cm, grouping them into so called clans.
{: .hands_on}

</div>



<div class="Executing-subworkflows-using-ENA-data" markdown="1">

> <hands-on-title>Downloading datasets</hands-on-title>
> 1. Click on {% icon galaxy-upload %} **Upload Data** at the top of left panel
>    - Select {% icon galaxy-wf-edit %} **Paste/Fetch Data** at the bottom
>    - Copy {% icon copy %} and paste the following URLs into the text box:
>    - Click on **Start**
>
>      ```
>      https://zenodo.org/records/13710235/files/ribo.claninfo
>      https://zenodo.org/records/13710235/files/accessions.csv
>      ```
>
>    - **Close** the window
>    - **Note**: The file `ribo.claninfo` is required for the workflow to run properly. `ribo.claninfo` provides classification information for the rRNA models in ribo.cm, grouping them into so called clans.
>
> 2. {% tool [fastq-dl](toolshed.g2.bx.psu.edu/repos/iuc/fastq_dl/fastq_dl/3.0.0+galaxy0) %} using the parameters:
>    - {% icon param-files %} *"select input type"*: `A list of ENA accession IDs, one per row`
>    - {% icon param-files %} *"Accession"*: `accessions.csv`
{: .hands_on}

</div>

<div class="Executing-subworkflows-using-own-data" markdown="1">

> <hands-on-title>Downloading datasets</hands-on-title>
>
> 1. Click on {% icon galaxy-upload %} **Upload Data** at the top of left panel
>    - Select {% icon galaxy-wf-edit %} **Paste/Fetch Data** at the bottom
>    - Copy {% icon copy %} and paste the following URLs into the text box:
>    - Click on **Start**
>
>      ```
>      https://zenodo.org/records/13710235/files/ribo.claninfo
>      ```
>
>   - **Close** the window
>   - **Note**: The file `ribo.claninfo` is required for the workflow to run properly. `ribo.claninfo` provides classification information for the rRNA models in ribo.cm, grouping them into so called clans.
>
{: .hands_on}

</div>



<div class="Executing-the-full-workflow-using-own-data" markdown="1">

> <hands-on-title>Downloading datasets</hands-on-title>
> 1. Click on {% icon galaxy-upload %} **Upload Data** at the top of left panel
>    - Select {% icon galaxy-wf-edit %} **Paste/Fetch Data** at the bottom
>    - Copy {% icon copy %} and paste the following URLs into the text box:
>    - Click on **Start**
>
>      ```
>      https://zenodo.org/records/13710235/files/ribo.claninfo
>      ```
>
>   - **Close** the window
>   - **Note**: The file `ribo.claninfo` is required for the workflow to run properly. `ribo.claninfo` provides classification information for the rRNA models in ribo.cm, grouping them into so called clans.
{: .hands_on}

</div>


<div class="Executing-the-full-workflow-using-own-data" markdown="1">
# Upload own data files

> <hands-on-title>Upload own data file</hands-on-title>
>
> 1. Upload the reads in `fastqsanger` or `fastqsanger.gz` format
>
>    {% snippet faqs/galaxy/datasets_upload.md %}
>
> 2. Create a collection out of the single-end datasets
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
> 3. Create a paired-end end collection for the paired-end datasets
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
{: .hands_on}

</div>

# Workflow Selection Guidance
Before running the rRNA or ITS sub-workflows, it's important to know the nature of your amplicon data:
- If your sequencing targets 16S or 18S rRNA genes, use the rRNA-prediction workflow.
- If your sequencing targets fungal ITS regions, use the ITS workflow.
- You do not need to run both workflows unless your dataset includes both types of sequences.

# Quality control
Quality control (QC) is a crucial first step in any metagenomic analysis pipeline. It ensures that sequencing reads are free from contaminants, low-quality bases, and adapter sequences, which can negatively impact downstream analyses. In the MGnify v5.0 pipeline, QC involves trimming, filtering, and assessing read quality using tools like Trimmomatic, SeqPrep, and FastQC. This process guarantees that only high-quality data progresses, providing reliable results for subsequent steps like taxonomic classification and diversity analysis.<br>
The quality control phase in this amplicon pipeline is split into two sub-workflows: one dedicated to paired-end reads and another for single-end reads.
In addition to quality-controlled single- and paired-end reads, both subworkflows also generate a MultiQC report that summarizes quality metrics across all input files. To help you interpret the MultiQC report, we recommend reviewing the Quality Control tutorial linked in the requirements section.

## Single-end reads
The quality control sub-workflow for single-end reads involves a series of tools that perform trimming, filtering, and quality assessment to ensure high-quality sequencing data. The workflow includes the following tools:
- {% tool [Trimmomatic](toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.39+galaxy2) %} trims low quality sequencing regions. Using the following parameters:
    - *"SLIDING WINDOW - Number of bases to average across"*: `4`
    - *"SLIDING WINDOW - Average quality required"*: `15`
    - *"LEADING"*: `3`
    - *"TRAILING"*: `3`
    - *"MINLEN"*: `100`
    - *"Quality score encoding"*: `Phred33`
- {% tool [Filter FASTQ](toolshed.g2.bx.psu.edu/repos/devteam/fastq_filter/fastq_filter/1.1.5) %} filters sequences shorter than 100 nucleotides in length out
- {% tool [PRINSEQ](toolshed.g2.bx.psu.edu/repos/iuc/prinseq/prinseq/0.20.4+galaxy2) %} provides advanced filtering, trimming, and data quality control, allowing for the removal of duplicate reads, low-complexity sequences, and additional quality filtering. Using the following parameters:
    - *"Is this library paired- or single-end?"*: `Single-end`
    - *"Apply filter treatments?"*: `Yes`
    - *"Filter sequences based on their length?"*: `No`
    - *"Filter sequences based on quality score?"*: `No`
    - *"Filter sequences based on their base content?"*: `Yes`
    - *"Filter sequences based on their GC percentage?"*: `No`
    - *"Filter sequences based on their number of N bases?"*: `No`
    - *"Filter sequences based on their percentage of N bases?"*: `Yes`
    - *"Maximal N percentage threshold to conserve sequences"*: `10`
    - *"Filter sequences with characters other than A, T, C, G and N?"*: `Yes`
    - *"Filter sequences based on their complexity?"*: `No`
    - *"Apply trimming treatments?"*: `No`
- {% tool [convert FASTQ to FASTA](toolshed.g2.bx.psu.edu/repos/devteam/fastqtofasta/fastq_to_fasta_python/1.1.5) %} converts the quality control processed reads from FASTQ into FASTA format
- {% tool [FASTQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy0) %} performs comprehensive quality checks on sequencing data, providing visual summaries and detailed metrics for read quality
- {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy1) %} aggregates reports from multiple quality control FASTQC reports, into a single, consolidated report for easier interpretation. Using the following parameters:
    - **Which tool was used generate logs?**: `FastQC`
    - **Type of FastQC output?**: `Raw data`

<div class="Executing-subworkflows-using-ENA-data" markdown="1">
> <hands-on-title>Executing the single-end quality control sub-workflow</hands-on-title>
> 1. **Import** the [single-end quality control sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-quality-control-single-end.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Run** {% icon workflow-run %} the single-end sub-workflow using the following input:
>    - **Single-end reads**: `Single-end data` collection
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

<div class="Executing-subworkflows-using-own-data" markdown="1">
> <hands-on-title>Executing the single-end quality control sub-workflow</hands-on-title>
> 1. **Import** the [single-end quality control sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-quality-control-single-end.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. Click on {% icon galaxy-upload %} **Upload Data** at the top of left panel
> 3. Click on {% icon tutorial %} **Choose local file** and select the desired single-end files in `fastqsanger` or `fastqsanger.gz` format
> 4. Click on **Start** and **Close** the window
> 5. When the upload is finished create a collection for all single-end reads files:
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
> 5. **Run** {% icon workflow-run %} the single-end sub-workflow using the following input:
>    - **Single-end reads**: Single-end reads collection we just created
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

## Paired-end reads
The quality control sub-workflow for paired-end reads is designed to optimize the accuracy and reliability of sequencing data by addressing the unique challenges of handling paired-end sequences. The tools included are:
- {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.23.4+galaxy1) %} A versatile tool that performs quality filtering, trimming, adapter removal, and other preprocessing tasks specifically for paired-end reads. Using the following parameters:
    - *"Single-end or paired reads"*: `Paired`
    - *"Filter Options"*:
        - *"Quality filtering options"*:
            - *"Qualified quality phred"*: `20`
            - *"Unqualified percent limit"*: `20`
        - *"Length filtering options"*:
            - *"Length required"*: `70`
    - *"Read Modification Options"*:
        - *"Base correction by overlap analysis options"*:
            - *"Enable base correction"*: `Yes`
- {% tool [SeqPrep](toolshed.g2.bx.psu.edu/repos/iuc/seqprep/seqprep/1.3.2+galaxy0) %} Merges the forward and reverse reads. Using the default parameters.
- {% tool [Trimmomatic](toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.39+galaxy2) %} trims low quality sequencing regions. Using the following parameters:
    - *"SLIDING WINDOW - Number of bases to average across"*: `4`
    - *"SLIDING WINDOW - Average quality required"*: `15`
    - *"LEADING"*: `3`
    - *"TRAILING"*: `3`
    - *"MINLEN"*: `100`
    - *"Quality score encoding"*: `Phred33`
- {% tool [Filter FASTQ](toolshed.g2.bx.psu.edu/repos/devteam/fastq_filter/fastq_filter/1.1.5) %} filters sequences shorter than 100 nucleotides in length out
- {% tool [PRINSEQ](toolshed.g2.bx.psu.edu/repos/iuc/prinseq/prinseq/0.20.4+galaxy2) %} provides advanced filtering, trimming, and data quality control, allowing for the removal of duplicate reads, low-complexity sequences, and additional quality filtering. Using the following parameters:
    - *"Is this library paired- or single-end?"*: `Single-end`
    - *"Apply filter treatments?"*: `Yes`
    - *"Filter sequence based on their length?"*: `No`
    - *"Filter sequences based on quality score?"*: `No`
    - *"Filter sequences based on their base content?"*: `Yes`
    - *"Filter sequences based on their GC percentage?"*: `No`
    - *"Filter sequences based on their number of N bases?"*: `No`
    - *"Filter sequences based on their percentage of N bases?"*: `Yes`
    - *"Maximal N percentage threshold to conserve sequences"*: `10`
    - *"Filter sequences with characters other than A, T, C, G and N?"*: `Yes`
    - *"Filter sequences based on their complexity?"*: `No`
    - *"Apply trimming treatments?"*: `No`
- {% tool [convert FASTQ to FASTA](toolshed.g2.bx.psu.edu/repos/devteam/fastqtofasta/fastq_to_fasta_python/1.1.5) %} converts the quality control processed reads from FASTQ into FASTA format
- {% tool [FASTQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy0) %} performs comprehensive quality checks on sequencing data, providing visual summaries and detailed metrics for read quality
- {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy1) %} aggregates reports from multiple quality control FASTQC reports, into a single, consolidated report for easier interpretation. Using the following parameters:
    - *"Which tool was used generate logs?"*: `FastQC`
    - *"Type of FastQC output?"*: `Raw data`

<div class="Executing-subworkflows-using-ENA-data" markdown="1">
> <hands-on-title>Executing the paired-end quality control sub-workflow</hands-on-title>
> 1. **Import** the [paired-end quality control sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-quality-control-paired-end.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Run** {% icon workflow-run %} the paired-end sub-workflow using the following input:
>    - **Paired-end reads**: `Pair-end data` collection
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

<div class="Executing-subworkflows-using-own-data" markdown="1">
> <hands-on-title>Executing the paired-end quality control sub-workflow</hands-on-title>
> 1. **Import** the [paired-end quality control sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-quality-control-paired-end.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. Click on {% icon galaxy-upload %} **Upload Data** at the top of left panel
> 3. Click on {% icon tutorial %}**Choose local file** and select the desired paired-end files in `fastqsanger` or `fastqsanger.gz` format
> 4. Click on **Start** and **Close** the window
> 5. When the upload is finished create a paired collection for all paired-end reads files:
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
> 6. **Run** {% icon workflow-run %} the paired-end sub-workflow using the following input:
>    - **Paired-end reads**: Paired-end reads collection, which was created in the steps above
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

# rRNA-prediction
The rRNA prediction sub-workflow focuses on identifying and classifying ribosomal RNA (rRNA) sequences, particularly small subunit (SSU) and large subunit (LSU) rRNAs. This workflow uses several tools for comparison against covariance models, filtering, classification, and visualization:
- {% tool [CMseach](toolshed.g2.bx.psu.edu/repos/bgruening/infernal/infernal_cmsearch/1.1.4+galaxy0) %} the datasets are compared against the LSU and SSU models from Rfam (v13.0). Using the following parameters:
    - *"Subject covariance models"*: `Locally installed covariance models`
    - *"Covariance models"*: `Mgnify 5.0 RFAM Models`
    - *"Calculate E-values as if the search space size is 'x' megabases (Mb)"*: `1000`
    - *"Options controlling acceleration heuristics"*: `Use HMM only, don't use a CM at all (--hmmonly)`
    - *"Omit the alignment section from the main input"*: `Yes`
- {% tool [cmsearch-deoverlap](toolshed.g2.bx.psu.edu/repos/rnateam/cmsearch_deoverlap/cmsearch_deoverlap/0.08+galaxy2) %} removes overlapping reads that belong to the same clan. Using the default parameters.
- {% tool [bedtools getfasta](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_getfastabed/2.30.0+galaxy1) %} extract sequences from a FASTA file
- {% tool [MAPseq](toolshed.g2.bx.psu.edu/repos/iuc/mapseq/mapseq/2.1.1+galaxy0) %} classifies matching OTUs to the reads, used in conjunction with SILVA (v132) SSU and LSU databases. Using the following parameters:
    - *"Use cached database or database from history"*: `Cached database`
    - *"Using built-in mapseq DB**. MAPseq is used in conjunction with the following databases:
        1. `MGnify LSU (v5.0.7) - silva_lsu-20200130 downloaded at 2023-09-26`
        2. `MGnify SSU (v5.0.7) - silva_ssu-20200130 downloaded at 2023-09-26`
    - *"Create OTU table"*: `Yes`
    - *"Create taxon table for Krona"*: `Yes`
    - *"Top hits"*: `80`
    - *"Top OTUs"*: `40`
    - *"Output format"*: `simple`
- {% tool [Krona](toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1+galaxy0) %} generates pie charts demonstrating the taxonomic abundance. Using the following parameters:
    - *"What is the type of your input data"*: `Tabular`
    - *"Provide a name for the basal rank"*: `Root`
- {% tool [biom-convert](toolshed.g2.bx.psu.edu/repos/iuc/biom_convert/biom_convert/2.1.15+galaxy1) %} converts the taxonomic abundance tables to JSON and HDF5 formats. Using the following parameters:
    - *"Choose the source BIOM format"*: `Tabular File`
    - *"Process metadata associated with observations when converting"*: `taxonomy`
    - *"Choose the output type"*: `BIOM`
    - *"Table Type"*: `OTU table`
    - *"BIOM format type"*:
        1. `JSON-formatted table(BIOM1)`
        2. `HDF5-formatted table(BIOM2)`

<div class="Executing-subworkflows-using-ENA-data" markdown="1">

> <hands-on-title>Executing the rRNA-prediction sub-workflow</hands-on-title>
>
> 1. **Import** the [rRNA-prediction sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-rrna-prediction.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. Using {% tool [Merge collections](__MERGE_COLLECTION__) %} merge both quality processed single- and paired-end reads collections: `Single-end post quality control FASTA files` and `Paired-end post quality control FASTA files`
> 3. **Run** {% icon workflow-run %} the rRNA-prediction sub-workflow using the following inputs:
>    - **Processed sequences**: Quality processed sequences (merged collection)
>    - **Clan information file**: `ribo.claninfo`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

<div class="Executing-subworkflows-using-own-data" markdown="1">
> <hands-on-title>Executing the rRNA-prediction sub-workflow</hands-on-title>
> 1. **Import** the [rRNA-prediction sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-rrna-prediction.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
> 2. Using {% tool [Merge collections](__MERGE_COLLECTION__) %} merge both quality processed single- and paired-end reads collections: `Single-end post quality control FASTA files` and `Paired-end post quality control FASTA files`
> 3. **Run** {% icon workflow-run %} the rRNA-prediction sub-workflow using the following inputs:
>    - **Processed sequences**: Quality processed sequences (merged collection)
>    - **Clan information file**: `ribo.claninfo`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

# ITS
The ITS (Internal Transcribed Spacer) workflow is designed for the identification and taxonomic classification of fungal sequences, specifically focusing on ITS regions. The sub-workflow includes several tools to mask, classify, and visualize taxonomic data:
- {% tool [bedtools MaskFastaBed](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_maskfastabed/2.30.0) %} masks SSU and LSU regions from a FASTA file based on intervals defined in a BED regions file. Using the default parameters.
- {% tool [MAPseq](toolshed.g2.bx.psu.edu/repos/iuc/mapseq/mapseq/2.1.1+galaxy0) %} classifies matching OTUs to the reads, used in conjunction with the UNITE (v8.0) and ITSoneDB (v1.138) databases. Using the following parameters:
    - *"Use cached database or database from history"*: `Cached database`
    - *"Using built-in mapseq DB**. MAPseq is used in conjunction with the following databases:
        1. `MGnify ITS ITSonedb (v5.0.7) - ITSoneDB-20200214 downloaded at 2023-09-26`
        2. `MGnify ITS UNITE (v5.0.7) - UNITE-20200214 downloaded at 2023-09-26`
    - *"Create OTU table"*: `Yes`
    - *"Create taxon table for Krona"*: `Yes`
    - *"Top hits"*: `80`
    - *"Top OTUs"*: `40`
    - *"Output format"*: `simple`
- {% tool [Krona](toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1+galaxy0) %} generates pie charts demonstrating the taxonomic abundance. Using the following parameters:
    - *"What is the type of your input data"*: `Tabular`
    - *"Provide a name for the basal rank"*: `Root`
- {% tool [biom-convert](toolshed.g2.bx.psu.edu/repos/iuc/biom_convert/biom_convert/2.1.15+galaxy1) %} converts the taxonomic abundance tables to JSON and HDF5 formats. Using the following parameters:
    - *"Choose the source BIOM format"*: `Tabular File`
    - *"Process metadata associated with observations when converting"*: `taxonomy`
    - *"Choose the output type"*: `BIOM`
    - *"Table Type"*: `OTU table`
    - *"BIOM format type"*:
        1. `JSON-formatted table(BIOM1)`
        2. `HDF5-formatted table(BIOM2)`

<div class="Executing-subworkflows-using-ENA-data" markdown="1">
> <hands-on-title>Executing the ITS sub-workflow</hands-on-title>
> 1. **Import** the [ITS sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-its.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Run** {% icon workflow-run %} the ITS sub-workflow using the following inputs:
>    - *"LSU and SSU BED"*: `LSU and SSU BED regions`
>    - *"Processed sequence"*: Quality processed sequences (merged single-end and paired-end collection)
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

<div class="Executing-subworkflows-using-own-data" markdown="1">
> <hands-on-title>Executing the ITS sub-workflow</hands-on-title>
> 1. **Import** the [ITS sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-its.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Run** {% icon workflow-run %} the ITS sub-workflow using the following inputs:
>    - *"LSU and SSU BED"*: `LSU and SSU BED regions`
>    - *"Processed sequence"*: Quality processed sequences (merged single-end and paired-end collection)
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

# Summary tables
The summary tables sub-workflow takes as input the taxonomic abundance tables generated from all datasets within the pipeline. It processes these tables to create two comprehensive taxonomic summary tables:
1. Full Taxonomic Summary Table: This table provides detailed taxonomic information spanning all levels, from superkingdom down to species. It offers a complete overview of the taxonomic composition across all samples, facilitating in-depth analyses of microbial communities.

2. Phylum-Level Summary Table: Focused specifically on the phylum level, this table aggregates data to provide a clear view of the phylum distribution within the datasets. This level-specific summary is particularly useful for high-level comparisons and identifying major trends within the microbial communities.

Both summary tables serve as essential inputs for downstream interpretation, helping to visualize and compare the taxonomic profiles across multiple samples (for example, see our tutorial on[Calculating α and β diversity]({% link topics/microbiome/tutorials/diversity/tutorial.md %})).

<div class="Executing-subworkflows-using-ENA-data" markdown="1">

> <hands-on-title>Executing the summary tables sub-workflow</hands-on-title>
> 1. **Import** the [summary tables sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-summary-tables.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Run** {% icon workflow-run %} the summary tables sub-workflow 4 times using the following inputs:
>    - First run:
>      - *"OTU tables"*: `LSU OTU tables (SILVA DB)`
>      - *"Taxonomic abundance summary table name"*:
>        ```
>        LSU taxonomic abundance summary table
>        ```
>      - *"Phylum level taxonomic abundance summary table name"*:
>        ```
>        LSU phylum level taxonomic abundance summary table
>        ```
>    - Second run:
>      - *"OTU tables"*: `SSU OTU tables (SILVA DB)`
>      - *"Taxonomic abundance summary table name"*:
>        ```
>        SSU taxonomic abundance summary table
>        ```
>      - *"Phylum level taxonomic abundance summary table name"*:
>        ```
>        SSU phylum level taxonomic abundance summary table
>        ```
>    - Third run:
>      - *"OTU tables"*: `ITS OTU tables (UNITE DB)`
>      - *"Taxonomic abundance summary table name"*:
>        ```
>        UNITE taxonomic abundance summary table
>        ```
>      - *"Phylum level taxonomic abundance summary table name"*:
>        ```
>        UNITE phylum level taxonomic abundance summary table
>        ```
>    - Fourth run:
>      - *"OTU tables"*: `ITS OTU tables (ITSoneDB)`
>      - *"Taxonomic abundance summary table name"*:
>        ```
>        ITSoneDB taxonomic abundance summary table
>        ```
>      - *"Phylum level taxonomic abundance summary table name"*:
>        ```
>        ITSoneDB phylum level taxonomic abundance summary table
>        ```
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

<div class="Executing-subworkflows-using-own-data" markdown="1">
> <hands-on-title>Executing the summary tables sub-workflow</hands-on-title>
> 1. **Import** the [summary tables sub-workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-summary-tables.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Run** {% icon workflow-run %} the summary tables sub-workflow 4 times using the following inputs:
>    - First run:
>      - *"OTU tables"*: `LSU OTU tables (SILVA DB)`
>      - *"Taxonomic abundance summary table name"*:
>        ```
>        LSU taxonomic abundance summary table
>        ```
>      - *"Phylum level taxonomic abundance summary table name"*:
>        ```
>        LSU phylum level taxonomic abundance summary table
>        ```
>    - Second run:
>      - *"OTU tables"*: `SSU OTU tables (SILVA DB)`
>      - *"Taxonomic abundance summary table name"*:
>        ```
>        SSU taxonomic abundance summary table
>        ```
>      - *"Phylum level taxonomic abundance summary table name"*:
>        ```
>        SSU phylum level taxonomic abundance summary table
>        ```
>    - Third run:
>      - *"OTU tables"*: `ITS OTU tables (UNITE DB)`
>      - *"Taxonomic abundance summary table name"*:
>        ```
>        UNITE taxonomic abundance summary table
>        ```
>      - *"Phylum level taxonomic abundance summary table name"*:
>        ```
>        UNITE phylum level taxonomic abundance summary table
>        ```
>    - Fourth run:
>      - *"OTU tables"*: `ITS OTU tables (ITSoneDB)`
>      - *"Taxonomic abundance summary table name"*:
>        ```
>        ITSoneDB taxonomic abundance summary table
>        ```
>      - *"Phylum level taxonomic abundance summary table name"*:
>        ```
>        ITSoneDB phylum level taxonomic abundance summary table
>        ```
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

# Complete Workflow (including all sub-workflows)
If you're working with SSU, LSU, and ITS data, it's beneficial to run the entire workflow to take full advantage of its capabilities. Even if you're unsure which of these datasets you have, executing the complete workflow ensures that all relevant analyses are applied, maximizing the insight gained from your microbial data.

<div class="Executing-the-full-workflow-using-ENA-data" markdown="1">
> <hands-on-title>Executing the complete MGnify amplicon pipeline</hands-on-title>
> 1. **Import** the [complete MGnify amplicon workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-complete.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Run** {% icon workflow-run %} the workflow using the following inputs:
>    - *"SRA accession list"*: `accessions.csv`
>    - *"Clan information file"*: `ribo.claninfo`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}
</div>

<div class="Executing-the-full-workflow-using-own-data" markdown="1">
> <hands-on-title>Executing the complete MGnify amplicon pipeline</hands-on-title>
> 1. **Import** the [complete MGnify amplicon workflow]({{ site.baseurl }}{{ page.dir }}workflows/mgnify-amplicon-pipeline-v5-complete.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. Adjust workflow:
>    - Click on {% icon galaxy-workflows-activity %} **Workflows** in the left side-bar
>    - Search for *"MGnify's amplicon pipeline v5.0 - complete"* workflow
>    - Click on {% icon galaxy-wf-edit %} **Edit**
>    - **Remove** the steps:
>      - *"1: SRA accession list"*
>      - *"22: fastq-dl"*
>      - *"23: Convert compressed file to uncompressed"
>      - *"24: Convert compressed file to uncompressed"
>      - *"25: Text reformatting"
>      - *"26: Text reformatting"
>      - *"27: Convert uncompressed file to compressed"
>      - *"28: Convert uncompressed file to compressed"
>    - Click on **Inputs** in the left side-panel and add an **Input dataset collection**
>      - Add the *"Label"*: `Single-end reads`
>      - Connect it to the *"Single-end reads"* input in step *"29: MGnify's amplicon pipeline v5.0 - Quality control SE"*
>    - Click on **Inputs** in the left side-panel and add an **Input dataset collection**
>      - Add the *"Label"*: `Paired-end reads`
>      - Change *"Collection type"* to `list:paired`
>        - Connect it to the *"Paired-end reads"* input in step *"30: MGnify's amplicon pipeline v5.0 - Quality control PE"*
>    - Click on {% icon galaxy-save %} **Save** and then {% icon workflow-run %} **Run** in the right side-panel
> 3. **Run** {% icon workflow-run %} the workflow using the following inputs:
>    - *"Single-end reads"*: Single-end reads collection
>    - *"Paired-end reads"*: Paired-end reads collection
>    - *"Clan information file"*: Upload the clan information file, which contains details about the sequence family assignments required for accurate taxonomic classification `ribo.claninfo`
>    - *"Covariance models"*: Provide the covariance models necessary for the rRNA prediction step, enabling accurate alignment and prediction of ribosomal RNA sequences `ribo.cm`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
> > <tip-title>Input reformatting</tip-title>
> > To achieve results comparable to those from MGnify, it is recommended that your sequence headers follow this format: `@accession.spotID-$spotName-$readNumber`. For example: `@ERR2015153.1-M01170:20:000000000-AMCTR:1:1101:17071:1623-2`
> {: .tip}
>
{: .hands_on}
</div>

# MAPseq-to-ampvis2
The MAPseq to ampvis2 workflow transforms MAPseq OTU tables and accompanying metadata into a format compatible with ampvis2. This process restructures the MAPseq output into well-organized object files that can be directly used in ampvis2 for downstream analyses such as rank abundance plots, heatmaps, and other visualizations.

> <hands-on-title>Downloading datasets</hands-on-title>
> 1. Click on {% icon galaxy-upload %} **Upload Data** at the top of left panel
>    - Select {% icon galaxy-wf-edit %} **Paste/Fetch Data** at the bottom
>    - Copy {% icon copy %} and paste the following:
>      ```
>      https://zenodo.org/records/13347829/files/SRR11032273.tsv
>      https://zenodo.org/records/13347829/files/SRR11032274.tsv
>      https://zenodo.org/records/13347829/files/SRR11032275.tsv
>      https://zenodo.org/records/13347829/files/SRR11032276.tsv
>      https://zenodo.org/records/13347829/files/SRR11032277.tsv
>      https://zenodo.org/records/13347829/files/SRR11038211.tsv
>      https://zenodo.org/records/13347829/files/SRR11038212.tsv
>      https://zenodo.org/records/13347829/files/SRR11038213.tsv
>      https://zenodo.org/records/13347829/files/SRR11038214.tsv
>      https://zenodo.org/records/13347829/files/SRR11038215.tsv
>      https://zenodo.org/records/13347829/files/test_metadata_formatted.tabular
>      ```
>    - Click on **Start**
> 2. Create a collection out of the tsv files
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
{: .hands_on}

> <hands-on-title>Executing the MAPseq-to-ampvis2 workflow</hands-on-title>
> 1. **Import** the [MAPseq-to-ampvis2 workflow]({{ site.baseurl }}{{ page.dir }}workflows/mapseq-to-ampvis2.ga)
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Run** {% icon workflow-run %} the workflow using the following inputs:
>      - *"MAPseq outputs"*: Provide the OTU tables generated by MAPseq, which need to be reformatted for ampvis2
>      - *"Metadata"*: Contextual information associated with the data from the OTU tables
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}

With the ampvis2 object now successfully generated, users can proceed to create a variety of informative visualizations—such as heatmaps—to explore and interpret microbial community composition, identify patterns across samples, and support further ecological or statistical analysis.

> <hands-on-title>Utilizing the ampvis2 object to generate a heatmap</hands-on-title>
>
> 1. {% tool [ampvis2 heatmap](toolshed.g2.bx.psu.edu/repos/iuc/ampvis2_heatmap/ampvis2_heatmap/2.8.9+galaxy1) %}:
>    - **Ampvis2 RDS dataset**: Choose `Ampvis2 object` from your history
>    - **The taxonomic level to aggregate the OTUs**: `Order`
>    - **How to show OTUs without taxonomic information**: `Remove OTUs without taxonomic information`
>    - Select light blue for **Start color for the heatmap**
>    - Select dark red for **End color for the heatmap**
>
{: .hands_on}

Let us now take a look at the generated heatmap.

![Heatmap produced by the workflow](./images/heatmap.png){: style="width:75%;"}


> <question-title>i</question-title>
>
> 1. Which sample has the highest relative abundance of Lactobacillales?
> 2. Which taxonomic order is most dominant in sample SRR1038213?
>
> > <solution-title></solution-title>
> >
> > 1. Sample SRR11032276 has the highest relative abundance of Lactobacillales at 97.7%, indicating it is strongly dominated by this bacterial order.
> > 2. In sample SRR11038213, Tremellales is the most dominant order, with a relative abundance of 50%.
> >
> {: .solution}
>
{: .question}

This was a brief glimpse into the capabilities of ampvis2. We encourage you to further explore the full range of features available in Ampvis2, as well as other powerful visualization tools offered on the Galaxy platform.

# Conclusion
This tutorial provided a step-by-step guide to executing the different sub-workflows of the MGnify v5.0 amplicon pipeline on Galaxy. By following these steps, you should be able to process, analyze, and visualize your amplicon sequencing data efficiently.

tools_json:
["LotuS2", "toolshed.g2.bx.psu.edu/repos/earlhaminst/lotus2/lotus2/2.32+galaxy0", "fastq-dl", "toolshed.g2.bx.psu.edu/repos/iuc/fastq_dl/fastq_dl/3.0.0+galaxy0", "Trimmomatic", "toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.39+galaxy2", "Filter FASTQ", "toolshed.g2.bx.psu.edu/repos/devteam/fastq_filter/fastq_filter/1.1.5", "PRINSEQ", "toolshed.g2.bx.psu.edu/repos/iuc/prinseq/prinseq/0.20.4+galaxy2", "convert FASTQ to FASTA", "toolshed.g2.bx.psu.edu/repos/devteam/fastqtofasta/fastq_to_fasta_python/1.1.5", "FASTQC", "toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy0", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy1", "fastp", "toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.23.4+galaxy1", "SeqPrep", "toolshed.g2.bx.psu.edu/repos/iuc/seqprep/seqprep/1.3.2+galaxy0", "CMseach", "toolshed.g2.bx.psu.edu/repos/bgruening/infernal/infernal_cmsearch/1.1.4+galaxy0", "cmsearch-deoverlap", "toolshed.g2.bx.psu.edu/repos/rnateam/cmsearch_deoverlap/cmsearch_deoverlap/0.08+galaxy2", "bedtools getfasta", "toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_getfastabed/2.30.0+galaxy1", "MAPseq", "toolshed.g2.bx.psu.edu/repos/iuc/mapseq/mapseq/2.1.1+galaxy0", "Krona", "toolshed.g2.bx.psu.edu/repos/crs4/taxonomy_krona_chart/taxonomy_krona_chart/2.7.1+galaxy0", "biom-convert", "toolshed.g2.bx.psu.edu/repos/iuc/biom_convert/biom_convert/2.1.15+galaxy1", "Merge collections", "__MERGE_COLLECTION__", "bedtools MaskFastaBed", "toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_maskfastabed/2.30.0", "ampvis2 heatmap", "toolshed.g2.bx.psu.edu/repos/iuc/ampvis2_heatmap/ampvis2_heatmap/2.8.9+galaxy1"]


topic: microbiome
tutorial: metatranscriptomics
title: Metatranscriptomics analysis using microbiome RNA-seq data
difficulty: introductory
time_estimation: 5H
content: {% include topics/microbiome/tutorials/metatranscriptomics/content.md short=false %}
tools_json: []
requirements_json: []
tags_json: ["metatranscriptomics", "microgalaxy"]
content_hash: 29f09e32001e8c823803990143837078
last_modified: 2026-05-27T09:01:49.369971
zenodo_link: https://zenodo.org/record/4776250
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/metatranscriptomics/tutorial.html


topic: microbiome
tutorial: clinical-mp-2-discovery
title: Clinical Metaproteomics 2: Discovery
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: ["[label-TMT11]"]
content_hash: 6dfe42e96d7b7bfa9a0f8ec8e07c93a9
last_modified: 2026-05-27T09:01:49.453785
zenodo_link: https://doi.org/10.5281/zenodo.10105821
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/clinical-mp-2-discovery/tutorial.html

content:
This tutorial can be followed with any user-defined database but would work best if the clinical metaproteomics database generation module was used (see [Database Generation tutorial](https://github.com/subinamehta/training-material/blob/main/topics/proteomics/tutorials/clinical-mp-database-generation/tutorial.md)). The MetaNovo tool generates a more manageable database that contains identified proteins. The MetaNovo-generated database merged with Human SwissProt (reviewed only) and contaminants (cRAP) databases to generate a compact database (~21.2k protein sequences) that will be used for peptide identification.


# Peptide identification
The MSMS data will be searched against the compact database `Human UniProt Microbial Proteins (from MetaNovo) and cRAP` to identify peptide and protein sequences via sequence database searching. For this tutorial, two peptide identification programs will be used: SearchGUI/PeptideShaker and MaxQuant. However, you could use other software too, such as Fragpipe or Scribe. For the purpose of this tutorial, a dataset of the 4 RAW/MGF files will be used as the MS/MS input.


![Discovery Workflow]({% link topics/proteomics/images/clinical-mp/clinical-mp-discovery.JPG %})


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Database Searching
This step is to identify proteins based on mass spectrometry data. The algorithms identify peptides in the spectra and search a protein sequence database to match observed peptide data with theoretical peptide masses and spectra. Scoring and false discovery rate control help assess the reliability of matches, followed by protein inference to determine the proteins present in the sample. These algorithms are essential for interpreting mass spectrometry data, aiding in protein identification, quantification, and insights into biological processes and disease mechanisms in proteomics research.

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/10105821/files/Human_UniProt_Microbial_Proteins_(from_MetaNovo)_and_cRAP.fasta
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F10_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F11_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F13_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F15_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/Experimental-Design_Discovery_MaxQuant.tabular
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Add to each database a tag corresponding to user.
> 6. Create a dataset collection of all the raw files and MGF files.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Import Workflow


> <hands-on-title>Running the Workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/proteomics/tutorials/clinical-mp-2-discovery/workflows/WF2_Discovery-Workflow.ga" title="Discovery Workflow" %}
>
> 2. Run **Workflow** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *" RAW files"*: `RAW dataset collection`
>    - {% icon param-file %} *" Human UniProt Microbial Proteins (from MetaNovo) and cRAP"*: `Human_UniProt_Microbial_Proteins_(from_MetaNovo)_and_cRAP.fasta`
>    - {% icon param-file %} *" Experimental Design Discovery MaxQuant"*: `Experimental-Design_Discovery_MaxQuant.tabular`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}


# Peptide identification
Using the compact database generated by MetaNovo as the input database, we will match MS/MS data to peptide sequences via sequence database searching.

For this tutorial, two peptide identification programs will be used: SearchGUI/PeptideShaker and MaxQuant. For both programs, the created dataset of the four MS datasets  in the history will be used as the MS/MS input. The RAW MS/MS data files will be converted into mascot generic format (MGF) files as that is the standard format in which MS/MS searches are performed.

Peptides identified from each program will be verified with the PepQuery tool to generate a master list of confident verified microbial peptides.

## Appending decoy sequenced to FASTA database with **FastaCLI**

Using the FastaCLI tool, decoy sequences will be appended to the FASTA database. Decoy sequences are protein sequences are not expected to be present in samples. For more information on how to generate and append decoy sequences, see [GTN Protein FASTA Database Handling](https://training.galaxyproject.org/archive/2019-12-01/topics/proteomics/tutorials/database-handling/tutorial.html#creating-a-decoy-database).

> <hands-on-title> FastaCLI </hands-on-title>
>
> 1. {% tool [FastaCLI](toolshed.g2.bx.psu.edu/repos/galaxyp/peptideshaker/fasta_cli/4.0.41+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Protein Database"*: `output` (Input dataset)
>
>
{: .hands_on}

## Converting RAW files to MGF files with **msconvert**

The msconvert tool allows for the conversion of mass spectrometry data files between different formats, such as thermo.raw, mgf, or mzml.

> <hands-on-title> msconvert: RAW to MGF </hands-on-title>
>
> 1. {% tool [msconvert](toolshed.g2.bx.psu.edu/repos/galaxyp/msconvert/msconvert/3.0.20287.2) %} with the following parameters:
>    - {% icon param-collection %} *"Input unrefined MS data"*: `output` (Input dataset collection)
>    - *"Do you agree to the vendor licenses?"*: `Yes`
>    - *"Output Type"*: `mgf`
>    - In *"Data Processing Filters"*:
>        - *"Apply peak picking?"*: `Yes`
>        - *"(Re-)calculate charge states?"*: `no`
>
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Why do we need to use MGF instead of RAW files for Search GUI?
>
> > <solution-title></solution-title>
> >
> > 1. SearchGUI is compatible only with MGF files, hence you have to use msconvert or Thermofile converter tools to convert the RAW format to MGF fomat.
> >
> {: .solution}
>
{: .question}

## Perform Database searching with **Search GUI**
SearchGUI is a database-searching tool that comprises different search engines to match sample MS/MS spectra to known peptide sequences. In our analysis, we will use X!Tandem and MS-GF+ as search algorithms within SearchGUI for matching spectra from mass spectrometry data against peptides from the protein sequence database.

The SearchGUI tool will perform a database search based on the parameters we've set and will generate a file (called a SearchGUI archive file) that will serve as the input for the PeptideShaker tool. The SearchGUI archive file contains Peptide-Spectral Matches (PSMs), and PeptideShaker is a post-processing software that will assess the confidence of the data. PeptideShaker also infers the identities of proteins based on the matched peptide sequences, and users are able to visualize these outputs to interpret results.  More information about database searching using SearchGUI and PeptideShaker is accessible at [Metaproteomics tutorial](https://gxy.io/GTN:T00221).

> <hands-on-title> Peptide discovery using SearchGUI </hands-on-title>
>
> 1. {% tool [Search GUI](toolshed.g2.bx.psu.edu/repos/galaxyp/peptideshaker/search_gui/4.0.41+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Identification Parameters file"*: `Identification_Parameters_File` (output of **Identification Parameters** {% icon tool %})
>    - {% icon param-file %} *"Fasta file"*: `input_database_concatenated_target_decoy` (output of **FastaCLI** {% icon tool %})
>    - {% icon param-file %} *"Input Peak Lists"*: `output` (output of **msconvert** {% icon tool %})
>    - *"SearchGUI Options"*: `Default`
>
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Why do we need to add decoy sequences to our FASTA database for Search GUI? And how many do we need to add?
>
> > <solution-title></solution-title>
> >
> > 1. Adding decoy sequences helps in FDR estimation, discriminating true positives from false positives, and quality control of the data. The number of decoy sequences you need to add to your database depends on the desired FDR level you want to achieve. A common practice is to use a 1:1 ratio of target sequences to decoy sequences. In other words, for every real protein sequence in your database, you would add a decoy sequence. This allows you to estimate the FDR at 1%, 5%, or any other chosen threshold.
> >
> {: .solution}
>
{: .question}
> <question-title></question-title>
>
> 1. What is the Identification Parameters tool?
>
> > <solution-title></solution-title>
> >
> > 1. Identification Parameters tool is an input required by the search GUI tool, it contains all the parameters required to run the search algorithms.
> >
> {: .solution}
>
{: .question}

## Post-processing of SearchGUI output using with **Peptide Shaker**

> <hands-on-title> Peptide Shaker </hands-on-title>
>
> 1. {% tool [Peptide Shaker](toolshed.g2.bx.psu.edu/repos/galaxyp/peptideshaker/peptide_shaker/2.0.33+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Compressed SearchGUI results"*: `searchgui_results` (output of **Search GUI** {% icon tool %})
>    - In *"Exporting options"*:
>        - *"Follow-up analysis export options"*: `Do not export`
>        - *"Identification features reports to be generated"*: `PSM Report` `Peptide Report` `Protein Report` `Certificate of Analysis`
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. What are the differences between the following reports from PeptideShaker: PSM report, Peptide report, and Protein report?
>
> > <solution-title></solution-title>
> >
> > 1. PSM reports focus on individual peptide-spectrum matches, providing detailed information about each spectrum and its assigned peptide sequence. Peptide reports summarize information about unique peptides and their properties. Protein reports, on the other hand, focus on proteins, including protein inference, grouping, and quantification, making them more suitable for understanding the overall protein composition in a sample. These reports serve different purposes in proteomic data analysis and are used to extract various levels of information from mass spectrometry results.
> >
> {: .solution}
>
{: .question}

##  Using Text Manipulation Tools to Manage Microbial Outputs from SearchGUI/PeptideShaker

> <hands-on-title> Selecting microbial peptides from SearchGUI/PeptideShaker with Select tool </hands-on-title>
>
> 1. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `output_peptides` (output of **Peptide Shaker** {% icon tool %})
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_HUMAN)|(_REVERSED)|(REV_)|(CON)|(con)`
>    - *"Keep header line"*: `Yes`
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. What is the purpose of this step?
>
> > <solution-title></solution-title>
> >
> > 1. This step is to extract microbial peptides or to remove any peptides that match humans, reverse, contaminants, etc.
> >
> {: .solution}
>
{: .question}

> <hands-on-title> Selecting microbial PSMs from SearchGUI/PeptideShaker with Select </hands-on-title>
>
> 1. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `output_psm` (output of **Peptide Shaker** {% icon tool %})
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_HUMAN)|(_REVERSED)|(REV_)|(CON)|(con)`
>    - *"Keep header line"*: `Yes`
>
>
{: .hands_on}


> <hands-on-title> Filtering confident microbial peptides from SGPS  with Filter </hands-on-title>
>
> 1. {% tool [Filter](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `out_file1` (output of **Select** {% icon tool %})
>    - *"With following condition"*: `c17=='Confident'`
>    - *"Number of header lines to skip"*: `1`
>
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. In the Filtering steps, what does “Confidence” mean quantitatively, i.e. what is the percentage cutoff?
>
> > <solution-title></solution-title>
> >
> > 1. The term "Confidence" in the context of proteomic data analysis often refers to a measure of how reliable or trustworthy a particular protein or peptide identification is. However, the specific numerical value or percentage cutoff for confidence can vary depending on the software or approach you are using and the goals of your analysis. In many proteomics studies, researchers use a false discovery rate (FDR) to set a quantitative confidence threshold. Here we have set it as 1%FDR, which means that you're accepting only 1% or less of your reported identifications as likely to be false positives.
> >
> {: .solution}
>
{: .question}


> <hands-on-title> Filtering confident microbial PSMs from SGPS with Filter </hands-on-title>
>
> 1. {% tool [Filter](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `out_file1` (output of **Select** {% icon tool %})
>    - *"With following condition"*: `c24=='Confident'`
>    - *"Number of header lines to skip"*: `1`
>
>
{: .hands_on}


We will generate and merge the Human SwissProt Protein Database and contaminants (cRAP) and convert the resulting FASTA file to a tabular file that will be used in the Query Tabular tool to generate distinct microbial peptides from SearchGUI/PeptideShaker.

> <hands-on-title> Merging Human SwissProt and cRAP databases for Query Tabular with FASTA Merge Files and Filter Unique Sequences </hands-on-title>
>
> 1. {% tool [FASTA Merge Files and Filter Unique Sequences](toolshed.g2.bx.psu.edu/repos/galaxyp/fasta_merge_files_and_filter_unique_sequences/fasta_merge_files_and_filter_unique_sequences/1.2.0) %} with the following parameters:
>    - *"Run in batch mode?"*: `Merge individual FASTAs (output collection if input is collection)`
>        - In *"Input FASTA File(s)"*:
>            - {% icon param-repeat %} *"Insert Input FASTA File(s)"*
>                - {% icon param-file %} *"FASTA File"*: `Human Swissprot Protein Database` (output of **Protein Database Downloader** {% icon tool %})
>                - - {% icon param-file %} *"FASTA File"*: `Contaminants cRAP database` (output of **Protein Database Downloader** {% icon tool %})
>
>
{: .hands_on}


> <hands-on-title> Converting FASTA sequences to TAB-delimited file with FASTA-to-Tabular </hands-on-title>
>
> 1. {% tool [FASTA-to-Tabular](toolshed.g2.bx.psu.edu/repos/devteam/fasta_to_tabular/fasta2tab/1.1.0) %} with the following parameters:
>    - {% icon param-file %} *"Convert these sequences"*: `output` (output of **FASTA Merge Files and Filter Unique Sequences** {% icon tool %})
>
>
{: .hands_on}


> <hands-on-title> Filtering out accession numbers from TAB-delimited file  with Filter Tabular </hands-on-title>
>
> 1. {% tool [Filter Tabular](toolshed.g2.bx.psu.edu/repos/iuc/filter_tabular/filter_tabular/3.3.0) %} with the following parameters:
>    - {% icon param-file %} *"Tabular Dataset to filter"*: `output` (output of **FASTA-to-Tabular** {% icon tool %})
>    - In *"Filter Tabular Input Lines"*:
>        - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>            - *"Filter By"*: `select columns`
>                - *"enter column numbers to keep"*: `1`
>        - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>            - *"Filter By"*: `regex replace value in column`
>                - *"enter column number to replace"*: `1`
>                - *"regex pattern"*: `^[^|]+[|]([^| ]+).*$`
>                - *"replacement expression"*: `\1`
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. What’s the difference between a FASTA and Tabular output?
>
> > <solution-title></solution-title>
> >
> > 1. FASTA Output: Typically used to report identified peptide or protein sequences, which are useful for building or updating sequence databases, for downstream sequence analysis, or for re-searching against the sequences.
> > Tabular Output: Used for presenting various information related to identified peptides or proteins, such as accession numbers, scores, abundance values, and other attributes. Tabular output facilitates data analysis, comparisons, and custom data processing.
> >
> >
> {: .solution}
>
{: .question}


> <hands-on-title> Querying protein accession numbers and peptide sequences of confident microbial PSMs (from SGPS) with Query Tabular </hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.0) %} with the following parameters:
>    - In *"Database Table"*:
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `out_file1` (output of **Filter** {% icon tool %})
>            - In *"Filter Dataset Input"*:
>                - In *"Filter Tabular Input Lines"*:
>                    - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>                        - *"Filter By"*: `skip leading lines`
>                            - *"Skip lines"*: `1`
>                    - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>                        - *"Filter By"*: `prepend a line number column`
>            - In *"Table Options"*:
>                - *"Specify Name for Table"*: `psms`
>                - *"Specify Column Names (comma-separated list)"*: `ln,id,Proteins,Sequence`
>                - *"Only load the columns you have named into database"*: `Yes`
>                - In *"Table Index"*:
>                    - {% icon param-repeat %} *"Insert Table Index"*
>                        - *"Index on Columns"*: `ln`
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `out_file1` (output of **Filter** {% icon tool %})
>            - In *"Filter Dataset Input"*:
>                - In *"Filter Tabular Input Lines"*:
>                    - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>                        - *"Filter By"*: `skip leading lines`
>                            - *"Skip lines"*: `1`
>                    - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>                        - *"Filter By"*: `prepend a line number column`
>                    - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>                        - *"Filter By"*: `normalize list columns, replicates row for each item in list`
>                            - *"enter column numbers to normalize"*: `3`
>            - In *"Table Options"*:
>                - *"Specify Name for Table"*: `prots`
>                - *"Specify Column Names (comma-separated list)"*: `ln,id,prot`
>                - *"Only load the columns you have named into database"*: `Yes`
>                - In *"Table Index"*:
>                    - {% icon param-repeat %} *"Insert Table Index"*
>                        - *"This is a unique index"*: `Yes`
>                        - *"Index on Columns"*: `prot,ln`
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `output` (output of **Filter Tabular** {% icon tool %})
>            - In *"Table Options"*:
>                - *"Specify Name for Table"*: `Uniprot`
>                - *"Specify Column Names (comma-separated list)"*: `prot`
>                - In *"Table Index"*:
>                    - {% icon param-repeat %} *"Insert Table Index"*
>                        - *"Index on Columns"*: `prot`
>    - *"SQL Query to generate tabular output"*: `SELECT id,Proteins,Sequence FROM psms WHERE psms.ln NOT IN (SELECT distinct prots.ln FROM prots JOIN Uniprot ON prots.prot = Uniprot.prot) ORDER BY psms.ln`
>    - *"include query result column headers"*: `Yes`
>
>
{: .hands_on}


> <hands-on-title> Cutting out peptide sequences from Query Tabular with Cut </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c3`
>    - {% icon param-file %} *"From"*: `output` (output of **Query Tabular** {% icon tool %})
>
>
{: .hands_on}


> <hands-on-title> Grouping distinct (unique) peptides from SGPS with Group </hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `out_file1` (output of **Cut** {% icon tool %})
>    - *"Group by column"*: `c1`
>
>
{: .hands_on}


## Perform peptide discovery with **MaxQuant**
MaxQuant is an MS-based proteomics platform that is capable of processing raw data and provides improved mass precision and high precursor mass accuracy (HPMA), which resulted in increased protein identification and more in-depth proteomic analysis. Raw MS/MS spectra will be searched against the reduced MetaNovo-generated database (~21.2k sequences). More information about analysis using MaxQuant is available, including [Label-free data analysis](https://gxy.io/GTN:T00218) and [MaxQuant and MSstats for the analysis of TMT data](https://gxy.io/GTN:T00220).

> <hands-on-title> Peptide discovery using MaxQuant </hands-on-title>
>
> 1. {% tool [MaxQuant](toolshed.g2.bx.psu.edu/repos/galaxyp/maxquant/maxquant/2.0.3.0+galaxy0) %} with the following parameters:
>    - In *"Input Options"*:
>        - {% icon param-file %} *"FASTA files"*: `output` (Input dataset)
>    - In *"Search Options"*:
>        - {% icon param-file %} *"Specify an experimental design template (if needed). For detailed instructions see the help text."*: `output` (Input dataset)
>        - *"minimum peptide length"*: `8`
>        - *"Match between runs"*: `Yes`
>        - *"Maximum peptide length for unspecific searches"*: `50`
>    - In *"Protein quantification"*:
>        - *"Use only unmodified peptides"*: `Yes`
>            - *"Modifications used in protein quantification"*: `Oxidation (M)`
>        - In *"LFQ Options"*:
>            - *"iBAQ (calculates absolute protein abundances by normalizing to copy number and not protein mass)"*: `No`
>    - In *"Parameter Group"*:
>        - {% icon param-repeat %} *"Insert Parameter Group"*
>            - {% icon param-collection %} *"Infiles"*: `output` (Input dataset collection)
>            - *"fixed modifications"*: `Carbamidomethyl (C)`
>            - *"variable modifications"*: `Oxidation (M)`
>            - *"enzyme"*: `Trypsin/P`
>            - *"Quantitation Methods"*: `reporter ion MS2`
>                - *"isobaric labeling"*: `TMT11plex`
>                - *"Filter by PIF"*: `True`
>    - *"Generate PTXQC (proteomics quality control pipeline) report? (experimental setting)"*: `False`
>    - In *"Output Options"*:
>        - *"Select the desired outputs."*: `Protein Groups` `mqpar.xml` `Peptides` `MSMS` `msms scans` `summary` `MaxQuant and PTXQC log` `yaml config file`
>
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. What is the Experimental Design file for MaxQuant?
> 
> > <solution-title></solution-title>
> >
> > 1. In MaxQuant, the **Experimental Design** file is used to specify the experimental conditions, sample groups, and the relationships between different samples in a proteomics experiment. This file is a crucial component of the MaxQuant analysis process because it helps the software correctly organize and analyze the mass spectrometry data. The **Experimental Design** file typically has a ".txt" extension and is a tab-delimited text file. Here's what you might include in an Experimental Design file for MaxQuant: **Sample Names** (You specify the names of each sample in your experiment. These names should be consistent with the naming conventions used in your raw data files.), **Experimental Conditions** (You define the experimental conditions or treatment groups associated with each sample. For example, you might have control and treated groups, and you would assign the appropriate condition to each sample.), **Replicates** (You indicate the replicates for each sample, which is important for assessing the statistical significance of your results. Replicates are typically denoted by numeric values (e.g., "1," "2," "3") or by unique identifiers (e.g., "Replicate A," "Replicate B")), **Labels** (If you're using isobaric labeling methods like TMT (Tandem Mass Tag) or iTRAQ (Isobaric Tags for Relative and Absolute Quantitation), you specify the labels associated with each sample. This is important for quantification.), **Other Metadata** (You can include additional metadata relevant to your experiment, such as the biological source, time points, or any other information that helps describe the samples and experimental conditions.)
> >
> {: .solution}
>
{: .question}


## Using Text Manipulation Tools to Manage Microbial Outputs from MaxQuant

> <hands-on-title> Selecting microbial peptides from MaxQuant with Select </hands-on-title>
>
> 1. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `peptides` (output of **MaxQuant** {% icon tool %})
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_HUMAN)|(_REVERSED)|(REV_)|(CON)|(con)`
>    - *"Keep header line"*: `Yes`
>
>
{: .hands_on}


> <hands-on-title> Cutting out microbial peptide sequences with Cut </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - {% icon param-file %} *"From"*: `out_file1` (output of **Select** {% icon tool %})
>
>
{: .hands_on}


> <hands-on-title> Remove header line from MaxQuant peptide output with Remove beginning </hands-on-title>
>
> 1. {% tool [Remove beginning](Remove beginning1) %} with the following parameters:
>    - {% icon param-file %} *"from"*: `out_file1` (output of **Cut** {% icon tool %})
>
>
{: .hands_on}


> <hands-on-title> Grouping distinct (unique) peptide sequences from MaxQuant with Group </hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `out_file1` (output of **Remove beginning** {% icon tool %})
>    - *"Group by column"*: `c1`
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. How case-sensitive is the Group tool? Can I only group by column values, and not row values?
>
> > <solution-title></solution-title>
> >
> > 1. You can make it case sensitive, by default it is not. The tool here does column grouping only.
> >
> {: .solution}
>
{: .question}


## Process SGPS and MaxQuant peptides to compile one list of unique microbial peptides


> <hands-on-title> Concatenate SGPS and MaxQuant peptides into a singular database with Concatenate datasets </hands-on-title>
>
> 1. {% tool [Concatenate datasets](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/0.1.1) %} with the following parameters:
>    - {% icon param-files %} *"Datasets to concatenate"*: `out_file1` (output of **Group** {% icon tool %}), `out_file1` (output of **Group** {% icon tool %})
>
>
{: .hands_on}


> <hands-on-title> Group the peptides from SGPS and MaxQuant to remove duplicates with Group </hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `out_file1` (output of **Concatenate datasets** {% icon tool %})
>    - *"Group by column"*: `c1`
>
>
{: .hands_on}


# Conclusion
By following this tutorial, you have effectively conducted a search of your MS/MS data against the compact database and successfully retrieved reliable microbial peptides. After identifying these microbial peptides with the assistance of MaxQuant and SearchGUI, the next step is to verify the presence of these peptides. This compiled list of unique peptides will serve as the input for PepQuery to validate the confident identification of microbial peptides with the help of the verification workflow.

tools_json:
["FastaCLI", "toolshed.g2.bx.psu.edu/repos/galaxyp/peptideshaker/fasta_cli/4.0.41+galaxy1", "msconvert", "toolshed.g2.bx.psu.edu/repos/galaxyp/msconvert/msconvert/3.0.20287.2", "Search GUI", "toolshed.g2.bx.psu.edu/repos/galaxyp/peptideshaker/search_gui/4.0.41+galaxy1", "Peptide Shaker", "toolshed.g2.bx.psu.edu/repos/galaxyp/peptideshaker/peptide_shaker/2.0.33+galaxy1", "Select", "Grep1", "Filter", "Filter1", "FASTA Merge Files and Filter Unique Sequences", "toolshed.g2.bx.psu.edu/repos/galaxyp/fasta_merge_files_and_filter_unique_sequences/fasta_merge_files_and_filter_unique_sequences/1.2.0", "FASTA-to-Tabular", "toolshed.g2.bx.psu.edu/repos/devteam/fasta_to_tabular/fasta2tab/1.1.0", "Filter Tabular", "toolshed.g2.bx.psu.edu/repos/iuc/filter_tabular/filter_tabular/3.3.0", "Query Tabular", "toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.0", "Cut", "Cut1", "Group", "Grouping1", "MaxQuant", "toolshed.g2.bx.psu.edu/repos/galaxyp/maxquant/maxquant/2.0.3.0+galaxy0", "Remove beginning", "Remove beginning1", "Concatenate datasets", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/0.1.1"]


topic: microbiome
tutorial: mothur-miseq-sop-short
title: 16S Microbial Analysis with mothur (short)
difficulty: intermediate
time_estimation: 2h
content: {% include topics/microbiome/tutorials/mothur-miseq-sop/content.md short=true %}
tools_json: []
requirements_json: []
tags_json: ["metabarcoding", "16S", "microgalaxy"]
content_hash: 6efb0c6c4c5f1e7bc21f79187177bb02
last_modified: 2026-05-27T09:01:49.374952
zenodo_link: https://doi.org/10.5281/zenodo.800651
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/mothur-miseq-sop-short/tutorial.html


topic: microbiome
tutorial: clinical-mp-4-quantitation
title: Clinical Metaproteomics 4: Quantitation
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: ["[label-TMT11]"]
content_hash: c177e1b90366ad012fb84648be68fcb3
last_modified: 2026-05-27T09:01:49.454010
zenodo_link: https://doi.org/10.5281/zenodo.10105821
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/clinical-mp-4-quantitation/tutorial.html

content:
The next step of the clinical metaproteomics workflow is the quantification workflow. Running a quantification workflow in proteomics is essential for several critical purposes. It allows researchers to measure and compare the abundance of proteins or peptides in biological samples, offering valuable insights into biomarker discovery, comparative analysis, and differential expression studies. Quantitative proteomics helps reveal the functional roles of proteins, the stoichiometry of protein complexes, and the effects of drugs on protein expression in pharmacological studies. Additionally, it serves as a quality control measure, validating initial protein identifications, and providing data normalization for increased accuracy. Quantitative data are indispensable for hypothesis testing, systems biology, and their clinical relevance in areas such as disease diagnosis, prognosis, and therapeutic decision-making. In summary, the quantitation workflow in proteomics is a cornerstone for deciphering the complexities of protein expression and regulation, facilitating a wide array of biological and clinical applications.

In this current workflow, we perform Quantification using the MaxQuant tool and the output will be interpreted in our next module.

![Quantitation workflow]({% link topics/proteomics/images/clinical-mp/clinical-mp-quantification.JPG %})



> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F10_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F11_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F13_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F15_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/Experimental-Design_Discovery_MaxQuant.tabular
>    https://zenodo.org/records/10105821/files/Quantitation_Database_for_MaxQuant.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Add to each database a tag corresponding to input files.
> 6. Create a dataset of the RAW files.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
{: .hands_on}


# Import Workflow

> <hands-on-title>Running the Workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/proteomics/tutorials/clinical-mp-4-quantitation/workflows/WF4_Quantitation_Workflow.ga" title="Quantitation Workflow" %}
>
> 2. Run **Workflow** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *" Quantitation_Database-For-MaxQuant * "*: `Quantitation_Database_for_MaxQuant.fasta`
>    - {% icon param-file %} *" Experimental-Design Discovery MaxQuant"*: `Experimental-Design_Discovery_MaxQuant.tabular`
>    - {% icon param-file %} *" Input Raw-files"*: `RAW dataset collection`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}


# Peptide quantification

In the [Discovery Module](https://github.com/subinamehta/training-material/blob/main/topics/proteomics/tutorials/clinical-mp-discovery/tutorial.md), we used MaxQuant to identify peptides for verification. Now, we will again use MaxQuant to further quantify the PepQuery-verified peptides, both microbial and human. More information about quantitation using MaxQuant is available, including [Label-free data analysis](https://gxy.io/GTN:T00218) and [MaxQuant and MSstats for the analysis of TMT data](https://gxy.io/GTN:T00220).

The outputs we are most interested in consist of the `MaxQuant Evidence file`, `MaxQuant Protein Groups`, and `MaxQuant Peptides`. The `MaxQuant Peptides` file will allow us to group them to generate a list of quantified microbial peptides.

> <hands-on-title> Quantify verified peptides (from PepQuery2) </hands-on-title>
>
> 1. {% tool [MaxQuant](toolshed.g2.bx.psu.edu/repos/galaxyp/maxquant/maxquant/1.6.17.0+galaxy4) %} with the following parameters:
>    - In *"Input Options"*:
>        - {% icon param-file %} *"FASTA files"*: `Quantitation Database for MaxQuant` (Input dataset)
>    - In *"Search Options"*:
>        - {% icon param-file %} *"Specify an experimental design template (if needed). For detailed                           instructions see the help text."*: `output` (Input dataset)
>        - *"minimum peptide length"*: `8`
>        - *"Match between runs"*: `Yes`
>        - *"Maximum peptide length for unspecific searches"*: `50`
>    - In *"Protein quantification"*:
>        - *"Use only unmodified peptides"*: `Yes`
>            - *"Modifications used in protein quantification"*: `Oxidation (M)`
>        - In *"LFQ Options"*:
>            - *"iBAQ (calculates absolute protein abundances by normalizing to copy number and not protein mass)"*: `No`
>    - In *"Parameter Group"*:
>        - {% icon param-repeat %} *"Insert Parameter Group"*
>            - {% icon param-collection %} *"Infiles"*: `output` (Input dataset collection)
>            - *"fixed modifications"*: `Carbamidomethyl (C)`
>            - *"variable modifications"*: `Oxidation (M)`
>            - *"enzyme"*: `Trypsin/P`
>            - *"Quantitation Methods"*: `reporter ion MS2`
>                - *"isobaric labeling"*: `TMT11plex`
>                - *"Filter by PIF"*: `Yes`
>    - In *"Output Options"*:
>        - *"Select the desired outputs."*: `Protein Groups` `mqpar.xml` `Peptides` `Evidence` `MSMS`
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Why can we switch back to using RAW files for MaxQuant, instead of using MGF files?
>
> > <solution-title></solution-title>
> >
> > 1. MaxQuant prefers RAW format compared to MGF as it has more information compared to MGF.
> >
> {: .solution}
>
{: .question}
> <question-title></question-title>
>
> 1. Previously, we used MaxQuant in the Discovery workflow. Why are we using MaxQuant again, instead of Search GUI/PeptideShaker?
>
> > <solution-title></solution-title>
> >
> > 1. We are using MaxQuant for quantification purposes only. SearchGUI Peptide Shaker doesn't have the capability to perform quantification of peptides or proteins.
> >
> {: .solution}
>
{: .question}

## Using Text Manipulation Tools to Manage MaxQuant Outputs

> <hands-on-title> Select microbial protein groups from MaxQuant with Select </hands-on-title>
>
> 1. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `proteinGroups` (output of **MaxQuant** {% icon tool %})
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_HUMAN)|(_REVERSED)|(REV_)|(CON)|(con)`
>
>
> 2. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `peptides` (output of **MaxQuant** {% icon tool %})
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_HUMAN)|(_REVERSED)|(REV_)|(CON)|(con)`
>
>
> 3. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - {% icon param-file %} *"From"*: `out_file1` (output of **Select** {% icon tool %})
>
>
>
> 4. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - {% icon param-file %} *"From"*: `out_file1` (output of **Select** {% icon tool %})
>
>
{: .hands_on}

## Generating a list of quantified proteins and peptides

> <hands-on-title> Group quantified proteins </hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `out_file1` (output of **Cut** {% icon tool %})
>    - *"Group by column"*: `c1`
>
>
{: .hands_on}
> <hands-on-title> Group quantified peptides </hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `out_file1` (output of **Cut** {% icon tool %})
>    - *"Group by column"*: `c1`
>
>
{: .hands_on}


# Conclusion

In summary, the implementation of a quantitation workflow using MaxQuant represents a significant advancement in quantitative proteomic research. This approach enables precise measurement of protein and peptide abundances, enhancing our ability to unravel the complexities of biological systems. This workflow is instrumental in biomarker discovery, comparative analysis, and understanding differential protein expression by offering detailed insights into quantitative changes across different experimental conditions. Its capacity to generate accurate data supports a wide spectrum of applications, including disease research, drug development, and systems biology investigations. Furthermore, the MaxQuant-based quantitation workflow ensures data quality, enabling reliable and reproducible results. It serves as a vital step for quality control, allowing researchers to draw meaningful conclusions from proteomic experiments confidently.

tools_json:
["MaxQuant", "toolshed.g2.bx.psu.edu/repos/galaxyp/maxquant/maxquant/1.6.17.0+galaxy4", "Select", "Grep1", "Cut", "Cut1", "Group", "Grouping1"]


topic: microbiome
tutorial: clinical-mp-1-database-generation
title: Clinical Metaproteomics 1: Database-Generation
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: ["[label-TMT11]"]
content_hash: 113e6a7484bc258c821fcfc28ea5ed66
last_modified: 2026-05-27T09:01:49.453643
zenodo_link: https://doi.org/10.5281/zenodo.10105821
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/microbiome/tutorials/clinical-mp-1-database-generation/tutorial.html

content:
Metaproteomics is the large-scale characterization of the entire complement of proteins expressed by microbiota. However, metaproteomics analysis of clinical samples is challenged by the presence of abundant human (host) proteins which hampers the confident detection of lower abundant microbial proteins {% cite Batut2018 %} ; {% cite Jagtap2015 %} .

To address this, we used tandem mass spectrometry (MS/MS) and bioinformatics tools on the Galaxy platform to develop a metaproteomics workflow to characterize the metaproteomes of clinical samples. This clinical metaproteomics workflow holds potential for general clinical applications such as potential secondary infections during COVID-19 infection, microbiome changes during cystic fibrosis as well as broad research questions regarding host-microbe interactions.

![Clinical Metaproteomics workflow]({% link topics/proteomics/images/clinical-mp/clinical-mp-overview.JPG %})


The first workflow for the clinical metaproteomics data analysis is the Database generation workflow. The Galaxy-P team has developed a workflow wherein a large database is generated by downloading protein sequences of known disease-causing microorganisms and then generating a compact database from the comprehensive database using the Metanovo tool.


![Database Generation Workflow]({% link topics/proteomics/images/clinical-mp/clinical-mp-database-generation.JPG %})



> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data Upload

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/10105821/files/HUMAN_SwissProt_Protein_Database.fasta
>    https://zenodo.org/records/10105821/files/Species_UniProt_FASTA.fasta
>    https://zenodo.org/records/10105821/files/Contaminants_(cRAP)_Protein_Database.fasta
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F10_9Aug19_Rage_Rep-19-06-08.mgf
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F11_9Aug19_Rage_Rep-19-06-08.mgf
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F13_9Aug19_Rage_Rep-19-06-08.mgf
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F15_9Aug19_Rage_Rep-19-06-08.mgf
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
>
> 5. Optional-Add to each database a tag corresponding to the file name.
> 6. Create a dataset collection of the 4 MGF datasets.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Import Workflow


> <hands-on-title>Running the Workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/proteomics/tutorials/clinical-mp-1-database-generation/workflows/WF1_Database_Generation_Workflow.ga" title="Database Generation" %}
>
>
> 2. Run **Workflow** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *" Input Dataset collection"*: `MGF dataset collection`
>    - {% icon param-file %} *" Species_tabular"*: `Species_tabular.tabular`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}

# Step-by-step analysis

## Download Protein Sequences using taxon names
First, we want to generate a large comprehensive protein sequence database using the UniProt XML Downloader to extract sequences for species of interest. To do so, you will need a tabular file that contains a list of species.

For this tutorial, a literature survey was conducted to obtain 118 taxonomic species of organisms that are commonly associated with the female reproductive tract {% cite AfiuniZadeh2018 %}. This species list was used to generate a protein sequence FASTA database was generated using the UniProt XML Downloader tool within the Galaxy framework. In this tutorial, the Species FASTA database (~3.38 million sequences) has already been provided as input. However, if you have your own list of species of interest as a tabular file (`Your_Species_tabular.tabular`), steps to generate a FASTA file from a tabular file are included:


## Download Species Protein Sequences using UniProt XML downloader with **UniProt**

> <hands-on-title> UniProt XML downloader</hands-on-title>
> 1. {% tool [UniProt](toolshed.g2.bx.psu.edu/repos/galaxyp/uniprotxml_downloader/uniprotxml_downloader/2.3.0) %} with the following parameters:
>    - *"Select"*: `Your_Species_tabular.tabular`
>        - {% icon param-file %} *"Dataset (tab separated) with Taxon ID/Name column"*: `output` (Input dataset)
>        - *"Column with Taxon ID/name"*: `c1`
>    - *"UniProt output format"*: `fasta`
> 2. Rename the output as Species_UniProt_FASTA.fasta
>
>    > <comment-title> UniProt description </comment-title>
>    >
>    > This tool will help download the protein fasta sequences by inputting the taxon names.
>    {: .comment}
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Can we use a higher taxonomy clade than species for the UniProt XML downloader?
> 2. Why are we using the tools separately? Can we run it all together?
> 3. Can we select multiple files together?
> 4. How many FASTA files can be merged at once, i.e. is there a limit on the number/size of files?
>
> > <solution-title></solution-title>
> >
> >
> > 1. Yes, the UniProt XML downloader can also be used for generating a database from Genus, Family, Order, or any other higher taxonomy clade.
> > 2. The tools are run separately to reduce the load on the server and tool. If you have a limited number of taxon names, then you can run it all together.
> > 3. Yes, that certainly can be done. We used one input file at a time to maintain the order of sequences in the database.
> > 4. There is no limit.
> >
> {: .solution}
>
{: .question}


## Merging databases to obtain a large comprehensive database for MetaNovo
Once generated, the Species UniProt database (~3.38 million sequences) will be merged with the Human SwissProt database (reviewed only; ~20.4K sequences) and contaminant (cRAP) sequences database (116 sequences) and filtered to generate the large comprehensive database (~2.59 million sequences). The large comprehensive database will be used to generate a compact database using MetaNovo, which is much more manageable.

> <hands-on-title> Download contaminants with **Protein Database Downloader** </hands-on-title>
>
> 1. {% tool [Protein Database Downloader](toolshed.g2.bx.psu.edu/repos/galaxyp/dbbuilder/dbbuilder/0.3.4) %} with the following parameters:
>    - *"Download from?"*: `cRAP (contaminants)`
> 2. Rename as "Protein Database Contaminants (cRAP)"
>
{: .hands_on}

> <hands-on-title> Human SwissProt (reviewed) database</hands-on-title>
> 1. {% tool [Protein Database Downloader](toolshed.g2.bx.psu.edu/repos/galaxyp/dbbuilder/dbbuilder/0.3.4) %} with the following parameters:
>    - *"Download from?"*: `UniProtKB(reviewed only)`
>        - In *"Taxonomy"*: `Homo sapiens (Human)`
>        - In *"reviewed"*: `UniProtKB/Swiss-Prot (reviewed only)`
>	 - In  *"Proteome Set"*: `Reference Proteome Set`
>	 - In  *"Include isoform data"*: `False`
> 2. Rename as "Protein Database Human SwissProt".
{: .hands_on}

> <question-title></question-title>
>
> 1. How often is the Protein Database Downloader updated?
>
>
> > <solution-title></solution-title>
> >
> >
> > 1. It is updated every 3 months.
> >
> {: .solution}
>
{: .question}


> <hands-on-title> FASTA Merge Files and Filter Unique Sequences </hands-on-title>
>
> 1. {% tool [FASTA Merge Files and Filter Unique Sequences](toolshed.g2.bx.psu.edu/repos/galaxyp/fasta_merge_files_and_filter_unique_sequences/fasta_merge_files_and_filter_unique_sequences/1.2.0) %} with the following parameters:
>    - *"Run in batch mode?"*: `Merge individual FASTAs (output collection if input is collection)`
>        - In *"Input FASTA File(s)"*:
>            - {% icon param-repeat %} *"Insert Input FASTA File(s)"*
>                - {% icon param-file %} *"FASTA File"*: `Protein Database Human SwissProt` (output of **Protein Database Downloader** {% icon tool %})
>                - {% icon param-file %} *"FASTA File"*: `Protein Database Contaminants (cRAP)` (output of **Protein Database Downloader** {% icon tool %})
>                - {% icon param-file %} *"FASTA File"*: `Species_UniProt_FASTA ` (output of **UniProt XML downloader** {% icon tool %})
>
> 2. Rename out as "Human UniProt Microbial Proteins cRAP for MetaNovo".
>
{: .hands_on}

# Reducing Database size

## Metanovo tool generates a compact database from your comprehensive database with **MetaNovo**
Next, the large comprehensive database of ~2.59 million sequences can be reduced using the MetaNovo tool to generate a more manageable database that contains identified proteins.

The compact MetaNovo-generated database (~1.9K sequences) will be merged with Human SwissProt (reviewed only) and contaminants (cRAP) databases to generate the reduced database (~21.2k protein sequences) that will be used for peptide identification (see [Discovery Module tutorial](https://github.com/subinamehta/training-material/blob/main/topics/proteomics/tutorials/clinical-mp-discovery/tutorial.md)).

> <hands-on-title> MetaNovo </hands-on-title>
>
> 1. {% tool [MetaNovo](toolshed.g2.bx.psu.edu/repos/galaxyp/metanovo/metanovo/1.9.4+galaxy4) %} with the following parameters:
>    - *"MGF Input Type"*: `Collection`
>        - {% icon param-collection %} *"MGF Collection"*: `output` (Input dataset collection)
>    - {% icon param-file %} *"FASTA File"*: `output` (output of **FASTA Merge Files and Filter Unique Sequences** {% icon tool %})
>    - In *"Spectrum Matching Parameters"*:
>        - *"Fragment ion mass tolerance"*: `0.01`
>        - *"Enzyme"*: `Trypsin (no P rule)`
>        - *"Fixed modifications as comma separated list"*: `Carbamidomethylation of C` `TMT 10-plex of K` `TMT 10-plex of peptide N-term`
>        - *"Variable modifications as comma separated list"*: `Oxidation of M`
>        - *"Maximal charge to search for"*: `5`
>    - In *"Import Filters"*:
>        - *"The maximal peptide length to consider when importing identification files"*: `50`
> 2. Rename as "MetaNovo Compact Database".
>
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Why are we reducing the size of the database?
> 2. Why is this running TMT10 plex modification when the data is 11-plex?
> 3. Regarding MetaNovo Spectrum Matching parameters, what are the most “important” parameters? Meaning, that if a user wants to reduce or increase the sensitivity/number of output sequences, what should they change?
>
> > <solution-title></solution-title>
> >
> >
> > 1. Reducing the size of the database improves search speed, FDR, and sensitivity.
> > 2. There is no option for 11-plex modifications in Metanovo, hence we use the TMT-10plex.
> > 3. The most important parameters are the tolerance (MS1 and MS2) and any modifications introduced during the processing of the data.
> >
> {: .solution}
>
{: .question}

## Merging databases to obtain reduced MetaNovo database for peptide discovery with **FASTA Merge Files and Filter Unique Sequences**

> <hands-on-title> FASTA Merge Files and Filter Unique Sequences</hands-on-title>
>
> 1. {% tool [FASTA Merge Files and Filter Unique Sequences](toolshed.g2.bx.psu.edu/repos/galaxyp/fasta_merge_files_and_filter_unique_sequences/fasta_merge_files_and_filter_unique_sequences/1.2.0) %} with the following parameters:
>    - *"Run in batch mode?"*: `Merge individual FASTAs (output collection if input is collection)`
>        - In *"Input FASTA File(s)"*:
>            - {% icon param-repeat %} *"Insert Input FASTA File(s)"*
>                - {% icon param-file %} *"FASTA File"*: `Protein Database Human SwissProt` (output of **Protein Database Downloader** {% icon tool %})
>                - {% icon param-file %} *"FASTA File"*: `Protein Database Contaminants (cRAP)` (output of **Protein Database Downloader** {% icon tool %})
>                - {% icon param-file %} *"FASTA File"*: `MetaNovo Compact Database` (output of **MetaNovo** {% icon tool %})
>
>
{: .hands_on}

# Conclusion

The first step for the Clinical Metaproteomics study is database generation. As we didn’t have a reference database or information from 16srRNA-seq data, we generated a fasta database doing a literature survey, however, if 16S rRNA data is present, the taxon identified can be used for a customized database generation. As the size of the comprehensive database is generally  too large, we used the Metanovo tool to reduce the size of the database. This reduced database will be then used for clinical metaproteomics discovery workflow.

tools_json:
["UniProt", "toolshed.g2.bx.psu.edu/repos/galaxyp/uniprotxml_downloader/uniprotxml_downloader/2.3.0", "Protein Database Downloader", "toolshed.g2.bx.psu.edu/repos/galaxyp/dbbuilder/dbbuilder/0.3.4", "FASTA Merge Files and Filter Unique Sequences", "toolshed.g2.bx.psu.edu/repos/galaxyp/fasta_merge_files_and_filter_unique_sequences/fasta_merge_files_and_filter_unique_sequences/1.2.0", "MetaNovo", "toolshed.g2.bx.psu.edu/repos/galaxyp/metanovo/metanovo/1.9.4+galaxy4"]


topic: data-science
tutorial: snakemake
title: Make & Snakemake
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: d7846bceb4d9de84c7f1d6ac7af23c17
last_modified: 2026-05-27T09:01:47.965292
zenodo_link: https://zenodo.org/record/5562251
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/snakemake/tutorial.html

content:
Here you will learn to write both Make and Snakemake workflows. We teach two workflow engines because Snakemake uses a lot of the concepts of Make, and these concepts are somewhat complex and a very different way of thinking than you might be used to with workflow design.

This tutorial is aimed at students and Galaxy community members who might want to convert Snakemake workflows into Galaxy workflows, but need to understand how Snakemake workflows work.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

We're going to go through this tutorial as "the evolution of a pipeline" from a simple bash script that we might run as individual commands on the command line, all the way up to a Snakemake workflow. This should give you some perspective of why the systems exist, and what benefits each system brings.

# Bash

We've set up a simple bash pipeline. It downloads some read files from a website, decompresses them, builds an index for a `genome.fa` we already have, and then does the alignment against that genome.

```bash
# Downloading our datasets
wget https://zenodo.org/record/5562251/files/GCA_000017985.1_ASM1798v1_genomic.fna.gz
wget https://zenodo.org/record/5562251/files/SRR2584866_1.fq.gz
wget https://zenodo.org/record/5562251/files/SRR2584866_2.fq.gz
wget https://zenodo.org/record/5562251/files/SRR2589044_1.fq.gz
wget https://zenodo.org/record/5562251/files/SRR2589044_2.fq.gz

# Generate FastQC Report
fastqc *.fq

# Run Trimmomatic to trim the bad reads out.
trimmomatic PE SRR2589044_1.fq.gz SRR2589044_2.fq.gz \
               SRR2589044_1.trim.fq SRR2589044_1un.trim.fq \
               SRR2589044_2.trim.fq SRR2589044_2un.trim.fq \
               SLIDINGWINDOW:4:20 MINLEN:25 ILLUMINACLIP:NexteraPE-PE.fa:2:40:15

trimmomatic PE SRR2584866_1.fq.gz SRR2589044_2.fq.gz \
               SRR2584866_1.trim.fq SRR2589044_1un.trim.fq \
               SRR2584866_2.trim.fq SRR2589044_2un.trim.fq \
               SLIDINGWINDOW:4:20 MINLEN:25 ILLUMINACLIP:NexteraPE-PE.fa:2:40:15

# Generate Updated FastQC Report
fastqc *.trim.fq

# Generate the genome index
bwa index GCA_000017985.1_ASM1798v1_genomic.fna

# Align reads to the genome
bwa mem GCA_000017985.1_ASM1798v1_genomic.fna SRR2589044_1.trim.fq SRR2589044_2.trim.fq | samtools sort -O bam -o SRR2589044.bam
bwa mem GCA_000017985.1_ASM1798v1_genomic.fna SRR2584866_1.trim.fq SRR2584866_2.trim.fq | samtools sort -O bam -o SRR2584866.bam
```

This is a fine start to our analysis pipeline, and might simply be summarising
the commands we executed interactively at the command line.

# Make

Make is a build automation tool that has been around since 1976. It lets you describe build processes, writing down the individual processes which will occur, and then providing you a single entrypoint to run your workflow, a lot like how {SciWMS} work! But importantly it is declarative, rather than imperative which is a big change if you're familiar with programming languages like bash or python.

Instead of defining step by step all of the steps that should be executed like the bash example above,
you instead write more general rules on how to produce individual files. We'll look at them one-by-one.

In bash scripts you write something like:
- `command input` and it implicitly creates the output file automatically, often by adding a suffix (e.g. `bwa index genome.fa` produces an index you refer to with `genome`)
- `command input > output` redirecting output of the command to a file
- `command input -O output` where we explicitly state where the output is

In Make you'll need to do that, but you'll also need to declare the inputs and outputs of every step!

<pre class="highlight"><code><span class="nb">output</span>: <span class="kt">input</span>
	<span class="s2">command [and here the correct variant from above]</span>
</code></pre>

This may seem tedious, to annotate the inputs and outputs of each step, but because Make knows what input files need to exist before it can execute this step, and it knows precisely which output files you will generate. It needs this information so that when you have multiple rules, it can decide which dependencies need to be executed.

## Line-By-Line Comparison

### Downloading Data

> <code-in-title>Bash</code-in-title>
> <pre class="highlight"><code><span class="s2">wget https://.../GCA_000017985.1_ASM1798v1_genomic.fna.gz
> wget https://.../SRR2584866_1.fq.gz</span>
> </code></pre>
> With bash we download each read file one by one
{: .code-in}
> <code-out-title>Make</code-out-title>
> <pre class="highlight"><code><span class="nb">%.gz</span>:
> 	<span class="s2">wget https://.../$@ -O $@</span>
> </code></pre>
> But with Make we write is a generic rule which can be used any time you need to download a file for this project. `$@` is used as the name of the output file and in the templated source url we will download from. If you ran `make SRR2584866_1.fq.gz` it would template out the `wget` command and run it. **Speed bonus**: It can parallelise these download jobs for us!
{: .code-out}

### FastQC

> <code-in-title>Bash</code-in-title>
> <pre class="highlight"><code><span class="s2">fastqc *.fq</span></code></pre>
> Generate ALL of the FastQC reports
{: .code-in}
> <code-out-title>Make</code-out-title>
> <pre class="highlight"><code><span class="nb">%.fastqc.html</span>: <span class="kt">%.fq</span>
> 	<span class="s2">fastqc $<</span>
> </code></pre>
> Here is a rule to generate a single FastQC report from a single FastQ file
{: .code-out}

### Trimming Data

> <code-in-title>Bash</code-in-title>
> <pre class="highlight"><code><span class="s2">trimmomatic PE</span> <span class="kt">SRR2589044_1.fq.gz SRR2589044_2.fq.gz</span> \
>                <span class="nb">SRR2589044_1.trim.fq SRR2589044_1un.trim.fq \
>                SRR2589044_2.trim.fq SRR2589044_2un.trim.fq</span> \
>                <span class="s2">SLIDINGWINDOW:4:20 MINLEN:25 ILLUMINACLIP:NexteraPE-PE.fa:2:40:15</span>
>
> <span class="s2">trimmomatic PE</span> <span class="kt">SRR2584866_1.fq.gz SRR2584866_2.fq.gz</span> \
>                <span class="nb">SRR2584866_1.trim.fq SRR2584866_1un.trim.fq \
>                SRR2584866_2.trim.fq SRR2584866_2un.trim.fq</span> \
>                <span class="s2">SLIDINGWINDOW:4:20 MINLEN:25 ILLUMINACLIP:NexteraPE-PE.fa:2:40:15</span>
> </code></pre>
> Run these two individual trimmomatic commands with these hardcoded filenames. If you need to run a new file make sure to carefully change all of the uses of it!
{: .code-in}
> <code-out-title>Make</code-out-title>
> <pre class="highlight"><code><span class="nb">%_1.trim.fq %_2.trim.fq %_1un.trim.fq %_2un.trim.fq</span>: <span class="kt">%_1.fq.gz %_2.fq.gz</span>
> 	<span class="s2">trimmomatic PE $^ \
> 		$(shell basename $(word 1,$^) .fq).trim.fq \
> 		$(shell basename $(word 1,$^) .fq)un.trim.fq \
> 		$(shell basename $(word 2,$^) .fq).trim.fq \
> 		$(shell basename $(word 2,$^) .fq)un.trim.fq \
> 		SLIDINGWINDOW:4:20 MINLEN:25 ILLUMINACLIP:NexteraPE-PE.fa:2:40:15</span>
> </code></pre>
>
> A more generic rule to generate the trimmed fastq files from whichever pair of sequence identifiers were provided. By using `%` we also get some safety in this function that we didn't have in the bash version! We know for sure those are identical values, and no one accidentally made a small typo in any one of the **6** times the same identifier was repeated. <u>But it comes at the cost</u> of using some complex Make statements like `$(shell)` which lets us execute arbitrary shell commands inside our pipeline.
>
> You'll notice that what we're really doing is extracting the inputs and outputs from the command and making them much more uniform
{: .code-out}

So when we translated from Bash to Make, every time

- we annotated inputs
- we annotated outputs
- we made the command a 'template'

> <comment-title>DO NOT REMEMBER THIS</comment-title>
> There is a lot of Make specific things going on in the above examples. You do not need to know it! We just wanted to provide working, correct examples. This tutorial is focused on Snakemake, so read these for context and understanding of why we use Snakemake, not reading them to learn Makefiles.
{: .comment}

## Why Make?

So now comes the question, why Make? Why would you want to write the rules in this declarative way, rather than the imperative way that is so much easier and requires so much less work? *Speed!* Why would you want to write these generic rules that require learning a lot of Make's syntax? *Reusability*.

Speed
:  Every time you want to run your declarative pipeline, you either need to run it from start to finish every time (re-downloading sequence data, re-building genome indicies, re-mapping reads) or to write a lot of bash code to check if those files exist and only downloading/indexing/alinging as-needed.

Reusability
:  That pipeline can only download those specific files, unless you write additional code to template out the name of the reads and genome you want to align. By writing more generic rules, as soon as someone gives you new datasets, you can immediately start processing those with your re-usable pipeline.

However with `make`, you've written generic rules which can be used to download any fastq files. And best of all, make can check if the files already exist, and if they do, it skips the step and goes on to the next step, rather than re-creating the file. Make does this by checking the

Aspect | Bash | Make
--- |--- | ---
Language style | Individual steps performed line-by-line | Generic rules are written which say how to do the operation, but not on which data
Partial re-run | You must run the entire script every time (or write extra code) | Files are only created when you ask for them
How it is invoked | `bash script.sh` | `make`
Running with different data? | Edit the script to replace the identifiers, or support templated identifiers. | `make read1111.sam read2222.sam read3333.same read4444.sam`
Paralellisation? | None by default, you must edit the script to add it. | `make -j 8` runs each build step on one thread, with 8 threads available, until all tasks are finished.
Filename | Anything ending in `.sh` | `Makefile` is the default name, and you should name your makefile this, unless you want people to have to type `make -f other-file.mk`
Dependencies | Up to you to manage | Up to you to manage
Multiple output files per step | n/a | [Very tricky to get completely right](https://www.gnu.org/software/automake/manual/html_node/Multiple-Outputs.html), Makefiles really expect one rule produces one output file (and can only check e.g. update times of a single file.)
Cluster/HPC Friendliness | Everything is manual | Everything is manual

## Backwards

A full makefile example

```
all: SRR2589044.bam

# Here we've hardcoded the genome name because it's less likely to change for a
# single pipeline than the individual data files are.
%.bam: %_1.trim.fq %_1.trim.fq GCA_000017985.1_ASM1798v1_genomic.fna.gz.bwt
	bwa mem GCA_000017985.1_ASM1798v1_genomic.fna $(word 1,$^) $(word 2,$^) | \
		samtools sort -O bam -o $@

# This indexing step however will work for any possible
%.fna.gz.bwt: %.fna.gz
	bwa index $<

# This handles ALL fastqc reporting, so we don't have to do it in two sections,
# but, unless we ask for these reports they won't be generated.
%.fastqc.html: %.fq
	fastqc $<

# This rule violates some Makefile internal expectations by having multiple
# outputs which is not handled well by all implementations
%_1.trim.fq %_2.trim.fq %_1un.trim.fq %_2un.trim.fq: %_1.fq.gz %_2.fq.gz
	trimmomatic PE $^ \
		$(shell basename $(word 1,$^) .fq).trim.fq \
		$(shell basename $(word 1,$^) .fq)un.trim.fq \
		$(shell basename $(word 2,$^) .fq).trim.fq \
		$(shell basename $(word 2,$^) .fq)un.trim.fq \
		SLIDINGWINDOW:4:20 MINLEN:25 ILLUMINACLIP:NexteraPE-PE.fa:2:40:15

# And here's finally our download step
%:
	https://zenodo.org/record/5562251/files/$(shell basename $@) \
		-O $@
```

You'll notice a couple things about this above example:

1. There is a new rule called `all`. In a makefile, by default, the very first rule is executed when you run `make`. If you want to execute other rules you can, but it defaults to the first one. By convention, many people name it `all`.
2. Inside there we've also written a file we'd like created, `SRR2589044.bam`, which doesn't exist yet. Make sees this as a dependency to finishing the (empty) all rule, and then goes on to figure out how to create it.
3. It is written backwards, we've started with what we want to output, and for each line, we figured out what we needed for that, and wrote a rule on how to create it. This is relatively common in makefiles.

You'll also notice some weird additional things we've had to do like `$(word 2,$^)` to get the second input file to that rule, this is really kind of ugly and hard to understand and is a great motivation for learning Snakemake which helps address these issues.

Make will read the above makefile like so:

1. You're running `make` so I will find the first task and run it
2. `all` is the first task and it depends on this BAM file
3. The BAM file needs me to run bwa-mem and bwa-index, in two separate steps
	1. For bwa index I need to index the genome
	2. And I need to download it first!
4. Also I need the trimmed fastq files, both forward and reverse files.
5. And I need to have the untrimmed files to trim them
6. And I need to download those first!

Notice that Make isn't running every task, it's reading the one task you asked for, and seeing what's required for that based on your annotations of inputs and outputs.

> <question-title>Check your understanding</question-title>
>
> 1. If a file is not part of the final output, or not the requested task, will it be created?
> 2. Do you spot any omissions in the above pipeline?
>
> > <solution-title></solution-title>
> > 1. No, Make only runs the tasks for files it needs, it won't run any of the other tasks. This is part of what makes `make` fast and easily parallelisable.
> > 2. Yes! FastQC is missing, it won't be created unless we ask for it.
> {: .solution}
{: .question}

Reading the above you should be able to imagine a tree of tasks that Make is creating internally:

TODO

This is called a {DAG}, it is a graph of nodes (tasks that need to be executed), with connections between nodes that have a direction (this task depends on outputs of that task), and there are no cycles (no outputs depend on inputs.) These are very common in {SciWMS}s because they make computation faster. Instead of executing step-by-step, we can build this graph of what work needs to be done, and then starting with the leaves of the graph (the end nodes without other dependencies) we can start executing and removing them.

This also how we can really easily parallelise workflows: because we know the dependencies of each step, we know which can be executed right now, and we can execute them in parallel because we know for sure they do not depend on each other.

# Snakemake

> The Snakemake workflow management system is a tool to create reproducible and scalable data analyses. Workflows are described via a human readable, Python based language. They can be seamlessly scaled to server, cluster, grid and cloud environments, without the need to modify the workflow definition. Finally, Snakemake workflows can entail a description of required software, which will be automatically deployed to any execution environment.
{: .quote cite="https://snakemake.readthedocs.io/en/stable/index.html"}

Snakemake addresses a lot of the issues with make for use in scientific contexts: clearer pipelines and dependencies. We did not talk about it in the previous section, but where did `bowtie2` and `bowtie2-build` come from? How did those get installed? What versions are they? None of that information is included in the Makefile

Snakemake rules are a bit more complex, in Snakemake you will write rules that follow this form:

<pre class="highlight"><code>rule my-rule-name
	<span class="kt">input:
		"something",
		"something-else"</span>
	<span class="nb">output:
		"output-1",
		"output-2"</span>
	<span class="s2">shell:
		"cat {input} > {output}"</span>
</code></pre>

## Line-By-Line Comparison

### Downloading Data

> <code-in-title>Make</code-in-title>
> <pre class="highlight"><code><span class="nb">%.fq.gz</span>:
> 	<span class="s2">wget https://zenodo.org/record/5562251/files/$@</span>
> </code></pre>
>
> Generic download rule, the <code>$@</code> and <code>%</code> used are a bit opaque, you need to know what they mean to understand how the rule works.
{: .code-in}
> <code-out-title>Snakemake</code-out-title>
> <pre class="highlight"><code>rule download:
> 	<span class="nb">output:
> 		"{sample}.fq.gz"</span>
> 	<span class="s2">shell:
> 		"wget https://zenodo.org/record/5562251/files/{wildcards.sample}.fq.gz -O {output}"</span>
> </code></pre>
>
> This is much more explicit, the outputs are listed and `{sample}` is used as the variable to be templated out, a lot like you might recognise from Python's `format` function or `f""` strings. The rule also has a name which serves as really nice documentation for what that step does, you don't have to read the command to figure it out.
{: .code-out}

### FastQC

> <code-in-title>Make</code-in-title>
> <pre class="highlight"><code><span class="nb">%.fastqc.html</span>: <span class="kt">%.fq</span>
> 	<span class="s2">fastqc $<</span>
> </code></pre>
>
> Here is a rule to generate a single FastQC report from a single FastQ file
{: .code-in}
> <code-out-title>Snakemake</code-out-title>
> <pre class="highlight"><code>rule fastqc:
> 	<span class="kt">input:
> 		"{sample}.fq.gz"</span>
> 	<span class="nb">output:
> 		"{sample}_fastqc.html"</span>
> 	<span class="s2">shell:
> 		"fastqc {input} --outdir fastqc/"</span>
> </code></pre>
>
> Essentially the same, but now we've also added a Conda environment in which our job will run. This makes dependency management a lot simpler.
{: .code-out}


### Trimming Data


> <code-in-title>Make</code-in-title>
> <pre class="highlight"><code><span class="nb">%_1.trim.fq %_2.trim.fq %_1un.trim.fq %_2un.trim.fq</span>: <span class="kt">%_1.fq.gz %_2.fq.gz</span>
> 	<span class="s2">trimmomatic PE $^ \
> 		$(shell basename $(word 1,$^) .fq).trim.fq \
> 		$(shell basename $(word 1,$^) .fq)un.trim.fq \
> 		$(shell basename $(word 2,$^) .fq).trim.fq \
> 		$(shell basename $(word 2,$^) .fq)un.trim.fq \
> 		SLIDINGWINDOW:4:20 MINLEN:25 ILLUMINACLIP:NexteraPE-PE.fa:2:40:15</span>
> </code></pre>
>
> Here we take our very complicated and hard to understand Make rule (shell? basename? word?) with ugly and potentially quite broken multiple output syntax
>
{: .code-in}
> <code-out-title>Snakemake</code-out-title>
> <pre class="highlight"><code>rule trimmomatic:
> 	<span class="kt">input:
> 		r1="{sample}_1.fq.gz",
> 		r2="{sample}_2.fq.gz"</span>
> 	<span class="nb">output:
> 		o1="{sample}_1.trim.fq",
> 		o2="{sample}_2.trim.fq",
> 		o1un="{sample}_1un.trim.fq",
> 		o2un="{sample}_2un.trim.fq"</span>
> 	<span class="s2">shell:
> 		"trimmomatic PE "
> 		"{input.r1} {input.r2} "
> 		"{output.o1} {output.o1un} "
> 		"{output.o2} {output.o2un} "
> 		"SLIDINGWINDOW:4:20 MINLEN:25 "
> 		"ILLUMINACLIP:NexteraPE-PE.fa:2:40:15"</span>
> </code></pre>
>
> And turn it into a much more readable and clear Snakemake step! Now instead of using more opaque terms like `$(word 1,$^)` we can just declare "our output should have this name and use {sample} as part of the name" and Snakemake takes care of the rest. Then in our commandline we can clearly reference exactly what we want.
>
> Note: We also use `fq.gz` as our input file because trimmomatic we know can accept gzipped fastq files, but we output plain fastq files because re-compressing those is another step and it's not necessary right now for our small datasets. In a real world situation you might also choose not to compress intermediate datasets because you want better performance and you know you'll throw away the intermediates and keep the source and final datasets.
{: .code-out}

Now that you have seen a few rules, let's write the rest.

> <hands-on-title>Save this Snakefile to the filesystem</hands-on-title>
> From here on you are going to finish writing this Snakefile on your own! We'll give you the bits you have seen up until now, and future tasks will require you to add your own rules and get to actually test out the pipeline!
>
> ```Snakemake
> rule download:
> 	output:
> 		"{sample}.fq.gz"
> 	shell:
> 		"wget https://zenodo.org/record/5562251/files/{wildcards.sample}.fq.gz -O {output}"
>
> rule download_genome:
> 	output:
> 		"GCA_000017985.1_ASM1798v1_genomic.fna.gz"
> 	shell:
> 		"wget https://zenodo.org/record/5562251/files/GCA_000017985.1_ASM1798v1_genomic.fna.gz -O {output}"
>
> rule fastqc:
> 	input:
> 		"{sample}.fq.gz"
> 	output:
> 		"{sample}_fastqc.html"
> 	shell:
> 		"fastqc {input} --outdir fastqc/"
>
> rule trimmomatic:
> 	input:
> 		r1="{sample}_1.fq.gz",
> 		r2="{sample}_2.fq.gz"
> 	output:
> 		o1="{sample}_1.trim.fq",
> 		o2="{sample}_2.trim.fq",
> 		o1un="{sample}_1un.trim.fq",
> 		o2un="{sample}_2un.trim.fq"
> 	shell:
> 		"trimmomatic PE "
> 		"{input.r1} {input.r2} "
> 		"{output.o1} {output.o1un} "
> 		"{output.o2} {output.o2un} "
> 		"SLIDINGWINDOW:4:20 MINLEN:25 "
> 		"ILLUMINACLIP:NexteraPE-PE.fa:2:40:15"
> ```
{: .hands_on}

> <hands-on-title>Install Snakemake</hands-on-title>
> We're about to start doing things really with snakemake, so, it's time to install it.
> 1. [Install Miniconda](https://docs.conda.io/en/latest/miniconda.html)
> 2. Create an environment for Snakemake:
>
>    ```bash
>    conda create -n snakemake
>    ```
>
> 3. Activate it
>
>    ```bash
>    conda activate snakemake
>    ```
>
> 4. And install snakemake in this environment
>
>    ```bash
>    conda install snakemake=6.10.0
>    ```
>
{: .hands_on}


> <hands-on-title>Try running snakemake!</hands-on-title>
>
> 1. Try running snakemake
>
>    > <code-in-title>CLI</code-in-title>
>    > ```
>    > snakemake -c1
>    > ```
>    {: .code-in}
>
>    > <tip-title>"specify the maximum number of jobs"</tip-title>
>    > If you see an error like this it might be due to an outdated version of snakemake
>    > ```
>    > Error: you need to specify the maximum number of jobs to be queued or executed at the same time with --jobs.
>    > ```
>    {: .tip}
>
>    > <code-out-title></code-out-title>
>    > ```
>    > Building DAG of jobs...
>    > WorkflowError:
>    > Target rules may not contain wildcards. Please specify concrete files or a rule without wildcards at the command line, or have a rule without wildcards at the very top of your workflow (e.g. the typical "rule all" which just collects all results you want to generate in the end).
>    > ```
>    {: .code-out}
>
>    Wait, that didn't work! This is the `all` rule we also saw in the makefile. But we can use Snakemake to build individual rule outputs, so let's try that now.
>
> 2. Run snakemake to download `SRR2584866_1.fq.gz` and `SRR2584866_2.fq.gz`
>
>    > <code-in-title>CLI</code-in-title>
>    > ```
>    > snakemake -c1 SRR2584866_1.fq.gz SRR2584866_2.fq.gz
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```
>    > Building DAG of jobs...
>    > Using shell: /usr/bin/bash
>    > Provided cores: 1 (use --cores to define parallelism)
>    > Rules claiming more threads will be scaled down.
>    > Conda environments: ignored
>    > Job stats:
>    > job         count    min threads    max threads
>    > --------  -------  -------------  -------------
>    > download        2              1              1
>    > total           2              1              1
>    >
>    > Select jobs to execute...
>    >
>    > [Fri Oct  8 16:06:32 2021]
>    > rule download:
>    >     output: SRR2584866_2.fq.gz
>    >     jobid: 1
>    >     wildcards: sample=SRR2584866_2
>    >     resources: tmpdir=/tmp
>    >
>    > --2021-10-08 16:06:32--  https://zenodo.org/record/5562251/SRR2584866_2.fq.gz
>    > Resolving localhost (localhost)... 127.0.0.1
>    > Connecting to localhost (localhost)|127.0.0.1|:8000... connected.
>    > HTTP request sent, awaiting response... 200 OK
>    > Length: 6319969 (6,0M) [application/gzip]
>    > Saving to: ‘SRR2584866_2.fq.gz’
>    >
>    > 2021-10-08 16:06:32 (538 MB/s) - ‘SRR2584866_2.fq.gz’ saved [6319969/6319969]
>    >
>    > [Fri Oct  8 16:06:32 2021]
>    > Finished job 1.
>    > 1 of 2 steps (50%) done
>    > Select jobs to execute...
>    >
>    > [Fri Oct  8 16:06:32 2021]
>    > rule download:
>    >     output: SRR2584866_1.fq.gz
>    >     jobid: 0
>    >     wildcards: sample=SRR2584866_1
>    >     resources: tmpdir=/tmp
>    >
>    > --2021-10-08 16:06:32--  https://zenodo.org/record/5562251/SRR2584866_1.fq.gz
>    > Resolving localhost (localhost)... 127.0.0.1
>    > Connecting to localhost (localhost)|127.0.0.1|:8000... connected.
>    > HTTP request sent, awaiting response... 200 OK
>    > Length: 6586718 (6,3M) [application/gzip]
>    > Saving to: ‘SRR2584866_1.fq.gz’
>    >
>    > 2021-10-08 16:06:32 (537 MB/s) - ‘SRR2584866_1.fq.gz’ saved [6586718/6586718]
>    >
>    > [Fri Oct  8 16:06:32 2021]
>    > Finished job 0.
>    > 2 of 2 steps (100%) done
>    > Complete log: /tmp/snake.q5mqtdhfg6/.snakemake/log/2021-10-08T160632.213680.snakemake.log
>    > ```
>    {: .code-out.code-max-300}
>
> 3. Check that it worked. Do you see files in your directory?
{: .hands_on}


Now that we've got our pipeline started, let's do some more with it!

> <question-title></question-title>
>
> How would you write the following task in Snakemake?
>
> The command is
> <pre class="highlight"><code><span class="s2">bwa index</span> <span class="kt">GCA_000017985.1_ASM1798v1_genomic.fna</span></code></pre>
> and it creates `GCA_000017985.1_ASM1798v1_genomic.fna.gz.bwt`
>
> > <solution-title></solution-title>
> >
> > <pre class="highlight"><code>rule indexgenome:
> > 	<span class="kt">input:
> > 		"GCA_000017985.1_ASM1798v1_genomic.fna.gz"</span>
> > 	<span class="nb">output:
> > 		"GCA_000017985.1_ASM1798v1_genomic.fna.gz.bwt"</span>
> > 	<span class="s2">shell:
> > 		"bwa index {input}"</span>
> > </code></pre>
> {: .solution}
{: .question}

> <question-title></question-title>
>
> The command is `bwa mem GCA_000017985.1_ASM1798v1_genomic.fna SRR2584866_1.trim.fq SRR2584866_2.trim.fq | samtools sort -O bam -o SRR2584866.bam`
>
> 1. What are the inputs?
> 2. What are the outputs?
> 3.  How would you write the following task in Snakemake?
>
>
> > <solution-title></solution-title>
> >
> > 1. `GCA_000017985.1_ASM1798v1_genomic.fna.gz.bwt`, the index file (but beware it does not get passed in as-is, the indexing tool expects just the `GCA_000017985.1_ASM1798v1_genomic.fna` portion.)
> >
> >    Also we have our two sequence files `SRR2584866_1.trim.fq SRR2584866_2.trim.fq`
> >
> > 2. Our output is `SRR2584866.bam`
> >
> > 3. There are a couple of options we have here, we can supply both the `.fna` and the `.fna.gz.bwt` file as inputs (not strictly true, we don't need the fasta file) and then just use the `fna` file in the command line, or we can pass in just the `.fna.gz.bwt` file and try and calculate the `.fna` version that is expected as the index name. We will show the second option as it is more complicated.
> > <pre class="highlight"><code>rule align:
> > 	<span class="kt">input:
> > 		r1="{sample}_1.trim.fq",
> > 		r2="{sample}_1.trim.fq",
> > 		index="{genome}.fna.gz.bwt"</span>
> > 	<span class="nb">output:
> > 		"{genome}/{sample}.bam"</span>
> > 	<span class="s2">shell:
> > 		"bwa mem {wildcards.genome}.fna {input.r1} {input.r2} | "
> > 		"samtools sort -O bam -o {output}"</span>
> > </code></pre>
> >
> > Here we used a number of features to accomplish what we need, and we'll now go through them. First is [Wildcards](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#wildcards) which can be used to take a portion of the output name or a portion of the input name and to re-use that in the command line. Here we declared that the first part of the index name up to `.fna.gz.bwt` was going to be the `genome` wildcard.
> >
> > Importantly, we also used this in our output. What would have happened if we didn't? It would be unresolvable! We would run `snakemake ... output.bam` and it would say "I don't know what value genome should be set to", so we need to have that value somewhere in our output filename in order to be able to figure that out.
> >
> > That isn't the only way to solve that problem, we could also hardcode this or write it in a [config file](https://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html) that is used by snakemake.
> {: .solution}
{: .question}

> <hands-on-title>Add the above outputs to your Pipeline</hands-on-title>
> If you haven't already, add the above outputs to your Snakemake pipeline. You should now have a few rules:
>
> - download
> - fastqc
> - trimmomatic
> - indexgenome
> - align
{: .hands_on}

## Best Practices

But this was our very first attempt at a workflow, so what might a best practice workflow look like?

### Conda for reproducibility

If you're not already using conda, you should be! Much scientific software you might be interested in using is already in there, probably mostly provided by the [BioConda](https://anaconda.org/bioconda/repo) repository. Writing an environment file can be pretty simple, you just need a file like this, which we customarily put in a folder named `envs/`

> <code-in-title>`envs/bwa.yaml`</code-in-title>
> ```yaml
> channels:
>   - conda-forge
>   - bioconda
>   - defaults
> dependencies:
>   - bwa=0.7.17
>   - samtools=1.13
> ```
{: .code-in}

In the above code sample you can see `bwa=0.7.17`, that's the version of the `bwa` mapper we want to use which was found on [the conda repository page](https://anaconda.org/bioconda/bwa). You'll notice that after it is `=h5bf99c6_8`, this is a specific revision of the package that has been 'pinned' (i.e. we want to ensure conda installs specifically the version we know works.) That step is not necessary, but if you really care about reproducibility it can be good in a pipeline

> <hands-on-title>Add conda</hands-on-title>
> Now you should add Conda environments where appropriate. But where is appropriate? Unix built in tools (wget, curl, gzip, cat, head, tail) generally do not need to be in conda, because they're very standard across all unix environments.
>
> Bioinformatics tools on the other hand, these need conda envs.
>
> Try and add these yourself, and check your work below.
>
> Please use the following versions:
>
> Package     | Version
> ---         | --
> fastqc      | 0.11.9
> trimmomatic | 0.39
> bwa         | 0.7.17
> samtools    | 1.13
>
> *Hints*
> 1. see above for what a conda environment looks like.
> 2. You'll need to install both `bwa` and `samtools` into the environment for the alignment step.
> 3. Create the `envs/` directory if it does not exist.
>
> > <solution-title></solution-title>
> >
> > `envs/bwa.yaml` should look like this:
> >
> > ```yaml
> > channels:
> >   - conda-forge
> >   - bioconda
> >   - defaults
> > dependencies:
> >   - bwa=0.7.17
> >   - samtools=1.13
> > ```
> >
> > And here is the `envs/fastqc.yaml`:
> >
> > ```yaml
> > channels:
> >   - conda-forge
> >   - bioconda
> >   - defaults
> > dependencies:
> >   - fastqc=0.11.9
> > ```
> >
> > And here is the `envs/trimmomatic.yaml`:
> >
> > ```yaml
> > channels:
> >   - conda-forge
> >   - bioconda
> >   - defaults
> > dependencies:
> >   - trimmomatic=0.39
> > ```
> >
> >
> > > <comment-title>How to read diffs</comment-title>
> > > This is a 'diff', it shows you the difference between two versions of a text file. Everything added is highlighed in light blue. Anything deleted is shown in black with a strikethrough. Importantly it shows you the context, the bits that are not highlighted, and this helps you know where the changes should go.
> > > The first line shows the original file, and the second line shows the new file. If the names are different, it means the file has been renamed.
> > >
> > > The `@@ ... @@` line shows the position in the file and how that changes, and often it will also show you the name of the function or in the case of Snakemake, the name of the rule that change appears in.
> > {: .comment}
> >
> > ```diff
> > --- a/Snakefile
> > +++ b/Snakefile
> > @@ -9,6 +9,8 @@ rule fastqc:
> >  		"{sample}.fq.gz"
> >  	output:
> >  		"{sample}_fastqc.html"
> > +	conda:
> > +		"envs/fastqc.yaml"
> >  	shell:
> >  		"fastqc {input} --outdir fastqc/"
> >
> > @@ -21,6 +23,8 @@ rule trimmomatic:
> >  		o2="{sample}_2.trim.fq",
> >  		o1un="{sample}_1un.trim.fq",
> >  		o2un="{sample}_2un.trim.fq"
> > +	conda:
> > +		"envs/trimmomatic.yaml"
> >  	shell:
> >  		"trimmomatic PE "
> >  		"{input.r1} {input.r2} "
> > @@ -34,6 +38,8 @@ rule indexgenome:
> >  		"GCA_000017985.1_ASM1798v1_genomic.fna.gz"
> >  	output:
> >  		"GCA_000017985.1_ASM1798v1_genomic.fna.gz.bwt"
> > +	conda:
> > +		"envs/bwa.yaml"
> >  	shell:
> >  		"bwa index {input}"
> >
> > @@ -44,6 +50,8 @@ rule align:
> >  		index="{genome}.fna.gz.bwt"
> >  	output:
> >  		"{genome}/{sample}.bam"
> > +	conda:
> > +		"envs/bwa.yaml"
> >  	shell:
> >  		"bwa mem {wildcards.genome}.fna {input.r1} {input.r2} | "
> >  		"samtools sort -O bam -o {output}"
> > ```
> >
> > Here in the last two steps we used a single environment, this is a technical decision we made. We could have used a single environment for every step, but with conda the more packages you add, the more complicated it is for Conda to find a version of all of those packages that work together OK. So by isolating packages to single environments, you speed up the installation process. But in two out of four steps above, we have used a shared environment. This is because `bwa` was required in both, and one just needs the additional `samtools` package. For this there is no real technical reason, just a feeling of "it's probably ok and won't be too difficult to resolve".
> {: .solution}
{: .hands_on}

### Log Files

Saving log files is key to making sure that you have a complete log of the execution of software. If you don't save the log files, and you're running a pipeline with 100 samples, it can be very easy to get confused about what went wrong and where. Unfortunately this is not a completely automatic process, and you'll need to manually configure it:

> <code-in-title>`Snakefile`</code-in-title>
> ```
> rule indexgenome:
> 	input:
> 		"{sample}.fna.gz"
> 	output:
> 		"{sample}.fna.gz.bwt"
> 	conda:
> 		"envs/bwa.yaml"
> 	log:
> 		out="logs/bwa.index.{sample}.out",
> 		err="logs/bwa.index.{sample}.err"
> 	shell:
> 		"bwa index {input} >{log.out} 2>{log.err}"
> ```
{: .code-in}

In the above rule we setup an `out` and `err` log file because every program that executes on Linux has two output streams known as `stderr` and `stdout`. By default these are just printed to your screen, and can be mixed together which is unhelpful when your program writes a lot of logging messages and you have to dig through it to find the one error message. So we setup the `out` and `err` files in a `logs/` directory with the name of the tool that will be executed as well.

Additionally we need to change the `shell` step to redirect `stdout` and `stderr` to the appropriate files. Please see [this bash tutorial](https://wiki.bash-hackers.org/howto/redirection_tutorial) for more information on how redirecting outputs works.

But **don't just copy/paste** the above example because:

- Some tools produce output on stdout, e.g. `bwa mem .. | samtools`, there `bwa mem` is writing to `stdout` which is getting piped to `samtools`, so, you should not redirect its output into the logging file or your pipeline will break. There you would only want to redirect `2>{log.err}`.
- Sometimes you have multiple commands in a pipeline, these all should go to different log files or you will have more trouble figuring out where your error came from.


> <hands-on-title>Add log files to every step</hands-on-title>
>
> Try and add log files everywhere (err and out where appropriate) to all of your rules. And use them in the `shell` sections as well. Also put your logs under the `logs/` folder!
>
> > <solution-title></solution-title>
> >
> > This is again a diff, things in blue were added to the file named at the top, things with black highlight were deleted. Think "track changes" mode in Google Docs or Word, except harder to read.
> >
> > ```diff
> > --- a/Snakefile
> > +++ b/Snakefile
> > @@ -1,14 +1,20 @@
> >  rule download:
> >  	output:
> >  		"{sample}.fq.gz"
> > +	log:
> > +		out="logs/download.{sample}.out",
> > +		err="logs/download.{sample}.err"
> >  	shell:
> > -		"wget https://zenodo.org/record/5562251/files/{wildcards.sample}.fq.gz -O {output}"
> > +		"wget https://zenodo.org/record/5562251/files/{wildcards.sample}.fq.gz -O {output} >{log.out} 2>{log.err}"
> >
> >  rule download_genome:
> >  	output:
> >  		"GCA_000017985.1_ASM1798v1_genomic.fna.gz"
> > +	log:
> > +		out="logs/download.out",
> > +		err="logs/download.err"
> >  	shell:
> > -		"wget https://zenodo.org/record/5562251/files/GCA_000017985.1_ASM1798v1_genomic.fna.gz -O {output}"
> > +		"wget https://zenodo.org/record/5562251/files/GCA_000017985.1_ASM1798v1_genomic.fna.gz -O {output} >{log.out} 2>{log.err}"
> >
> >  rule fastqc:
> >  	input:
> > @@ -17,8 +23,11 @@ rule fastqc:
> >  		"{sample}_fastqc.html"
> >  	conda:
> >  		"envs/fastqc.yaml"
> > +	log:
> > +		out="logs/fastqc.{sample}.out",
> > +		err="logs/fastqc.{sample}.err"
> >  	shell:
> > -		"fastqc {input} --outdir fastqc/"
> > +		"fastqc {input} --outdir fastqc/ >{log.out} 2>{log.err}"
> >
> >  rule trimmomatic:
> >  	input:
> > @@ -31,13 +40,16 @@ rule trimmomatic:
> >  		o2un="{sample}_2un.trim.fq"
> >  	conda:
> >  		"envs/trimmomatic.yaml"
> > +	log:
> > +		out="logs/trimmomatic.{sample}.out",
> > +		err="logs/trimmomatic.{sample}.err"
> >  	shell:
> >  		"trimmomatic PE "
> >  		"{input.r1} {input.r2} "
> >  		"{output.o1} {output.o1un} "
> >  		"{output.o2} {output.o2un} "
> >  		"SLIDINGWINDOW:4:20 MINLEN:25 "
> > -		"ILLUMINACLIP:NexteraPE-PE.fa:2:40:15"
> > +		"ILLUMINACLIP:NexteraPE-PE.fa:2:40:15 >{log.out} 2>{log.err}"
> >
> >  rule indexgenome:
> >  	input:
> > @@ -46,8 +58,11 @@ rule indexgenome:
> >  		"GCA_000017985.1_ASM1798v1_genomic.fna.gz.bwt"
> >  	conda:
> >  		"envs/bwa.yaml"
> > +	log:
> > +		out="logs/bwa.index.out",
> > +		err="logs/bwa.index.err"
> >  	shell:
> > -		"bwa index {input}"
> > +		"bwa index {input} >{log.out} 2>{log.err}"
> >
> >  rule align:
> >  	input:
> > @@ -58,6 +73,10 @@ rule align:
> >  		"{genome}/{sample}.bam"
> >  	conda:
> >  		"envs/bwa.yaml"
> > +	log:
> > +		bwaerr="logs/bwa.{genome}.{sample}.err",
> > +		out="logs/samtools.{genome}.{sample}.out",
> > +		err="logs/samtools.{genome}.{sample}.err"
> >  	shell:
> > -		"bwa mem {wildcards.genome}.fna {input.r1} {input.r2} | "
> > -		"samtools sort -O bam -o {output}"
> > +		"bwa mem {wildcards.genome}.fna {input.r1} {input.r2} 2>{log.bwaerr} | "
> > +		"samtools sort -O bam -o {output} >{log.out} 2>{log.err}"
> > ```
> >
> > The last one is the most complicated. Here we have `bwa mem` which writes to `stdout`. We know that because a `|` character comes next which indicates we're piping the output from one program to another, so there we want to store the error. However `samtools` has a `-o {output}` so we can infer from this that it writes to that output file by name. Anything it writes on `stdout` or `stderr` migth be important.
> {: .solution}
{: .hands_on}



### Use Folders

You've seen a couple examples above but it's best to use folders to help keep your data organised. Separate individual pipeline steps into different folders so you can more easily keep track of e.g. whether you're working with trimmed or untrimmed data.

> <question-title></question-title>
> What sort of folders would you establish for the example pipeline in this tutorial?
>
> > <solution-title></solution-title>
> >
> > One possible solution you could consider is:
> >
> > Folder | Purpose
> > --- | ---
> > envs | Conda environment files
> > reads | Your sequencing data
> > trim | Your sequences after they have been trimmed and cleaned
> > fastqc | The FastQC reports
> > alignments | All of your final output alignments
> > reference | Any reference data, indicies, etc.
> {: .solution}
{: .question}

> <hands-on-title>Do it!</hands-on-title>
> Update your pipeline to have all of datasets stored nicely in folders.
>
> > <solution-title></solution-title>
> >
> > ```diff
> > diff --git a/Snakefile b/Snakefile
> > index 59b983c..31ebdff 100644
> > --- a/Snakefile
> > +++ b/Snakefile
> > @@ -1,6 +1,6 @@
> >  rule download:
> >  	output:
> > -		"{sample}.fq.gz"
> > +		"reads/{sample}.fq.gz"
> >  	log:
> >  		out="logs/download.{sample}.out",
> >  		err="logs/download.{sample}.err"
> > @@ -9,7 +9,7 @@ rule download:
> >
> >  rule download_genome:
> >  	output:
> > -		"GCA_000017985.1_ASM1798v1_genomic.fa.gz"
> > +		"reference/GCA_000017985.1_ASM1798v1_genomic.fa.gz"
> >  	log:
> >  		out="logs/download.{sample}.out",
> >  		err="logs/download.{sample}.err"
> > @@ -18,9 +18,9 @@ rule download_genome:
> >
> >  rule fastqc:
> >  	input:
> > -		"{sample}.fq"
> > +		"reads/{sample}.fq.gz"
> >  	output:
> > -		"{sample}_fastqc.html"
> > +		"fastqc/{sample}_fastqc.html"
> >  	conda:
> >  		"envs/fastqc.yaml"
> >  	log:
> > @@ -31,13 +31,13 @@ rule fastqc:
> >
> >  rule trimmomatic:
> >  	input:
> > -		r1="{sample}_1.fq",
> > -		r2="{sample}_2.fq"
> > +		r1="reads/{sample}_1.fq",
> > +		r2="reads/{sample}_2.fq"
> >  	output:
> > -		o1="{sample}_1.trim.fq",
> > -		o2="{sample}_2.trim.fq",
> > -		o1un="{sample}_1un.trim.fq",
> > -		o2un="{sample}_2un.trim.fq"
> > +		o1="trim/{sample}_1.trim.fq",
> > +		o2="trim/{sample}_2.trim.fq",
> > +		o1un="trim/{sample}_1un.trim.fq",
> > +		o2un="trim/{sample}_2un.trim.fq"
> >  	conda:
> >  		"envs/trimmomatic.yaml"
> >  	log:
> > @@ -53,9 +53,9 @@ rule trimmomatic:
> >
> >  rule indexgenome:
> >  	input:
> > -		"GCA_000017985.1_ASM1798v1_genomic.fna"
> > +		"reference/GCA_000017985.1_ASM1798v1_genomic.fa"
> >  	output:
> > -		"GCA_000017985.1_ASM1798v1_genomic.fna.bwt"
> > +		"reference/GCA_000017985.1_ASM1798v1_genomic.fa.bwt"
> >  	conda:
> >  		"envs/bwa.yaml"
> >  	log:
> > @@ -66,11 +66,11 @@ rule indexgenome:
> >
> >  rule align:
> >  	input:
> > -		r1="{sample}_1.trim.fq",
> > -		r2="{sample}_1.trim.fq",
> > -		index="{genome}.fna.bwt"
> > +		r1="trim/{sample}_1.trim.fq",
> > +		r2="trim/{sample}_1.trim.fq",
> > +		index="reference/{genome}.fna.bwt"
> >  	output:
> > -		"{genome}/{sample}.bam"
> > +		"alignments/{genome}/{sample}.bam"
> >  	conda:
> >  		"envs/bwa.yaml"
> >  	log:
> > @@ -78,5 +78,5 @@ rule align:
> >  		out="logs/samtools.{genome}.{sample}.out",
> >  		err="logs/samtools.{genome}.{sample}.err"
> >  	shell:
> > -		"bwa mem {wildcard.genome}.fna {input.r1} {input.r2} 2>{log.bwaerr} | "
> > +		"bwa mem reference/{wildcards.genome}.fna.gz {input.r1} {input.r2} 2>{log.bwaerr} | "
> >  		"samtools sort -O bam -o {output} >{log.out} 2>{log.err}"
> > ```
> {: .solution}
{: .hands_on}

Notice how while updating data location, we only had to update the input and output boxes of the Snakefile, we didn't need to change any commands because there we used `{input}` or `{output}` which are automatically templated for us. Except for the last command where we had to use a wildcard and manually construct the path to work around the issue of 'fake' inputs where the tool expected a filename that was the common subset of all of the files it would produce.


### Set a Default Task

In the Makefile we had an `all` rule which was the first and default action to take. Let's reproduce that.

> <hands-on-title>Add an all task</hands-on-title>
> It should build `alignments/GCA_000017985.1_ASM1798v1_genomic/SRR2584863.bam`
>
> > <solution-title></solution-title>
> > We add this to the very top of our file.
> > ```diff
> > --- a/Snakefile
> > +++ b/Snakefile
> > @@ -1,3 +1,7 @@
> > +rule all:
> > +	input:
> > +		"alignments/GCA_000017985.1_ASM1798v1_genomic/SRR2584863.bam"
> > +
> >  rule download:
> >  	output:
> >  		"reads/{sample}.fq.gz"
> > ```
> >
> > > <tip-title>Why Input and not Output?</tip-title>
> > > Because rule all is taking in some other outputs as an input to it. If we used outputs, rule all would do nothing (it has no pre-requisite steps), and since it produces nothing by itself, snakemake would see that the outputs you expected have not been created and flag it as an error.
> > {: .tip}
> >
> {: .solution}
{: .hands_on}



## Final Pipeline

This is starting to look like a pretty good workflow! Let's preview how it will run:


> <code-in-title></code-in-title>
> ```
> $ snakemake -np
> ```
{: .code-in}
> <code-out-title></code-out-title>
> ```
> Building DAG of jobs...
> Job stats:
> job                count    min threads    max threads
> ---------------  -------  -------------  -------------
> align                  1              1              1
> all                    1              1              1
> download               2              1              1
> download_genome        1              1              1
> indexgenome            1              1              1
> trimmomatic            1              1              1
> total                  7              1              1
>
>
> [Fri Oct  8 16:59:39 2021]
> rule download:
>     output: reads/SRR2584863_2.fq.gz
>     log: logs/download.SRR2584863_2.out, logs/download.SRR2584863_2.err
>     jobid: 4
>     wildcards: sample=SRR2584863_2
>     resources: tmpdir=/tmp
>
> wget https://zenodo.org/record/5562251/SRR2584863_2.fq.gz -O reads/SRR2584863_2.fq.gz >logs/download.SRR2584863_2.out 2>logs/download.SRR2584863_2.err
>
> [Fri Oct  8 16:59:39 2021]
> rule download:
>     output: reads/SRR2584863_1.fq.gz
>     log: logs/download.SRR2584863_1.out, logs/download.SRR2584863_1.err
>     jobid: 3
>     wildcards: sample=SRR2584863_1
>     resources: tmpdir=/tmp
>
> wget https://zenodo.org/record/5562251/SRR2584863_1.fq.gz -O reads/SRR2584863_1.fq.gz >logs/download.SRR2584863_1.out 2>logs/download.SRR2584863_1.err
>
> [Fri Oct  8 16:59:39 2021]
> rule download_genome:
>     output: reference/GCA_000017985.1_ASM1798v1_genomic.fna.gz
>     log: logs/download.out, logs/download.err
>     jobid: 6
>     resources: tmpdir=/tmp
>
> wget https://zenodo.org/record/5562251/GCA_000017985.1_ASM1798v1_genomic.fna.gz -O reference/GCA_000017985.1_ASM1798v1_genomic.fna.gz >logs/download.out 2>logs/download.err
>
> [Fri Oct  8 16:59:39 2021]
> rule trimmomatic:
>     input: reads/SRR2584863_1.fq.gz, reads/SRR2584863_2.fq.gz
>     output: trim/SRR2584863_1.trim.fq, trim/SRR2584863_2.trim.fq, trim/SRR2584863_1un.trim.fq, trim/SRR2584863_2un.trim.fq
>     log: logs/trimmomatic.SRR2584863.out, logs/trimmomatic.SRR2584863.err
>     jobid: 2
>     wildcards: sample=SRR2584863
>     resources: tmpdir=/tmp
>
> trimmomatic PE reads/SRR2584863_1.fq.gz reads/SRR2584863_2.fq.gz trim/SRR2584863_1.trim.fq trim/SRR2584863_1un.trim.fq trim/SRR2584863_2.trim.fq trim/SRR2584863_2un.trim.fq SLIDINGWINDOW:4:20 MINLEN:25 ILLUMINACLIP:NexteraPE-PE.fa:2:40:15 >logs/trimmomatic.SRR2584863.out 2>logs/trimmomatic.SRR2584863.err
>
> [Fri Oct  8 16:59:39 2021]
> rule indexgenome:
>     input: reference/GCA_000017985.1_ASM1798v1_genomic.fna.gz
>     output: reference/GCA_000017985.1_ASM1798v1_genomic.fna.gz.bwt
>     log: logs/bwa.index.out, logs/bwa.index.err
>     jobid: 5
>     resources: tmpdir=/tmp
>
> bwa index reference/GCA_000017985.1_ASM1798v1_genomic.fna.gz >logs/bwa.index.out 2>logs/bwa.index.err
>
> [Fri Oct  8 16:59:39 2021]
> rule align:
>     input: trim/SRR2584863_1.trim.fq, trim/SRR2584863_1.trim.fq, reference/GCA_000017985.1_ASM1798v1_genomic.fna.gz.bwt
>     output: alignments/GCA_000017985.1_ASM1798v1_genomic/SRR2584863.bam
>     log: logs/bwa.GCA_000017985.1_ASM1798v1_genomic.SRR2584863.err, logs/samtools.GCA_000017985.1_ASM1798v1_genomic.SRR2584863.out, logs/samtools.GCA_000017985.1_ASM1798v1_genomic.SRR2584863.err
>     jobid: 1
>     wildcards: genome=GCA_000017985.1_ASM1798v1_genomic, sample=SRR2584863
>     resources: tmpdir=/tmp
>
> bwa mem reference/GCA_000017985.1_ASM1798v1_genomic.fna.gz trim/SRR2584863_1.trim.fq trim/SRR2584863_1.trim.fq 2>logs/bwa.GCA_000017985.1_ASM1798v1_genomic.SRR2584863.err | samtools sort -O bam -o alignments/GCA_000017985.1_ASM1798v1_genomic/SRR2584863.bam >logs/samtools.GCA_000017985.1_ASM1798v1_genomic.SRR2584863.out 2>logs/samtools.GCA_000017985.1_ASM1798v1_genomic.SRR2584863.err
>
> [Fri Oct  8 16:59:39 2021]
> localrule all:
>     input: alignments/GCA_000017985.1_ASM1798v1_genomic/SRR2584863.bam
>     jobid: 0
>     resources: tmpdir=/tmp
>
> Job stats:
> job                count    min threads    max threads
> ---------------  -------  -------------  -------------
> align                  1              1              1
> all                    1              1              1
> download               2              1              1
> download_genome        1              1              1
> indexgenome            1              1              1
> trimmomatic            1              1              1
> total                  7              1              1
>
> This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.
> ```
{: .code-out.code-max-300}

Gosh that's a lot of output! Let's build the {DAG} to see a more concise representation of what is going to happen:

> <code-in-title></code-in-title>
> ```bash
> snakemake --dag | dot -Tsvg > out.svg
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ![Image of the snakemake dag. Conspicuously missing is FastQC jobs!](../../images/


topic: data-science
tutorial: cli-advanced
title: Advanced CLI in Galaxy
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["bash"]
content_hash: 277e98728dcea8b445b26ba455dfe126
last_modified: 2026-05-15T22:28:10.896792
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/cli-advanced/tutorial.html

content:
This tutorial will walk you through the basics of how to use the Unix command line.

> <comment-title></comment-title>
>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) ["The Unix Shell"](https://swcarpentry.github.io/shell-novice/) lesson, which is licensed CC-BY 4.0. Adaptations have been made to make this work better in a GTN/Galaxy environment.
>
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Pipes and Filtering

Now that we know a few basic commands,
we can finally look at the shell's most powerful feature:
the ease with which it lets us combine existing programs in new ways.
We'll start with the directory called `shell-lesson-data/molecules`
that contains six files describing some simple organic molecules.
The `.pdb` extension indicates that these files are in Protein Data Bank format,
a simple text format that specifies the type and position of each atom in the molecule.

```bash
cd ~/Desktop/shell-lesson-data/
ls molecules
```


Let's go into that directory with `cd` and run an example command `wc cubane.pdb`:

```bash
cd molecules
wc cubane
```

`wc` is the 'word count' command:
it counts the number of lines, words, and characters in files (from left to right, in that order).

If we run the command `wc *.pdb`, the `*` in `*.pdb` matches zero or more characters,
so the shell turns `*.pdb` into a list of all `.pdb` files in the current directory:

```bash
wc *.pdb
```

Note that `wc *.pdb` also shows the total number of all lines in the last line of the output.

If we run `wc -l` instead of just `wc`,
the output shows only the number of lines per file:

```bash
wc -l *.pdb
```

The `-m` and `-w` options can also be used with the `wc` command, to show
only the number of characters or the number of words in the files.

> <tip-title>Why Isn't It Doing Anything?</tip-title>
>
> What happens if a command is supposed to process a file, but we
> don't give it a filename? For example, what if we type:
>
> ```bash
> $ wc -l
> ```
>
> but don't type `*.pdb` (or anything else) after the command?
> Since it doesn't have any filenames, `wc` assumes it is supposed to
> process input given at the command prompt, so it just sits there and waits for us to give
> it some data interactively. From the outside, though, all we see is it
> sitting there: the command doesn't appear to do anything.
>
> If you make this kind of mistake, you can escape out of this state by holding down
> the control key (<kbd>Ctrl</kbd>) and typing the letter <kbd>C</kbd> once and
> letting go of the <kbd>Ctrl</kbd> key.
> <kbd>Ctrl</kbd>+<kbd>C</kbd>
{: .tip}


## Capturing output from commands

Which of these files contains the fewest lines?
It's an easy question to answer when there are only six files,
but what if there were 6000?
Our first step toward a solution is to run the command:

```bash
wc -l *.pdb > lengths.txt
```

The greater than symbol, `>`, tells the shell to **redirect** the command's output
to a file instead of printing it to the screen. (This is why there is no screen output:
everything that `wc` would have printed has gone into the
file `lengths.txt` instead.)  The shell will create
the file if it doesn't exist. If the file exists, it will be
silently overwritten, which may lead to data loss and thus requires
some caution.

> <tip-title>No <code>&gt;</code> on an AZERTY keyboard?</tip-title>
> You can rewrite this using the tee command which writes out a file, while also showing the output to `stdout`.
> ```bash
> wc -l *.pdb | tee lengths.txt
> ```
> Or you can use copy and paste to copy the `>` character from the materials.
{: .tip}

`ls lengths.txt` confirms that the file exists:

```bash
ls lengths.txt
```

We can now send the content of `lengths.txt` to the screen using `cat lengths.txt`.
The `cat` command gets its name from 'concatenate' i.e. join together,
and it prints the contents of files one after another.
There's only one file in this case,
so `cat` just shows us what it contains:

```bash
cat lengths.txt
```

> <tip-title>Output Page by Page</tip-title>
>
> We'll continue to use `cat` in this lesson, for convenience and consistency,
> but it has the disadvantage that it always dumps the whole file onto your screen.
> More useful in practice is the command `less`,
> which you use with `less lengths.txt`.
> This displays a screenful of the file, and then stops.
> You can go forward one screenful by pressing the spacebar,
> or back one by pressing `b`.  Press `q` to quit.
{: .tip}


## Filtering output

Next we'll use the `sort` command to sort the contents of the `lengths.txt` file.
But first we'll use an exercise to learn a little about the sort command:

> <question-title>What Does `sort -n` Do?</question-title>
>
> The file `shell-lesson-data/numbers.txt`
> contains the following lines:
>
> ```
> 10
> 2
> 19
> 22
> 6
> ```
>
> If we run `sort` on this file, the output is:
>
> ```
> 10
> 19
> 2
> 22
> 6
> ```
>
> If we run `sort -n` on the same file, we get this instead:
>
> ```
> 2
> 6
> 10
> 19
> 22
> ```
>
> Explain why `-n` has this effect.
>
> > <solution-title></solution-title>
> > The `-n` option specifies a numerical rather than an alphanumerical sort.
> {: .solution}
{: .question}

We will also use the `-n` option to specify that the sort is
numerical instead of alphanumerical.
This does *not* change the file;
instead, it sends the sorted result to the screen:

```bash
sort -n lengths.txt
```


We can put the sorted list of lines in another temporary file called `sorted-lengths.txt`
by putting `> sorted-lengths.txt` after the command,
just as we used `> lengths.txt` to put the output of `wc` into `lengths.txt`.
Once we've done that,
we can run another command called `head` to get the first few lines in `sorted-lengths.txt`:

```bash
sort -n lengths.txt > sorted-lengths.txt
```

Using `-n 1` with `head` tells it that
we only want the first line of the file;
`-n 20` would get the first 20,
and so on.
Since `sorted-lengths.txt` contains the lengths of our files ordered from least to greatest,
the output of `head` must be the file with the fewest lines.

> <tip-title>Redirecting to the same file</tip-title>
>
> It's a very bad idea to try redirecting
> the output of a command that operates on a file
> to the same file. For example:
>
> ```
> $ sort -n lengths.txt > lengths.txt
> ```
>
> Doing something like this may give you
> incorrect results and/or delete
> the contents of `lengths.txt`.
{: .tip}

> <question-title>What Does `>>` Mean?</question-title>
>
> We have seen the use of `>`, but there is a similar operator `>>`
> which works slightly differently.
> We'll learn about the differences between these two operators by printing some strings.
> We can use the `echo` command to print strings e.g.
>
> > <code-in-title>Bash</code-in-title>
> > ```
> > $ echo The echo command prints text
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > The echo command prints text
> > ```
> {: .code-out}
>
> Now test the commands below to reveal the difference between the two operators:
>
> > <code-in-title>Bash</code-in-title>
> > ```
> > $ echo hello > testfile01.txt
> > ```
> {: .code-in}
>
> and:
>
> > <code-in-title>Bash</code-in-title>
> > ```
> > $ echo hello >> testfile02.txt
> > ```
> {: .code-in}
>
> **Hint**: Try executing each command twice in a row and then examining the output files.
>
> > ## Solution
> > In the first example with `>`, the string 'hello' is written to `testfile01.txt`,
> > but the file gets overwritten each time we run the command.
> >
> > We see from the second example that the `>>` operator also writes 'hello' to a file
> > (in this case`testfile02.txt`),
> > but appends the string to the file if it already exists
> > (i.e. when we run it for the second time).
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <question-title>Appending Data</question-title>
>
> We have already met the `head` command, which prints lines from the start of a file.
> `tail` is similar, but prints lines from the end of a file instead.
>
> Consider the file `shell-lesson-data/data/animals.txt`.
> After these commands, select the answer that
> corresponds to the file `animals-subset.txt`:
>
> ```
> $ head -n 3 animals.txt > animals-subset.txt
> $ tail -n 2 animals.txt >> animals-subset.txt
> ```
>
> 1. The first three lines of `animals.txt`
> 2. The last two lines of `animals.txt`
> 3. The first three lines and the last two lines of `animals.txt`
> 4. The second and third lines of `animals.txt`
>
> > <solution-title></solution-title>
> > Option 3 is correct.
> > For option 1 to be correct we would only run the `head` command.
> > For option 2 to be correct we would only run the `tail` command.
> > For option 4 to be correct we would have to pipe the output of `head` into `tail -n 2`
> > by doing `head -n 3 animals.txt | tail -n 2 > animals-subset.txt`
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```


## Passing output to another command
In our example of finding the file with the fewest lines,
we are using two intermediate files `lengths.txt` and `sorted-lengths.txt` to store output.
This is a confusing way to work because
even once you understand what `wc`, `sort`, and `head` do,
those intermediate files make it hard to follow what's going on.
We can make it easier to understand by running `sort` and `head` together:

```bash
sort -n lengths.txt | head -n 1
```

The vertical bar, `|`, between the two commands is called a **pipe**.
It tells the shell that we want to use
the output of the command on the left
as the input to the command on the right.

This has removed the need for the `sorted-lengths.txt` file.

## Combining multiple commands
Nothing prevents us from chaining pipes consecutively.
We can for example send the output of `wc` directly to `sort`,
and then the resulting output to `head`.
This removes the need for any intermediate files.

We'll start by using a pipe to send the output of `wc` to `sort`:

```bash
wc -l *.pdb | sort -n
```

We can then send that output through another pipe, to `head`, so that the full pipeline becomes:

```bash
wc -l *.pdb | sort -n | head -n 1
```

This is exactly like a mathematician nesting functions like *log(3x)*
and saying 'the log of three times *x*'.
In our case,
the calculation is 'head of sort of line count of `*.pdb`'.


The redirection and pipes used in the last few commands are illustrated below:

![Redirects and Pipes of different commands](../../images/carpentries-cli/redirects-and-pipes.svg)

`wc -l *.pdb` will direct the output to the shell. `wc -l *.pdb > lengths` will
direct output to the file lengths. `wc -l *.pdb | sort -n | head -n 1` will
build a pipeline where the output of the wc command is the input to the sort
command, the output of the sort command is the input to the head command and
the output of the head command is directed to the shell

> <question-title>Piping Commands Together</question-title>
>
> In our current directory, we want to find the 3 files which have the least number of
> lines. Which command listed below would work?
>
> 1. `wc -l * > sort -n > head -n 3`
> 2. `wc -l * | sort -n | head -n 1-3`
> 3. `wc -l * | head -n 3 | sort -n`
> 4. `wc -l * | sort -n | head -n 3`
>
> > <solution-title></solution-title>
> > Option 4 is the solution.
> > The pipe character `|` is used to connect the output from one command to
> > the input of another.
> > `>` is used to redirect standard output to a file.
> > Try it in the `shell-lesson-data/molecules` directory!
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```


## Tools designed to work together
This idea of linking programs together is why Unix has been so successful.
Instead of creating enormous programs that try to do many different things,
Unix programmers focus on creating lots of simple tools that each do one job well,
and that work well with each other.
This programming model is called 'pipes and filters'.
We've already seen pipes;
a **filter** is a program like `wc` or `sort`
that transforms a stream of input into a stream of output.
Almost all of the standard Unix tools can work this way:
unless told to do otherwise,
they read from standard input,
do something with what they've read,
and write to standard output.

The key is that any program that reads lines of text from standard input
and writes lines of text to standard output
can be combined with every other program that behaves this way as well.
You can *and should* write your programs this way
so that you and other people can put those programs into pipes to multiply their power.


> <question-title>Pipe Reading Comprehension</question-title>
>
> A file called `animals.txt` (in the `shell-lesson-data/data` folder) contains the following data:
>
> ```
> 2012-11-05,deer
> 2012-11-05,rabbit
> 2012-11-05,raccoon
> 2012-11-06,rabbit
> 2012-11-06,deer
> 2012-11-06,fox
> 2012-11-07,rabbit
> 2012-11-07,bear
> ```
>
> What text passes through each of the pipes and the final redirect in the pipeline below?
>
> ```
> $ cat animals.txt | head -n 5 | tail -n 3 | sort -r > final.txt
> ```
>
> Hint: build the pipeline up one command at a time to test your understanding
>
> > <solution-title></solution-title>
> > The `head` command extracts the first 5 lines from `animals.txt`.
> > Then, the last 3 lines are extracted from the previous 5 by using the `tail` command.
> > With the `sort -r` command those 3 lines are sorted in reverse order and finally,
> > the output is redirected to a file `final.txt`.
> > The content of this file can be checked by executing `cat final.txt`.
> > The file should contain the following lines:
> > ```
> > 2012-11-06,rabbit
> > 2012-11-06,deer
> > 2012-11-05,raccoon
> > ```
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <question-title>Pipe Construction</question-title>
>
> For the file `animals.txt` from the previous exercise, consider the following command:
>
> ```
> $ cut -d , -f 2 animals.txt
> ```
>
> The `cut` command is used to remove or 'cut out' certain sections of each line in the file,
> and `cut` expects the lines to be separated into columns by a <kbd>Tab</kbd> character.
> A character used in this way is a called a **delimiter**.
> In the example above we use the `-d` option to specify the comma as our delimiter character.
> We have also used the `-f` option to specify that we want to extract the second field (column).
> This gives the following output:
>
> ```
> deer
> rabbit
> raccoon
> rabbit
> deer
> fox
> rabbit
> bear
> ```
>
> The `uniq` command filters out adjacent matching lines in a file.
> How could you extend this pipeline (using `uniq` and another command) to find
> out what animals the file contains (without any duplicates in their
> names)?
>
> > <solution-title></solution-title>
> > ```
> > $ cut -d , -f 2 animals.txt | sort | uniq
> > ```
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <question-title>Which Pipe?</question-title>
>
> The file `animals.txt` contains 8 lines of data formatted as follows:
>
> ```
> 2012-11-05,deer
> 2012-11-05,rabbit
> 2012-11-05,raccoon
> 2012-11-06,rabbit
> ...
> ```
>
> The `uniq` command has a `-c` option which gives a count of the
> number of times a line occurs in its input.  Assuming your current
> directory is `shell-lesson-data/data/`, what command would you use to produce
> a table that shows the total count of each type of animal in the file?
>
> 1.  `sort animals.txt | uniq -c`
> 2.  `sort -t, -k2,2 animals.txt | uniq -c`
> 3.  `cut -d, -f 2 animals.txt | uniq -c`
> 4.  `cut -d, -f 2 animals.txt | sort | uniq -c`
> 5.  `cut -d, -f 2 animals.txt | sort | uniq -c | wc -l`
>
> > <solution-title></solution-title>
> > Option 4. is the correct answer.
> > If you have difficulty understanding why, try running the commands, or sub-sections of
> > the pipelines (make sure you are in the `shell-lesson-data/data` directory).
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

## Nelle's Pipeline: Checking Files

Nelle has run her samples through the assay machines
and created 17 files in the `north-pacific-gyre/2012-07-03` directory described earlier.
As a quick check, starting from her home directory, Nelle types:

```bash
cd ~/Desktop/shell-lesson-data/north-pacific-gyre/2012-07-03
wc -l *.txt
```

The output is 18 lines that look like this:

```
300 NENE01729A.txt
300 NENE01729B.txt
300 NENE01736A.txt
300 NENE01751A.txt
300 NENE01751B.txt
300 NENE01812A.txt
... ...
```

Now she types this:

```bash
wc -l *.txt | sort -n | head -n 5
```

Whoops: one of the files is 60 lines shorter than the others.
When she goes back and checks it,
she sees that she did that assay at 8:00 on a Monday morning --- someone
was probably in using the machine on the weekend,
and she forgot to reset it.
Before re-running that sample,
she checks to see if any files have too much data:

```bash
wc -l *.txt | sort -n | tail -n 5
```

Those numbers look good --- but what's that 'Z' doing there in the third-to-last line?
All of her samples should be marked 'A' or 'B';
by convention,
her lab uses 'Z' to indicate samples with missing information.
To find others like it, she does this:

```bash
ls *Z.txt
```

Sure enough,
when she checks the log on her laptop,
there's no depth recorded for either of those samples.
Since it's too late to get the information any other way,
she must exclude those two files from her analysis.
She could delete them using `rm`,
but there are actually some analyses she might do later where depth doesn't matter,
so instead, she'll have to be careful later on to select files using the wildcard expressions
`NENE*A.txt NENE*B.txt`.


> <question-title>Removing Unneeded Files</question-title>
>
> Suppose you want to delete your processed data files, and only keep
> your raw files and processing script to save storage.
> The raw files end in `.dat` and the processed files end in `.txt`.
> Which of the following would remove all the processed data files,
> and *only* the processed data files?
>
> 1. `rm ?.txt`
> 2. `rm *.txt`
> 3. `rm * .txt`
> 4. `rm *.*`
>
> > <solution-title></solution-title>
> > 1. This would remove `.txt` files with one-character names
> > 2. This is correct answer
> > 3. The shell would expand `*` to match everything in the current directory,
> > so the command would try to remove all matched files and an additional
> > file called `.txt`
> > 4. The shell would expand `*.*` to match all files with any extension,
> > so this command would delete all files
> {: .solution}
{: .question}


# Loops

**Loops** are a programming construct which allow us to repeat a command or set of commands
for each item in a list.
As such they are key to productivity improvements through automation.
Similar to wildcards and tab completion, using loops also reduces the
amount of typing required (and hence reduces the number of typing mistakes).

Suppose we have several hundred genome data files named `basilisk.dat`, `minotaur.dat`, and
`unicorn.dat`.
For this example, we'll use the `creatures` directory which only has three example files,
but the principles can be applied to many many more files at once. First, go
into the creatures directory.

```bash
# Change directories here!

```

The structure of these files is the same: the common name, classification, and updated date are
presented on the first three lines, with DNA sequences on the following lines.
Let's look at the files:

```bash
head -n 5 basilisk.dat minotaur.dat unicorn.dat
```

We would like to print out the classification for each species, which is given on the second
line of each file.
For each file, we would need to execute the command `head -n 2` and pipe this to `tail -n 1`.
We’ll use a loop to solve this problem, but first let’s look at the general form of a loop:

```
for thing in list_of_things
do
    operation_using $thing    # Indentation within the loop is not required, but aids legibility
done
```

and we can apply this to our example like this:

```bash
for filename in basilisk.dat minotaur.dat unicorn.dat
do
    head -n 2 $filename | tail -n 1
done
```


> <tip-title>Follow the Prompt</tip-title>
>
> The shell prompt changes from `$` to `>` and back again as we were
> typing in our loop. The second prompt, `>`, is different to remind
> us that we haven't finished typing a complete command yet. A semicolon, `;`,
> can be used to separate two commands written on a single line.
{: .tip}

When the shell sees the keyword `for`,
it knows to repeat a command (or group of commands) once for each item in a list.
Each time the loop runs (called an iteration), an item in the list is assigned in sequence to
the **variable**, and the commands inside the loop are executed, before moving on to
the next item in the list.
Inside the loop,
we call for the variable's value by putting `$` in front of it.
The `$` tells the shell interpreter to treat
the variable as a variable name and substitute its value in its place,
rather than treat it as text or an external command.

In this example, the list is three filenames: `basilisk.dat`, `minotaur.dat`, and `unicorn.dat`.
Each time the loop iterates, it will assign a file name to the variable `filename`
and run the `head` command.
The first time through the loop,
`$filename` is `basilisk.dat`.
The interpreter runs the command `head` on `basilisk.dat`
and pipes the first two lines to the `tail` command,
which then prints the second line of `basilisk.dat`.
For the second iteration, `$filename` becomes
`minotaur.dat`. This time, the shell runs `head` on `minotaur.dat`
and pipes the first two lines to the `tail` command,
which then prints the second line of `minotaur.dat`.
For the third iteration, `$filename` becomes
`unicorn.dat`, so the shell runs the `head` command on that file,
and `tail` on the output of that.
Since the list was only three items, the shell exits the `for` loop.

> <tip-title>Same Symbols, Different Meanings</tip-title>
>
> Here we see `>` being used as a shell prompt, whereas `>` is also
> used to redirect output.
> Similarly, `$` is used as a shell prompt, but, as we saw earlier,
> it is also used to ask the shell to get the value of a variable.
>
> If the *shell* prints `>` or `$` then it expects you to type something,
> and the symbol is a prompt.
>
> If *you* type `>` or `$` yourself, it is an instruction from you that
> the shell should redirect output or get the value of a variable.
{: .tip}

When using variables it is also
possible to put the names into curly braces to clearly delimit the variable
name: `$filename` is equivalent to `${filename}`, but is different from
`${file}name`. You may find this notation in other people's programs.

We have called the variable in this loop `filename`
in order to make its purpose clearer to human readers.
The shell itself doesn't care what the variable is called;
if we wrote this loop as:

```bash
for x in basilisk.dat minotaur.dat unicorn.dat
do
    head -n 2 $x | tail -n 1
done
```

or:

```bash
for temperature in basilisk.dat minotaur.dat unicorn.dat
do
    head -n 2 $temperature | tail -n 1
done
```

it would work exactly the same way.

# Don't do this.

Programs are only useful if people can understand them,
so meaningless names (like `x`) or misleading names (like `temperature`)
increase the odds that the program won't do what its readers think it does.

> <question-title>Variables in Loops</question-title>
>
> This exercise refers to the `shell-lesson-data/molecules` directory.
> `ls` gives the following output:
>
> ```
> cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> ```
>
> What is the output of the following code?
>
> ```
> for datafile in *.pdb
> do
>     ls *.pdb
> done
> ```
>
> Now, what is the output of the following code?
>
> ```
> for datafile in *.pdb
> do
>    ls $datafile
> done
> ```
>
> Why do these two loops give different outputs?
>
> > <solution-title></solution-title>
> > The first code block gives the same output on each iteration through
> > the loop.
> > Bash expands the wildcard `*.pdb` within the loop body (as well as
> > before the loop starts) to match all files ending in `.pdb`
> > and then lists them using `ls`.
> > The expanded loop would look like this:
> > ```
> > $ for datafile in cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> > > do
> > >     ls cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> > > done
> > ```
> >
> > ```
> > cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> > cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> > cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> > cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> > cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> > cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> > ```
> >
> > The second code block lists a different file on each loop iteration.
> > The value of the `datafile` variable is evaluated using `$datafile`,
> > and then listed using `ls`.
> >
> > ```
> > cubane.pdb
> > ethane.pdb
> > methane.pdb
> > octane.pdb
> > pentane.pdb
> > propane.pdb
> > ```
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <question-title>Limiting Sets of Files</question-title>
>
> What would be the output of running the following loop in thei
> `shell-lesson-data/molecules` directory?
>
> ```
> for filename in c*
> do
>     ls $filename
> done
> ```
>
> 1.  No files are listed.
> 2.  All files are listed.
> 3.  Only `cubane.pdb`, `octane.pdb` and `pentane.pdb` are listed.
> 4.  Only `cubane.pdb` is listed.
>
> > <solution-title></solution-title>
> > 4 is the correct answer. `*` matches zero or more characters, so any file name starting with
> > the letter c, followed by zero or more other characters will be matched.
> {: .solution}
>
> How would the output differ from using this command instead?
>
> ```
> for filename in *c*
> do
>     ls $filename
> done
> ```
>
> 1.  The same files would be listed.
> 2.  All the files are listed this time.
> 3.  No files are listed this time.
> 4.  The files `cubane.pdb` and `octane.pdb` will be listed.
> 5.  Only the file `octane.pdb` will be listed.
>
> > <solution-title></solution-title>
> > 4 is the correct answer. `*` matches zero or more characters, so a file name with zero or more
> > characters before a letter c and zero or more characters after the letter c will be matched.
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <question-title>Saving to a File in a Loop - Part One</question-title>
>
> In the `shell-lesson-data/molecules` directory, what is the effect of this loop?
>
> ```
> for alkanes in *.pdb
> do
>     echo $alkanes
>     cat $alkanes > alkanes.pdb
> done
> ```
>
> 1.  Prints `cubane.pdb`, `ethane.pdb`, `methane.pdb`, `octane.pdb`, `pentane.pdb` and
>    `propane.pdb`, and the text from `propane.pdb` will be saved to a file called `alkanes.pdb`.
> 2.  Prints `cubane.pdb`, `ethane.pdb`, and `methane.pdb`, and the text from all three files
>     would be concatenated and saved to a file called `alkanes.pdb`.
> 3.  Prints `cubane.pdb`, `ethane.pdb`, `methane.pdb`, `octane.pdb`, and `pentane.pdb`,
>     and the text from `propane.pdb` will be saved to a file called `alkanes.pdb`.
> 4.  None of the above.
>
> > <solution-title></solution-title>
> > 1 is correct. The text from each file in turn gets written to the `alkanes.pdb` file.
> > However, the file gets overwritten on each loop iteration, so the final content of `alkanes.pdb`
> > is the text from the `propane.pdb` file.
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <question-title>Saving to a File in a Loop - Part Two</question-title>
>
> Also in the `shell-lesson-data/molecules` directory,
> what would be the output of the following loop?
>
> ```
> for datafile in *.pdb
> do
>     cat $datafile >> all.pdb
> done
> ```
>
> 1.  All of the text from `cubane.pdb`, `ethane.pdb`, `methane.pdb`, `octane.pdb`, and
>     `pentane.pdb` would be concatenated and saved to a file called `all.pdb`.
> 2.  The text from `ethane.pdb` will be saved to a file called `all.pdb`.
> 3.  All of the text from `cubane.pdb`, `ethane.pdb`, `methane.pdb`, `octane.pdb`, `pentane.pdb`
>     and `propane.pdb` would be concatenated and saved to a file called `all.pdb`.
> 4.  All of the text from `cubane.pdb`, `ethane.pdb`, `methane.pdb`, `octane.pdb`, `pentane.pdb`
>     and `propane.pdb` would be printed to the screen and saved to a file called `all.pdb`.
>
> > <solution-title></solution-title>
> > 3 is the correct answer. `>>` appends to a file, rather than overwriting it with the redirected
> > output from a command.
> > Given the output from the `cat` command has been redirected, nothing is printed to the screen.
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

Let's continue with our example in the `shell-lesson-data/creatures` directory.
Here's a slightly more complicated loop:

```bash
for filename in *.dat
do
    echo $filename
    head -n 100 $filename | tail -n 20
done
```

The shell starts by expanding `*.dat` to create the list of files it will process.
The **loop body**
then executes two commands for each of those files.
The first command, `echo`, prints its command-line arguments to standard output.
For example:

```bash
echo hello there
```

prints:

```
hello there
```

In this case,
since the shell expands `$filename` to be the name of a file,
`echo $filename` prints the name of the file.
Note that we can't write this as:

```bash
for filename in *.dat
do
    $filename
    head -n 100 $filename | tail -n 20
done
```

because then the first time through the loop,
when `$filename` expanded to `basilisk.dat`, the shell would try to run `basilisk.dat` as a program.
Finally,
the `head` and `tail` combination selects lines 81-100
from whatever file is being processed
(assuming the file has at least 100 lines).

> <tip-title>Spaces in Names</tip-title>
>
> Spaces are used to separate the elements of the list
> that we are going to loop over. If one of those elements
> contains a space character, we need to surround it with
> quotes, and do the same thing to our loop variable.
> Suppose our data files are named:
>
> ```
> red dragon.dat
> purple unicorn.dat
> ```
>
> To loop over these files, we would need to add double quotes like so:
>
> ```
> $ for filename in "red dragon.dat" "purple unicorn.dat"
> > do
> >     head -n 100 "$filename" | tail -n 20
> > done
> ```
>
> It is simpler to avoid using spaces (or other special characters) in filenames.
>
> The files above don't exist, so if we run the above code, the `head` command will be unable
> to find them, however the error message returned will show the name of the files it is
> expecting:
>
> ```
> head: cannot open ‘red dragon.dat’ for reading: No such file or directory
> head: cannot open ‘purple unicorn.dat’ for reading: No such file or directory
> ```
>
> Try removing the quotes around `$filename` in the loop above to see the effect of the quote
> marks on spaces. Note that we get a result from the loop command for unicorn.dat
> when we run this code in the `creatures` directory:
>
> ```
> head: cannot open ‘red’ for reading: No such file or directory
> head: cannot open ‘dragon.dat’ for reading: No such file or directory
> head: cannot open ‘purple’ for reading: No such file or directory
> CGGTACCGAA
> AAGGGTCGCG
> CAAGTGTTCC
> ...
> ```
{: .tip}

We would like to modify each of the files in `shell-lesson-data/creatures`, but also save a version
of the original files, naming the copies `original-basilisk.dat` and `original-unicorn.dat`.
We can't use:

```
cp *.dat original-*.dat
```

because that would expand to:

```bash
cp basilisk.dat minotaur.dat unicorn.dat original-*.dat
```

This wouldn't back up our files, instead we get an error.

This problem arises when `cp` receives more than two inputs. When this happens, it
expects the last input to be a directory where it can copy all the files it was passed.
Since there is no directory named `original-*.dat` in the `creatures` directory we get an
error.

Instead, we can use a loop:
```bash
for filename in *.dat
do
    cp $filename original-$filename
done
```

This loop runs the `cp` command once for each filename.
The first time,
when `$filename` expands to `basilisk.dat`,
the shell executes:

```
cp basilisk.dat original-basilisk.dat
```

The second time, the command is:

```
cp minotaur.dat original-minotaur.dat
```

The third and last time, the command is:

```
cp unicorn.dat original-unicorn.dat
```

Since the `cp` command does not normally produce any output, it's hard to check
that the loop is doing the correct thing.
However, we learned earlier how to print strings using `echo`, and we can modify the loop
to use `echo` to print our commands without actually executing them.
As such we can check what commands *would be* run in the unmodified loop.

The following diagram
shows what happens when the modified loop is executed, and demonstrates how the
judicious use of `echo` is a good debugging technique.

![The for loop 'for filename in *.dat; do echo cp $filename original-$filename; done' will successively assign the names of all '*.dat' files in your current directory to the variable '$filename' and then execute the command. With the files 'basilisk.dat', 'minotaur.dat' and 'unicorn.dat' in the current directory the loop will successively call the echo command three times and print three lines: 'cp basislisk.dat original-basilisk.dat', then 'cp minotaur.dat original-minotaur.dat' and finally 'cp unicorn.dat original-unicorn.dat'](../../images/carpentries-cli/shell_script_for_loop_flow_chart.svg)

## Nelle's Pipeline: Processing Files

Nelle is now ready to process her data files using `goostats.sh` ---
a shell script written by her supervisor.
This calculates some statistics from a protein sample file, and takes two arguments:

1. an input file (containing the raw data)
2. an output file (to store the calculated statistics)

Since she's still learning how to use the shell,
she decides to build up the required commands in stages.
Her first step is to make sure that she can select the right input files --- remember,
these are ones whose names end in 'A' or 'B', rather than 'Z'.
Starting from her home directory, Nelle types:

```bash
cd ~/Desktop/shell-lesson-data/north-pacific-gyre/2012-07-03
for datafile in NENE*A.txt NENE*B.txt
do
    echo $datafile
done
```

Her next step is to decide
what to call the files that the `goostats.sh` analysis program will create.
Prefixing each input file's name with 'stats' seems simple,
so she modifies her loop to do that:

```bash
for datafile in NENE*A.txt NENE*B.txt
do
    echo $datafile stats-$datafile
done
```

She hasn't actually run `goostats.sh` yet,
but now she's sure she can select the right files and generate the right output filenames.

> <tip-title>Top Terminal Tip: Re-running previous commands</tip-title>
> Typing in commands over and over again is becoming tedious,
> though,
> and Nelle is worried about making mistakes,
> so instead of re-entering her loop,
> she presses <kbd>↑</kbd>.
> In response,
> the shell redisplays the whole loop on one line
> (using semi-colons to separate the pieces):
>
> ```
> for datafile in NENE*A.txt NENE*B.txt; do echo $datafile stats-$datafile; done
> ```
{: .tip}

Using the left arrow key,
Nelle backs up and changes the command `echo` to `bash goostats.sh`:

```
for datafile in NENE*A.txt NENE*B.txt; do bash goostats.sh $datafile stats-$datafile; done
```

When she presses <kbd>Enter</kbd>,
the shell runs the modified command.
However, nothing appears to happen --- there is no output.
After a moment, Nelle realizes that since her script doesn't print anything to the screen
any longer, she has no idea whether it is running, much less how quickly.
She kills the running command by typing <kbd>Ctrl</kbd>+<kbd>C</kbd>,
uses <kbd>↑</kbd> to repeat the command,
and edits it to read:

```
for datafile in NENE*A.txt NENE*B.txt; do echo $datafile; bash goostats.sh $datafile stats-$datafile; done
```

> <tip-title>Beginning and End</tip-title>
>
> We can move to the beginning of a line in the shell by typing <kbd>Ctrl</kbd>+<kbd>A</kbd>
> and to the end using <kbd>Ctrl</kbd>+<kbd>E</kbd>.
{: .tip}

When she runs her program now,
it produces one line of output every five seconds or so:

1518 times 5 seconds,
divided by 60,
tells her that her script will take about two hours to run.
As a final check,
she opens another terminal window,
goes into `north-pacific-gyre/2012-07-03`,
and uses `cat stats-NENE01729B.txt`
to examine one of the output files.
It looks good,
so she decides to get some coffee and catch up on her reading.

> <tip-title>Those Who Know History Can Choose to Repeat It</tip-title>
>
> Another way to repeat previous work is to use the `history` command to
> get a list of the last few hundred commands that have been executed, and
> then to use `!123` (where '123' is replaced by the command number) to
> repeat one of those commands. For example, if Nelle types this:
>
> ```
> $ history | tail -n 5
> ```
>
> ```
>   456  ls -l NENE0*.txt
>   457  rm stats-NENE01729B.txt.txt
>   458  bash goostats.sh NENE01729B.txt stats-NENE01729B.txt
>   459  ls -l NENE0*.txt
>   460  history
> ```
>
> then she can re-run `goostats.sh` on `NENE01729B.txt` simply by typing
> `!458`. This number will be different for you, you should check your history before running it!
{: .tip}

> <tip-title>Other History Commands</tip-title>
>
> There are a number of other shortcut commands for getting at the history.
>
> - <kbd>Ctrl</kbd>+<kbd>R</kbd> enters a history search mode 'reverse-i-search' and finds the
> most recent command in your history that matches the text you enter next.
> Press <kbd>Ctrl</kbd>+<kbd>R</kbd> one or more additional times to search for earlier matches.
> You can then use the left and right arrow keys to choose that line and edit
> it then hit <kbd>Return</kbd> to run the command.
> - `!!` retrieves the immediately preceding command
> (you may or may not find this more convenient than using <kbd>↑</kbd>)
> - `!$` retrieves the last word of the last command.
> That's useful more often than you might expect: after
> `bash goostats.sh NENE01729B.txt stats-NENE01729B.txt`, you can type
> `less !$` to look at the file `stats-NENE01729B.txt`, which is
> quicker than doing <kbd>↑</kbd> and editing the command-line.
{: .tip}

> <question-title>Doing a Dry Run</question-title>
>
> A loop is a way to do many things at once --- or to make many mistakes at
> once if it does the wrong thing. One way to check what a loop *would* do
> is to `echo` the commands it would run instead of actually running them.
>
> Suppose we want to preview the commands the following loop will execute
> without actually running those commands:
>
> ```
> cd ~/Desktop/shell-lesson-data/pdb/
> for datafile in *.pdb
> do
>     cat $datafile >> all.pdb
> done
> ```
>
> What is the difference between the two loops below, and which one would we
> want to run?
>
> > <code-in-title>Version 1</code-in-title>
> > ```
> > for datafile in *.pdb
> > do
> >     echo cat $datafile >> all.pdb
> > done
> > ```
> {: .code-in}
>
> > <code-in-title>Version 2</code-in-title>
> > ```
> > for datafile in *.pdb
> > do
> >     echo "cat $datafile >> all.pdb"
> > done
> > ```
> {: .code-in}
>
> > <solution-title></solution-title>
> > The second version is the one we want to run.
> > This prints to screen everything enclosed in the quote marks, expanding the
> > loop variable name because we have prefixed it with a dollar sign.
> >
> > The first version appends the output from the command `echo cat $datafile`
> > to the file, `all.pdb`. This file will just contain the list;
> > `cat cubane.pdb`, `cat ethane.pdb`, `cat methane.pdb` etc.
> >
> > Try both versions for yourself to see the output! Be sure to open the
> > `all.pdb` file to view its contents.
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <question-title>Nested Loops</question-title>
>
> Suppose we want to set up a directory structure to organize
> some experiments measuring reaction rate constants with different compounds
> *and* different temperatures.  What would be the
> result of the following code:
>
> ```
> for species in cubane ethane methane
> do
>     for temperature in 25 30 37 40
>     do
>         mkdir $species-$temperature
>     done
> done
> ```
>
> > <solution-title></solution-title>
> > We have a nested loop, i.e. contained within another loop, so for each species
> > in the outer loop, the inner loop (the nested loop) iterates over the list of
> > temperatures, and creates a new directory for each combination.
> >
> > Try running the code for yourself to see which directories are created!
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```


# Finding Things

In the same way that many of us now use 'Google' as a
verb meaning 'to find', Unix programmers often use the
word 'grep'.
'grep' is a contraction of 'global/regular expression/print',
a common sequence of operations in early Unix text editors.
It is also the name of a very useful command-line program.

`grep` finds and prints lines in files that match a pattern.
For our examples,
we will use a file that contains three haiku taken from a
1998 competition in *Salon* magazine. For this set of examples,
we're going to be working in the writing subdirectory:

```bash
cd
cd Desktop/shell-lesson-data/writing
cat haiku.txt
```


> <tip-title>Forever, or Five Years</tip-title>
>
> We haven't linked to the original haiku because
> they don't appear to be on *Salon*'s site any longer.
> As [Jeff Rothenberg said](https://www.clir.org/wp-content/uploads/sites/6/ensuring.pdf),
> 'Digital information lasts forever --- or five years, whichever comes first.'
> Luckily, popular content often [has backups](http://wiki.c2.com/?ComputerErrorHaiku).
{: .tip}

Let's find lines that contain the word 'not':

```bash
grep not haiku.txt
```

Here, `not` is the pattern we're searching for.
The grep command searches through the file, looking for matches to the pattern specified.
To use it type `grep`, then the pattern we're searching for and finally
the name of the file (or files) we're searching in.

The output is the three lines in the file that contain the letters 'not'.

By default, grep searches for a pattern in a case-sensitive way.
In addition, the search pattern we have selected does not have to form a complete word,
as we will see in the next example.

Let's search for the pattern: 'The'.

```bash
grep The haiku.txt
```

This time, two lines that include the letters 'The' are outputted,
one of which contained our search pattern within a larger word, 'Thesis'.

To restrict matches to lines containing the word 'The' on its own,
we can give `grep` with the `-w` option.
This will limit matches to word boundaries.

Later in this lesson, we will also see how we can change the search behavior of grep
with respect to its case sensitivity.

```bash
grep -w The haiku.txt
```

Note that a 'word boundary' includes the start and end of a line, so not
just letters surrounded by spaces.
Sometimes we don't
want to search for a single word, but a phrase. This is also easy to do with
`grep` by putting the phrase in quotes.

```bash
grep -w "is not" haiku.txt
```

We've now seen that you don't have to have quotes around single words,
but it is useful to use quotes when searching for multiple words.
It also helps to make it easier to distinguish between the search term or phrase
and the file being searched.
We will use quotes in the remaining examples.

Another useful option is `-n`, which numbers the lines that match:

```bash
grep -n "it" haiku.txt
```

Here, we can see that lines 5, 9, and 10 contain the letters 'it'.

We can combine options (i.e. flags) as we do with other Unix commands.
For example, let's find the lines that contain the word 'the'.
We can combine the option `-w` to find the lines that contain the word 'the'
and `-n` to number the lines that match:

```bash
grep -n -w "the" haiku.txt
```

Now we want to use the option `-i` to make our search case-insensitive:

```bash
grep -n -w -i "the" haiku.txt
```

Now, we want to use the option `-v` to invert our search, i.e., we want to output
the lines that do not contain the word 'the'.

```bash
grep -n -w -v "the" haiku.txt
```


If we use the `-r` (recursive) option,
`grep` can search for a pattern recursively through a set of files in subdirectories.

Let's search recursively for `Yesterday` in the `shell-lesson-data/writing` directory:

```bash
grep -r Yesterday .
```

`grep` has lots of other options. To find out what they are, we can type:

```bash
grep --help
```

> <question-title>Using `grep`</question-title>
>
> Which command would result in the following output:
>
> ```
> and the presence of absence:
> ```
>
> 1. `grep "of" haiku.txt`
> 2. `grep -E "of" haiku.txt`
> 3. `grep -w "of" haiku.txt`
> 4. `grep -i "of" haiku.txt`
>
> > <solution-title></solution-title>
> > The correct answer is 3, because the `-w` option looks only for whole-word matches.
> > The other options will also match 'of' when part of another word.
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <tip-title>Wildcards</tip-title>
>
> `grep`'s real power doesn't come from its options, though; it comes from
> the fact that patterns can include wildcards. (The technical name for
> these is **regular expressions**, which
> is what the 're' in 'grep' stands for.) Regular expressions are both complex
> and powerful; if you want to do complex searches, please look at the lesson
> on [our website](http://v4.software-carpentry.org/regexp/index.html). As a taster, we can
> find lines that have an 'o' in the second position like this:
>
> ```
> $ grep -E "^.o" haiku.txt
> ```
>
> ```
> You bring fresh toner.
> Today it is not working
> Software is like that.
> ```
>
> We use the `-E` option and put the pattern in quotes to prevent the shell
> from trying to interpret it. (If the pattern contained a `*`, for
> example, the shell would try to expand it before running `grep`.) The
> `^` in the pattern anchors the match to the start of the line. The `.`
> matches a single character (just like `?` in the shell), while the `o`
> matches an actual 'o'.
{: .tip}

```bash
# Explore the possible solutions here!
```

> <question-title>Tracking a Species</question-title>
>
> Leah has several hundred
> data files saved in one directory, each of which is formatted like this:
>
> ```
> 2013-11-05,deer,5
> 2013-11-05,rabbit,22
> 2013-11-05,raccoon,7
> 2013-11-06,rabbit,19
> 2013-11-06,deer,2
> ```
>
> She wants to write a shell script that takes a species as the first command-line argument
> and a directory as the second argument. The script should return one file called `species.txt`
> containing a list of dates and the number of that species seen on each date.
> For example using the data shown above, `rabbit.txt` would contain:
>
> ```
> 2013-11-05,22
> 2013-11-06,19
> ```
>
> Put these commands and pipes in the right order to achieve this:
>
> ```
> cut -d : -f 2
> >
> |
> grep -w $1 -r $2
> |
> $1.txt
> cut -d , -f 1,3
> ```
>
> Hint: use `man grep` to look for how to grep text recursively in a directory
> and `man cut` to select more than one field in a line.
>
> An example of such a file is provided in `shell-lesson-data/data/animal-counts/animals.txt`
>
> > <solution-title></solution-title>
> >
> > ```
> > grep -w $1 -r $2 | cut -d : -f 2 | cut -d , -f 1,3 > $1.txt
> > ```
> >
> > Actually, you can swap the order of the two cut commands and it still works. At the
> > command line, try changing the order of the cut commands, and have a look at the output
> > from each step to see why this is the case.
> >
> > You would call the script above like this:
> >
> > ```
> > $ bash count-species.sh bear .
> > ```
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <question-title>Little Women</question-title>
>
> You and your friend, having just finished reading *Little Women* by
> Louisa May Alcott, are in an argument.  Of the four sisters in the
> book, Jo, Meg, Beth, and Amy, your friend thinks that Jo was the
> most mentioned.  You, however, are certain it was Amy.  Luckily, you
> have a file `LittleWomen.txt` containing the full text of the novel
> (`shell-lesson-data/writing/data/LittleWomen.txt`).
> Using a `for` loop, how would you tabulate the number of times each
> of the four sisters is mentioned?
>
> Hint: one solution might employ
> the commands `grep` and `wc` and a `|`, while another might utilize
> `grep` options.
> There is often more than one way to solve a programming task, so a
> particular solution is usually chosen based on a combination of
> yielding the correct result, elegance, readability, and speed.
>
> > <solution-title></solution-title>
> > ```
> > for sis in Jo Meg Beth Amy
> > do
> > 	echo $sis:
> >	grep -ow $sis LittleWomen.txt | wc -l
> > done
> > ```
> >
> > Alternative, slightly inferior solution:
> > ```
> > for sis in Jo Meg Beth Amy
> > do
> > 	echo $sis:
> >	grep -ocw $sis LittleWomen.txt
> > done
> > ```
> >
> > This solution is inferior because `grep -c` only reports the number of lines matched.
> > The total number of matches reported by this method will be lower if there is more
> > than one match per line.
> >
> > Perceptive observers may have noticed that character names sometimes appear in all-uppercase
> > in c


topic: data-science
tutorial: python-math
title: Python - Math
difficulty: introductory
time_estimation: 30M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 7684f6d5d2b61a02bce04bddd0b252ea
last_modified: 2026-05-15T22:28:11.104360
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-math/tutorial.html

content:
Here we'll learn some of the fundamentals of python and how to do basic maths in Python.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Python Fundamentals

## Variables

Any Python interpreter can be used as a calculator. We can do simple sums

```python
3 + 5 - 4
```

Multiplication

```python
5 * 4
```

Division

```python
5 / 4
```

Just like you've probably learned in maths courses, we can assign values to variables

```python
x = 10
y = 2.5
z = 3
```

From now on, whenever we use `x` or `y` or `z`, Python will substitute the value we assigned to it.
You'll notice when we assign things to a variable, nothing is printed out. If we've assigned a value to a variable we can print it by doing:

```python
print(x)
print(y, z)
```

You can print out multiple variables if you separate them with a comma `,`.


> <tip-title>Variable Names</tip-title>
> In Python, variable names:
>
> - can include letters, digits, and underscores
> - cannot start with a digit
> - are case sensitive
>
> This means that, for example:
>
> - `x` is a valid name
> - `weight0` is a valid variable name, whereas `0weight` is not
> - `weight` and `Weight` are different variables
{: .tip}

```python
x * (y + z)
```

What do you think the following will result in? The best way to find out is trying things for yourself. If it's wrong, can you correct it?

```python
xy
```

Time to check what we've learned!

> <question-title>Exercise 0: Simple Equations</question-title>
> Given the equation:
>
> $$y = x * 921 + 534$$
>
> What is the value of `y` when `x = 452`
>
> You can test out solutions in the box below this question.
>
> > <solution-title></solution-title>
> > ```
> > x = 452
> > y = x * 921 + 534
> > # Remember to print it out!
> > print(y)
> > ```
> >
> > 416826
> {: .solution}
{: .question}

```python
# Test solutions here!
# By the way, lines starting with a # are comment lines!
# You can use that to take notes, and not affect how your code runes
# People use it for documentation: explaining what the function does,
# what a variable means, why they chose this or that algorithm.
```

## Libraries

A library is a collection of files (called modules) that contains functions for use by other programs. It may also contain data values (e.g., numerical constants) and other things. A library's contents are supposed to be related, but there's no way to enforce that. The Python standard library is an extensive suite of modules that comes with Python itself. Many additional libraries are available from [PyPI (the Python Package Index)](https://pypi.org/).

### Libraries and modules

A library is a collection of modules, but the terms are often used interchangeably, especially since many libraries only consist of a single module, so don't worry if you mix them.

### A program must import a library module before using it.

You can use `import` to load a library module into a program's memory, then refer to things from the `module as module_name.thing_name`. Python uses `.` to mean “part of”. For example, using `datetime`, one of the modules in the standard library:

```python
import datetime

# it is currently
datetime.datetime.now()
```


> <tip-title>More complicated importing</tip-title>
>
> First, you can use `help` to learn about the contents of a library module. it works just like help for a function.
>
> ```
> help(datetime)
> ```
>
> You can import specific items from a library module to shorten programs. You can use `from ... import ...` to load only specific items from a library module. Then refer to them directly without library name as prefix.
>
>
> ```
> from datetime import datetime
>
> datetime.now()
> ```
>
> You can create an alias for a library module when importing it to shorten programs. Use `import ... as ...` to give a library a short alias while importing it. Then refer to items in the library using that shortened name.
>
> ```
> from datetime import datetime as dt
>
> dt.now()
> ```
{: .tip}


## Math Module

Let's import the math module:

```python
import math
```

This imports the math module. If you ever don't know what a function does, you can use the `help()` command:

```python
help(math)
```

And you'll see a list of the functions and properties available. Let's try out one of those functions now:

```python
math.sqrt(9)
```

> <solution-title>Tip: Why `math.`?</solution-title>
> When we import a module like `import math`, we need to use that as a prefix. Imagine we had two different modules, `math` and `other_math`, and both have a `sqrt` function. How would Python know which `sqrt` function we wanted? So we use `math.sqrt` to be explicit about which function we need.
{: .solution}

You might also have done powers (e.g. 2 cubed, or $$2^3$$) in the past, too:

```python
math.pow(2, 3)
```

Above we computed 2 cubed, but how did we know what to write? We might have seen `math.pow` in the `help(math)` page above, but how did we know `2, 3`? The same way of course:

```python
help(math.pow)
```

That would tell us that if we want 2 cubed, we need to write `2` and `3`. So let's do some exercises now and practice some of our maths skills.


> <question-title>Exercise 1: Basics</question-title>
> Please convert this function from an equation, into python code:
>
> $$x = 2^8$$
>
> > <solution-title></solution-title>
> > ```
> > x = math.pow(2, 8)
> > print(x)
> > ```
> {: .solution}
{: .question}

```python
# Test solutions here!
x =
print(x)
```


> <question-title>Exercise 2: Averaging two values</question-title>
> See if you can find the average of these two values, using the math operations
>
> - 23484
> - 12345
>
> You can find the average by summing, and dividing by 2:
>
> > <solution-title></solution-title>
> > ```
> > (23484 + 12345) / 2
> > ```
> {: .solution}
{: .question}

```python
# Test solutions here!
```

> <question-title>Exercise 3: Round-trip</question-title>
> Please convert this function from an equation, into python code. Remember, you can assign values to variables, if you want to split this into multiple steps.
>
> $$x = \sqrt{9^2}$$
>
> > <solution-title></solution-title>
> > ```
> > y = math.pow(9, 2)
> > x = math.sqrt(y)
> > ```
> >
> > Or
> >
> > ```
> > x = math.sqrt(math.pow(9, 2))
> > ```
> {: .solution}
{: .question}

```python
# Test solutions here!
```

> <question-title>Exercise 4: Pythagorean Theorem</question-title>
> The formula for a 90° triangle can be expressed as:
>
> $$a^2 + b^2 = c^2$$
>
> Or, expressed in terms of c, $$c = \sqrt{a^2 + b^2}$$
>
> Please convert this to python and find `c` when
>
> ```
> a = 65
> b = 72
> ```
>
> > <solution-title></solution-title>
> > c = 97
> > ```
> > c = math.sqrt(math.pow(a, 2) + math.pow(b, 2))
> > ```
> {: .solution}
{: .question}

```python
# Test solutions here!
```

> <question-title>Exercise 5: Quadratic Roots</question-title>
> Way back in algebra class, you might have been given a quadratic equation, something like:
>
> $$y = 2*x^2 + x - 1$$ and were told to find the roots of this function, using a complicated equation. So challenge time: reproduce this equation in Python:
>
> Given the following variables:
>
> a = 2
> b = 1
> c = -1
>
> Convert the following formulas to Python, and find the answers
>
> $$\dfrac{-b + \sqrt{b^2 - 4ac}}{2a}$$ and $$\dfrac{-b - \sqrt{b^2 - 4ac}}{2a}$$
>
> Since we're checking your ability to write Python, not do math, it should give `-1` and `0.5` to let you check your work. If you got those values, you got it right!
>
> Make sure you save each root as it's own variable, and then print them out.
>
> > <solution-title></solution-title>
> > ```
> > root1 = (-b + math.sqrt(math.pow(b, 2) - 4 * a * c))/(2 * a)
> > root2 = (-b - math.sqrt(math.pow(b, 2) - 4 * a * c))/(2 * a)
> > print(root1)
> > print(root2)
> > ```
> {: .solution}
{: .question}

```python
# Test solutions here!
root1 =
root2 =
print(root1)
print(root2)
```


topic: data-science
tutorial: python-testing
title: Python - Testing
difficulty: intermediate
time_estimation: 45M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 74672dffb2b52381bf89058cd2994330
last_modified: 2026-05-15T22:28:11.108607
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-testing/tutorial.html

content:
Here we will cover the basics of testing, an important part of software development. Testing lets you know that your code is correct in many situations that matter to you.

> <comment-title></comment-title>
>
> This tutorial is significantly based on [the Carpentries](https://carpentries.org) lesson ["Intermediate Research Software Development"](https://carpentries-incubator.github.io/python-intermediate-development/).
>
{: .comment}

Being able to demonstrate that a process generates the right results is important in any field of research, whether it's software generating those results or not. So when writing software we need to ask ourselves some key questions:

- Does the code we develop work the way it should do?
- Can we (and others) verify these assertions for themselves?
- Perhaps most importantly, to what extent are we confident of the accuracy of results that appear in publications?

If we are unable to demonstrate that our software fulfills these criteria, why would anyone use it? Having well-defined tests for our software are useful for this, but manually testing software can prove an expensive process.

Automation can help, and automation where possible is a good thing - it enables us to define a potentially complex process in a repeatable way that is far less prone to error than manual approaches. Once defined, automation can also save us a lot of effort, particularly in the long run. In this episode we'll look into techniques of automated testing to improve the predictability of a software change, make development more productive, and help us produce code that works as expected and produces desired results.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## What Is Software Testing?

For the sake of argument, if each line we write has a 99% chance of being right, then a 70-line program will be wrong more than half the time. We need to do better than that, which means we need to test our software to catch these mistakes.

We can and should extensively test our software manually, and manual testing is well-suited to testing aspects such as graphical user interfaces and reconciling visual outputs against inputs. However, even with a good test plan, manual testing is very time consuming and prone to error. Another style of testing is automated testing, where we write code that tests the functions of our software. Since computers are very good and efficient at automating repetitive tasks, we should take advantage of this wherever possible.

There are three main types of automated tests:

- **Unit tests** are tests for fairly small and specific units of functionality, e.g. determining that a particular function returns output as expected given specific inputs.
- **Functional or integration tests** work at a higher level, and test functional paths through your code, e.g. given some specific inputs, a set of interconnected functions across a number of modules (or the entire code) produce the expected result. These are particularly useful for exposing faults in how functional units interact.
- **Regression tests** make sure that your program's output hasn't changed, for example after making changes your code to add new functionality or fix a bug.
- **Property tests** are an advanced testing strategy to find corner cases.

For the purposes of this course, we'll focus on unit tests. But the principles and practices we'll talk about can be built on and applied to the other types of tests too.

## Example Codebase

```python
def rle_encode(input_string):
    count = 1
    prev = ""
    lst = []
    for character in input_string:
        if character != prev:
            if prev:
                entry = (prev, count)
                lst.append(entry)
            count = 1
            prev = character
        else:
            count += 1
    entry = (character, count)
    lst.append(entry)
    return "".join([f"{k}{v}" for k, v in lst])


def rle_decode(lst):
    q = ""
    for character, count in zip(lst[::2], lst[1::2]):
        q += character * int(count)
    return q
```

This is a simple run length encoding and decoding function. It does not handle a lot of corner cases or odd input, but it may do what we need: compressing long strings of repetitive texts. It could potentially provide good genomic data compression. (Spoilers, it generally doesn't. There are better compression algorithms.)

```python
print(rle_encode("aaabbc"))
```

Let's look at how we can test this cod.


## Writing Tests to Verify Correct Behaviour

### One Way to Do It?

One way to test our functions would be to write a series of checks or tests, each executing a function we want to test with known inputs against known valid results, and throw an error if we encounter a result that is incorrect

```python
test_input = "abba"
test_output = "a1b2a1"

assert rle_encode(test_input) == test_output
```

So we use the assert keyword - part of Python - to test that our calculated result is the same as our expected result. This function explicitly checks that the two values are the same, and throws an AssertionError if they are not.

Let's write some more test cases:

```python
test_input = "wwwwssssb"
test_output = "w5s4b1"
assert rle_encode(test_input) == test_output

test_round_trip = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
assert test_round_trip == rle_decode(rle_encode(test_round_trip))

test_input = "baabaa"
test_output = "b1a2b1a2"
assert rle_encode(test_input) == test_output
```

However, if we were to enter these in this order, we’ll find we get the following after the first test:

```
AssertionError                            Traceback (most recent call last)
<ipython-input-30-05489b2ef047> in <module>
      1 test_input = "wwwwssssb"
      2 test_output = "w5s4b1"
----> 3 assert rle_encode(test_input) == test_output

AssertionError: 
```

This tells us our function couldn't handle these inputs.

We could put these tests in a separate script to automate the running of these tests. But a Python script halts at the first failed assertion, so the second and third tests aren't run at all. It would be more helpful if we could get data from all of our tests every time they're run, since the more information we have, the faster we're likely to be able to track down bugs. It would also be helpful to have some kind of summary report: if our set of tests - known as a **test suite** - includes thirty or forty tests (as it well might for a complex function or library that's widely used), we'd like to know how many passed or failed.


Going back to our failed first test, what was the issue? As it turns out, the test itself was incorrect, and should have read:

```python
test_input = "wwwwssssb"
test_output = "w4s4b1"
assert rle_encode(test_input) == test_output
```

Which highlights an important point: as well as making sure our code is returning correct answers, we also need to ensure the tests themselves are also correct. Otherwise, we may go on to fix our code only to return an incorrect result that *appears* to be correct. So a good rule is to make tests simple enough to understand so we can reason about both the correctness of our tests as well as our code. Otherwise, our tests hold little value.

### Using a Testing Framework

Keeping these things in mind, here's a different approach that builds on the ideas we've seen so far but uses a **unit testing framework**. In such a framework we define our tests we want to run as functions, and the framework automatically runs each of these functions in turn, summarising the outputs. And unlike our previous approach, it will run every test regardless of any encountered test failures.

Most people don't enjoy writing tests, so if we want them to actually do it, it must be easy to:

- Add or change tests,
- Understand the tests that have already been written,
- Run those tests, and
- Understand those tests' results

Test results must also be reliable. If a testing tool says that code is working when it's not, or reports problems when there actually aren't any, people will lose faith in it and stop using it.

```python
from main import rle_encode, rle_decode
import unittest

class TestRunLengthEncoding(unittest.TestCase):
    def test_encode(self):
      i = "aaabbcccd"
      expected = "a3b2c3d1"
      actual = rle_encode(i)
      self.assertEqual(expected, actual)

    def test_decode(self):
        i = "a3b2c3d1"
        expected = "aaabbcccd"
        actual = rle_decode(i)
        self.assertEqual(expected, actual)

    def test_empty(self):
        test_input = "wwwwssssb"
        test_output = "w4s4b1"
        self.assertEqual(test_input, test_output)

    def test_roundtrip_short(self):
        test_input = "bbb"
        test_output = rle_decode(rle_encode(test_input))
        self.assertEqual(test_input, test_output)

    def test_roundtrip_long(self):
        test_input = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
        test_output = rle_decode(rle_encode(test_input))
        self.assertEqual(test_input, test_output)

if __name__ == '__main__':
    unittest.main()
```

So here, although we have specified our tests as separate functions in a class, they run the same assertions. Each of these test functions, in a general sense, are called **test cases** - these are a specification of:

- Inputs 
- Execution conditions - any imports we might need to do
- Testing procedure
- Expected outputs

And here, we're defining each of these things for a test case we can run independently that requires no manual intervention.

> <hands-on-title>Run this locally</hands-on-title>
> 1. Create a new folder somewhere
> 2. Save the RLE encode/decode functions as `main.py` in that folder
> 3. Save the unittests above as `test.py` in that folder
> 4. Run `python test.py`
{: .hands_on}

Going back to our list of requirements, how easy is it to run these tests? We can do this using a Python package called `unittest`. This is a built-in testing framework that allows you to write test cases using Python. You can use it to test things like Python functions, database operations, or even things like service APIs - essentially anything that has inputs and expected outputs. 

> <tip-title>What About Unit Testing in Other Languages?</tip-title>
>
> Other unit testing frameworks exist for Python, including Nose and pyunit, and the approach to unit testing can be translated to other languages as well, e.g. FRUIT for Fortran, JUnit for Java (the original unit testing framework), Catch for C++, etc.
{: .tip}


### What About Testing for Errors?

There are some cases where seeing an error is actually the correct behaviour, and Python allows us to test for exceptions. Add this test in `tests/test_models.py`:

```python
import unittest

class ExpectedFailureCase(unittest.TestCase):

    @unittest.expectedFailure
    def test_bad_input():
        from main import rle_decode

        # invalid input, we expect pairs of symbols and numbers.
        rle_decode("a1b")
```

Run all your tests as before.

> <tip-title>Why Should We Test Invalid Input Data?</tip-title>
>
> Testing the behaviour of inputs, both valid and invalid, is a really good idea and is known as *data validation*. Even if you are developing command line software that cannot be exploited by malicious data entry, testing behaviour against invalid inputs prevents generation of erroneous results that could lead to serious misinterpretation (as well as saving time and compute cycles which may be expensive for longer-running applications). It is generally best not to assume your user's inputs will always be rational.
>
{: .tip}

### Testing Frameworks

There are many, many testing frameworks for Python that solve similar problems. Look around and see what fits well with your project!

- [unittest](https://docs.python.org/3/library/unittest.html) is built into Python's stdlib.
- [pytest](https://docs.pytest.org/en/7.1.x/) is a heavily recommended framework.
- [nose](https://nose.readthedocs.io/en/latest/) is an alternative to pytest that does many of the same things.
- [hypothesis](https://hypothesis.readthedocs.io/en/latest/index.html) does property testing, the author's favourite.
- [doctest](https://docs.python.org/3/library/doctest.html) tests examples written directly in the documentation, which keeps test cases close to functions.


topic: data-science
tutorial: data-manipulation-olympics-viz-r
title: Data visualisation Olympics - Visualization in R
difficulty: introductory
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: ["cyoa", "R"]
content_hash: e1908e840a0b1d366249ee9c9775b97f
last_modified: 2026-05-15T22:28:10.992985
zenodo_link: https://zenodo.org/record/6803028
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/data-manipulation-olympics-viz-r/tutorial.html

content:
In this tutorial, you will learn how to produce scatter plots, boxplots, and time series plots using ggplot. You will also learn how to set universal plot settings, modify the aesthetics of an existing ggplot plots (including axis labels and color), and learn how to facet in ggplot.

> <comment-title></comment-title>
>
> This tutorial is significantly based on [Data Carpentry](https://datacarpentry.org) lesson ["Data visualization with ggplot2"](https://datacarpentry.org/R-ecology-lesson/04-visualization-ggplot2.html).
>
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Background

In this tutorial, we will use as our dataset a table with results from the Olympics, from the games in Athens in 1896 until Tokyo in 2020. The objective is to familiarize you with a large number of the most important data visualisation tools in Galaxy. Much like the Olympics, there are many different disciplines (types of operations), and for each operation there are often multiple techniques (tools) available to athletes (data analysts, you) that are great for achieving the goal.

![image of olympic rings, logo and two athletes around the words "Data Analysis Olympics"]({% link topics/introduction/tutorials/data-manipulation-olympics/images/cover.jpg %})

We will show you many of these commonly needed visualisation operations, and some examples of how to perform them in R. We also provide many exercises so that you can train your skills and become a data visualisation Olympian!

## Data Visualization with ggplot2

We start by loading the required packages. `ggplot2` is included in the `tidyverse` package.

```r
library(tidyverse)
```

# Download Data

Before we can do any visualisation, we will need some data. Let's download our table with Olympics results now.

```r
olympics <- read_tsv("{{page.zenodo_link}}/files/olympics.tsv")
```

```r
View(olympics)
```

> <question-title></question-title>
>
> 1. What is the format of the file?
> 2. How is it structured?
> 3. How many lines are in the file?
> 4. How many columns?
>
> > <solution-title></solution-title>
> > 1. When you expand the `olympics.tsv` dataset in your history (see also screenshot below), you will see `format: tabular`, this is another term for a tab-separated (`tsv`) file.
> > 2. Each row represents an athlete's participation in an event. If an athlete competes in multiple events, there is a line for each event.
> > 3. 234,522. Look at the bottom of the View or Environment panels to see this number.
> > 4. There are 17 columns in this file. See View or Environment panels.
> {: .solution}
{: .question}


## About this dataset

The data was [obtained](https://github.com/UOSCS/Olympic_Athletes) from [Olympedia](https://www.olympedia.org/). The file `olympics.tsv` contains
234,522 rows and 17 columns. Each row corresponds to an individual athlete competing in an individual Olympic event. The columns are:

- **athlete_id** - Unique number for each athlete
- **name** - Athlete's name
- **sex** - M or F
- **birth_year** - 4-digit number
- **birth_day** - e.g. 24 July
- **birth_place** - town and/or country
- **height** - In centimeters (or `NA` if data not known)
- **weight** - In kilograms (or `NA` if data not known)
- **team** - Team name
- **noc** - National Olympic Committee 3-letter code
- **games** - Year and season
- **year** - Integer
- **season** - Summer or Winter
- **city** - Host city
- **sport** - Sport
- **event** - Event
- **medal** - Gold, Silver, Bronze (or `NA` if no medal was won)

We will use this dataset to practice our data visualisation skills in Galaxy.

## Plotting with `ggplot2`

`ggplot2` is a plotting package that provides helpful commands to create complex plots
from data in a data frame. It provides a more programmatic interface for
specifying what variables to plot, how they are displayed, and general visual
properties. Therefore, we only need minimal changes if the underlying data
change or if we decide to change from a bar plot to a scatterplot. This helps in
creating publication quality plots with minimal amounts of adjustments and
tweaking.

`ggplot2` refers to the name of the package itself. When using the package we use the
function `ggplot()` to generate the plots, and so references to using the function will
be referred to as `ggplot()` and the package as a whole as `ggplot2`

`ggplot2` plots work best with data in the 'long' format, i.e., a column for every variable,
and a row for every observation. Well-structured data will save you lots of time
when making figures with `ggplot2`

ggplot graphics are built layer by layer by adding new elements. Adding layers in
this fashion allows for extensive flexibility and customization of plots.

To build a ggplot, we will use the following basic template that can be used for different types of plots:

```
ggplot(data = <DATA>, mapping = aes(<MAPPINGS>)) +  <GEOM_FUNCTION>()
```

use the `ggplot()` function and bind the plot to a specific data frame using the `data` argument

```r
ggplot(data = olympics)
```

- define an aesthetic mapping (using the aesthetic (`aes`) function), by
  selecting the variables to be plotted and specifying how to present them in the
  graph, e.g., as x/y positions or characteristics such as size, shape, color, etc.

```r
ggplot(data = olympics, mapping = aes(x = year, y = height))
```

- add 'geoms' – graphical representations of the data in the plot (points,
  lines, bars). `ggplot2` offers many different geoms; we will use some
  common ones today, including:
  
  - `geom_point()` for scatter plots, dot plots, etc.
  - `geom_boxplot()` for, well, boxplots!
  - `geom_line()` for trend lines, time series, etc.

To add a geom to the plot use `+` operator. Because we have two continuous
variables, let's use `geom_point()` first:

```r
ggplot(data = olympics, aes(x = year, y = height)) +
  geom_point()
```

The `+` in the `ggplot2` package is particularly useful because it allows
you to modify existing `ggplot` objects. This means you can easily set up plot
"templates" and conveniently explore different types of plots, so the above
plot can also be generated with code like this:

```r
# Assign plot to a variable
height_plot <- ggplot(data = olympics,
                       mapping = aes(x = year, y = height))

# Draw the plot
height_plot +
    geom_point()
```

```r
## Create a ggplot and draw it.
height_plot <- ggplot(data = olympics,
                       aes(x = year, y = height))

height_plot +
  geom_point()
```

### Notes

- Anything you put in the `ggplot()` function can be seen by any geom layers
  that you add (i.e., these are universal plot settings). This includes the x-
  and y-axis you set up in `aes()`.
- You can also specify aesthetics for a given geom independently of the
  aesthetics defined globally in the `ggplot()` function.
- The `+` sign used to add layers must be placed at the end of each line
  containing a layer. If, instead, the `+` sign is added in the line before the
  other layer, `ggplot2` will not add the new layer and will return an error
  message.
- You may notice that we sometimes reference 'ggplot2' and sometimes 'ggplot'.
  To clarify, 'ggplot2' is the name of the most recent version of the package.
  However, any time we call the function itself, it's just called 'ggplot'.
- The previous version of the `ggplot2` package, called `ggplot`,
  which also contained the `ggplot()` function is now unsupported and has
  been removed from CRAN in order to reduce accidental installations
  and further confusion.

```r
# This is the correct syntax for adding layers
height_plot +
  geom_point()

# This will not add the new layer and will return an error message
# height_plot
#   + geom_point()
```

> <question-title noprefix>Challenge (optional)</question-title>
> 
> Scatter plots can be useful exploratory tools for small datasets. For data
> sets with large numbers of observations, such as the `olympics` data
> set, overplotting of points can be a limitation of scatter plots. One strategy
> for handling such settings is to use hexagonal binning of observations. The
> plot space is tessellated into hexagons. Each hexagon is assigned a color
> based on the number of observations that fall within its boundaries.  To use
> hexagonal binning with `ggplot2`, first install the R package `hexbin`
> from CRAN:
> 
> ```r
> # install.packages("hexbin")
> library(hexbin)
> ```
> 
> Then use the `geom_hex()` function:
> 
> ```r
> height_plot +
>  geom_hex()
> ```
> 
> What are the relative strengths and weaknesses of a hexagonal bin plot
> compared to a scatter plot? Examine the above scatter plot and compare it
> with the hexagonal bin plot that you created.
> 
{: .question}

## Data Cleaning & Calculations

We'll calculate some new fields to enable us to answer more questions, let's do that now.

```r
olympics <- olympics %>%
    mutate(age = year - birth_year) %>%
    mutate(weight = as.integer(weight)) %>%
    filter(!is.na(weight)) %>%
    filter(!is.na(height))
```

And to speed up future plots, let's pick three countries and three sports we're
interested in to reduce the amount of data we'll need to plot:

```r
sports = c("Archery", "Judo", "Speed Skating")
# You can change this to any of:
# c("Alpine Skiing", "Archery", "Art Competitions", "Artistic Gymnastics", "Artistic Swimming", "Athletics", "Badminton", "Baseball", "Basketball", "Biathlon", "Bobsleigh", "Bowling", "Boxing", "Canoe Marathon", "Canoe Slalom", "Canoe Sprint", "Cross Country Skiing", "Cycling BMX Freestyle", "Cycling BMX Racing", "Cycling Mountain Bike", "Cycling Road", "Cycling Track", "Diving", "Dogsled Racing", "Equestrian Dressage", "Equestrian Eventing", "Equestrian Jumping", "Fencing", "Figure Skating", "Freestyle Skiing", "Golf", "Handball", "Hockey", "Judo", "Karate", "Luge", "Marathon Swimming", "Military Ski Patrol", "Modern Pentathlon", "Nordic Combined", "Rhythmic Gymnastics", "Rowing", "Rugby", "Sailing", "Shooting", "Short Track Speed Skating", "Skateboarding", "Skeleton", "Ski Jumping", "Snowboarding", "Speed Skating", "Speed Skiing", "Surfing", "Swimming", "Table Tennis", "Taekwondo", "Tennis", "Trampolining", "Triathlon", "Tug-Of-War", "Volleyball", "Water Polo", "Weightlifting", "Winter Pentathlon", "Wrestling", "Wushu")

countries = c("NED", "USA", "CHN")

olympics_small <- olympics %>% filter(sport %in% sports) %>% filter(noc %in% countries)
```

## Building your plots iteratively

Building plots with `ggplot2` is typically an iterative process. We start by
defining the dataset we'll use, lay out the axes, and choose a geom:

```r
ggplot(olympics_small, aes(x=age, y=sport)) +
    geom_point()
```

Then, we start modifying this plot to extract more information from it. For
instance, we can add transparency (`alpha`) to avoid overplotting:

```r
ggplot(olympics_small, aes(x=age, y=sport)) +
    geom_point(alpha = 0.1)
```

We can also add colors for all the points:

```r
ggplot(olympics_small, aes(x=age, y=sport)) +
    geom_point(alpha = 0.1, color = "blue")
```

Or to color each species in the plot differently, you could use a vector as an input to the argument **color**. `ggplot2` will provide a different color corresponding to different values in the vector. Here is an example where we color with **`species_id`**:

```r
ggplot(olympics_small, aes(x=age, y=sport)) +
    geom_point(alpha = 0.1, aes(color = sex))
```


> <question-title noprefix>Challenge</question-title>
> 
> Use what you just learned to create a scatter plot of `height` over
> `sport` with the plot types showing the season in different colors.
> Is this a good way to show this type of data?
> 
> > <solution-title></solution-title>
> > 
> > ```r
> > ggplot(data = olympics,
> >        mapping = aes(x = height, y = sport)) +
> >    geom_point(aes(color = season))
> > ```
> {: .solution}
{: .question}


## Boxplot

We can use boxplots to visualize the distribution of height within each sport:

```r
ggplot(data = olympics_small, mapping = aes(x = height, y = sport)) +
    geom_boxplot()
```

But this is a bit boring with all three merged together, so let's colour by NOC.

```r
ggplot(olympics_small, aes(x = height, y = sport, fill=noc)) +
    geom_boxplot()
```

By adding points to the boxplot, we can have a better idea of the number of
measurements and of their distribution. Because the boxplot will show the outliers
by default these points will be plotted twice -- by `geom_boxplot` and
`geom_jitter`. To avoid this we must specify that no outliers should be added
to the boxplot by specifying `outlier.shape = NA`.

```r
ggplot(olympics_small, aes(x = height, y = sport, fill=noc)) +
    geom_boxplot(outlier.shape = NA) +
    geom_jitter(alpha = 0.3, color = "orange")
```

Notice how the boxplot layer is behind the jitter layer? What do you need to
change in the code to put the boxplot in front of the points such that it's not
hidden?

> <question-title noprefix>Challenge</question-title>
> 
> Boxplots are useful summaries, but hide the *shape* of the distribution. For
> example, if there is a bimodal distribution, it would not be observed with a
> boxplot. An alternative to the boxplot is the violin plot (sometimes known as
> a beanplot), where the shape (of the density of points) is drawn.
> 
> - Replace the box plot with a violin plot; see `geom_violin()`.
> 
> > <solution-title></solution-title>
> > ```
> > ggplot(olympics_small, aes(x = height, y = sport)) +
> >     geom_jitter(alpha = 0.3, color = "orange") +
> >     geom_violin()
> > ```
> {: .solution}
{: .question}

In many types of data, it is important to consider the *scale* of the
observations.  For example, it may be worth changing the scale of the axis to
better distribute the observations in the space of the plot.  Changing the scale
of the axes is done similarly to adding/modifying other components (i.e., by
incrementally adding commands). Try making these modifications:

- Represent height on the `log_10` scale; see `scale_y_log10()` and `scale_x_log10()`.

> <solution-title></solution-title>
> 
> ```r
> ggplot(olympics_small, aes(x = height, y = sport)) +
> scale_x_log10() +
> geom_jitter(alpha = 0.3, color = "orange") +
> geom_boxplot(outlier.shape = NA)
> ```
{: .solution}

So far, we've looked at the distribution of height within specific sports. Try making
a new plot to explore the distribution of another variable within each sport!

Create boxplot for `age` by sport.  Overlay the boxplot layer on a jitter
layer to show actual measurements.

> <solution-title></solution-title>
> ```r
> ggplot(olympics_small, aes(x = age, y = sport)) +
> geom_jitter(alpha = 0.3, color = "orange") +
> geom_boxplot(outlier.shape = NA)
> ```
{: .solution}

Add color to the data points on your boxplot according to the year from which
the sample was taken (`year`).

Hint: Check the class for `year`. Consider changing the class of `year`
from integer to factor. Why does this change how R makes the graph?


> <solution-title></solution-title>
> ```r
> ggplot(olympics_small, aes(x = age, y = sport)) +
>   geom_jitter(alpha = 0.3, aes(color=year)) +
>   geom_boxplot(outlier.shape = NA)
>
> # As a factor:
> ggplot(olympics_small, aes(x = age, y = sport)) +
>   geom_jitter(alpha = 0.3, aes(color=as.factor(year))) +
>   geom_boxplot(outlier.shape = NA)
> ```
{: .solution}

## Plotting time series data

Let's calculate number of participants per year for each games. First we need
to group the data and count records within each group:

```r
yearly_counts <- olympics %>% count(year, season)
```

Timelapse data can be visualized as a line plot with years on the x-axis and
counts on the y-axis:

```r
ggplot(data = yearly_counts, aes(x = year, y = n)) +
     geom_line()
```

Unfortunately, this does not work well because our data is quite sparse, datapoints only ever 2 or 4 years.
Let's instead use a box plot

```r
ggplot(data = yearly_counts, aes(x = year, y = n)) +
     geom_col()
```

We can't use `geom_box()` here, instead we should use `geom_col()` as our data is already aggregated.

We need to tell ggplot to draw a line for each season by modifying
the aesthetic function to include `group = season`:

```r
ggplot(data = yearly_counts, aes(x = year, y = n, group = season)) +
    geom_col()
```

We will be able to distinguish season in the plot if we add colors (using
`color` or `fill` also automatically groups the data:

```r
ggplot(data = yearly_counts, aes(x = year, y = n, fill = season)) +
    geom_col()
```

If you want the histograms to be side-by-side, we can do that with the "dodge" positioning:

```r
ggplot(data = yearly_counts, aes(x = year, y = n, fill = season)) +
    geom_col(position = "dodge")
```

## Integrating the pipe operator with ggplot2

In the previous lesson, we saw how to use the pipe operator `%>%` to use
different functions in a sequence and create a coherent workflow.
We can also use the pipe operator to pass the `data` argument to the
`ggplot()` function. The hard part is to remember that to build your ggplot,
you need to use `+` and not `%>%`.

```r
yearly_counts %>%
    ggplot(aes(x = year, y = n, fill = season)) +
    geom_col()
```

The pipe operator can also be used to link data manipulation with consequent data visualization.

```r
yearly_counts_graph <- olympics %>%
    count(year, season) %>%
    ggplot(mapping = aes(x = year, y = n, fill = season)) +
    geom_col()

yearly_counts_graph
```

## Faceting

`ggplot` has a special technique called *faceting* that allows the user to split
one plot into multiple plots based on a factor included in the dataset. We will
use it to make a time series plot for each season separately:

```r
yearly_counts %>%
    ggplot(aes(x = year, y = n, fill = season)) +
    geom_col() + facet_wrap(facets=vars(season))
```

Now we would like to split the line in each plot by the sex of each individual, sport, and their medal placement.
To do that we need to make counts in the data frame grouped by `year`, `season`, `sport`, `medal`, and `noc`.

```r
year_detail_counts <- olympics_small %>%
                      count(year, season, sport, medal, noc)
```

We can now make the faceted plot by splitting further by medal using `color`
(within a single plot), and per NOC:

```r
ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, color = medal)) +
  geom_line() +
  facet_wrap(facets =  vars(noc))
```

We can also facet both by NOC and sport:

```r
ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, color = medal)) +
  geom_line() +
  facet_grid(rows = vars(noc), cols =  vars(sport))
```

You can also organise the panels only by rows (or only by columns):

```r
# One column, facet by rows
ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, color = medal)) +
  geom_line() +
  facet_grid(rows = vars(noc))
```

```r
# One row, facet by column
ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, color = medal)) +
  geom_line() +
  facet_grid(cols = vars(noc))
```

> <tip-title>facet_grid/wrap with ~</tip-title>
> `ggplot2` before version 3.0.0 used formulas to specify how plots are faceted.
> If you encounter `facet_grid`/`wrap(...)` code containing `~`, please read
> [https://ggplot2.tidyverse.org/news/#tidy-evaluation](https://ggplot2.tidyverse.org/news/#tidy-evaluation).
{: .tip}

## `ggplot2` themes

Usually plots with white background look more readable when printed.
Every single component of a `ggplot` graph can be customized using the generic
`theme()` function, as we will see below. However, there are pre-loaded themes
available that change the overall appearance of the graph without much effort.

For example, we can change our previous graph to have a simpler white background
using the `theme_bw()` function:

```r
ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, color = medal))  +
    geom_line() +
    facet_grid(rows = vars(noc), cols =  vars(sport)) +
    theme_bw()
```

In addition to `theme_bw()`, which changes the plot background to white, `ggplot2`
comes with several other themes which can be useful to quickly change the look
of your visualization. The complete list of themes is [documented in the ggthemes reference](https://ggplot2.tidyverse.org/reference/ggtheme.html). `theme_minimal()` and
`theme_light()` are popular, and `theme_void()` can be useful as a starting
point to create a new hand-crafted theme.

The
[ggthemes](https://jrnold.github.io/ggthemes/reference/index.html) package
provides a wide variety of options.


> <question-title noprefix>Challenge</question-title>
> Use what you just learned to create a plot the relationship between height and weight,
> of participants, broken down by NOC and Sport.
> > <solution-title></solution-title>
> > ```r
> > ggplot(data = olympics_small, mapping = aes(x = height, y = weight, color = medal)) +
> >     geom_point() +
> >     facet_grid(rows = vars(noc), cols = vars(sport))
> > ```
> {: .solution}
{: .question}

## Customization

Take a look at the [`ggplot2` cheat sheet](https://posit.co/wp-content/uploads/2022/10/data-visualization-1.pdf), and
think of ways you could improve the plot.

Now, let's change names of axes to something more informative than 'year'
and 'n' and add a title to the figure:

```r
ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, fill = medal))  +
    geom_col() +
    facet_grid(rows = vars(noc), cols =  vars(sport)) +
    labs(title = "Participants and medals over the years",
         x = "Year",
         y = "Number of individuals") +
    theme_bw()
```

The axes have more informative names, but their readability can be improved by
increasing the font size. This can be done with the generic `theme()` function:

```r
ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, fill = medal))  +
    geom_col() +
    facet_grid(rows = vars(noc), cols =  vars(sport)) +
    labs(title = "Participants and medals over the years",
         x = "Year",
         y = "Number of individuals") +
    theme_bw()
    theme(text=element_text(size = 16))
```

Note that it is also possible to change the fonts of your plots. If you are on
Windows, you may have to install
the [`extrafont` package](https://github.com/wch/extrafont), and follow the
instructions included in the README for this package.

After our manipulations, you may notice that the values on the x-axis are still
not properly readable. Let's change the orientation of the labels and adjust
them vertically and horizontally so they don't overlap. You can use a 90 degree
angle, or experiment to find the appropriate angle for diagonally oriented
labels. We can also modify the facet label text (`strip.text`) to italicize the genus
names:

```r
ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, fill = medal))  +
    geom_col() +
    facet_grid(rows = vars(noc), cols =  vars(sport)) +
    labs(title = "Participants and medals over the years",
         x = "Year",
         y = "Number of individuals") +
    theme_bw()
    theme(axis.text.x = element_text(colour = "grey20", size = 12, angle = 90, hjust = 0.5, vjust = 0.5),
                        axis.text.y = element_text(colour = "grey20", size = 12),
                        strip.text = element_text(face = "italic"),
                        text = element_text(size = 16))
```

If you like the changes you created better than the default theme, you can save
them as an object to be able to easily apply them to other plots you may create:

```r
grey_theme <- theme(axis.text.x = element_text(colour="grey20", size = 12,
                                               angle = 90, hjust = 0.5,
                                               vjust = 0.5),
                    axis.text.y = element_text(colour = "grey20", size = 12),
                    text=element_text(size = 16))

ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, fill = medal))  +
    geom_col() +
    facet_grid(rows = vars(noc), cols =  vars(sport)) +
    grey_theme
```


> <question-title noprefix>Challenge</question-title>
>
> With all of this information in hand, please take another five minutes to either
> improve one of the plots generated in this exercise or create a beautiful graph
> of your own. Use the RStudio [`ggplot2` cheat sheet](https://posit.co/wp-content/uploads/2022/10/data-visualization-1.pdf)
> for inspiration.
> 
> Here are some ideas:
> 
> - See if you can change the plot type to another plot
> - Can you find a way to change the name of the legend? What about its labels?
> - Try using a different color palette (see [https://r-graphics.org/chapter-colors](https://r-graphics.org/chapter-colors)).
>
> > <solution-title></solution-title>
> > This optional exercise currently lacks solutions. If you have them, please feel free to contribute suggestions here :)
> {: .solution}
>
{: .question}


## Arranging plots

Faceting is a great tool for splitting one plot into multiple plots, but
sometimes you may want to produce a single figure that contains multiple plots
using different variables or even different data frames. The `patchwork`
package allows us to combine separate ggplots into a single figure while keeping
everything aligned properly. Like most R packages, we can install `patchwork`
from CRAN, the R package repository, if it isn't already available:

```r
# install.packages("patchwork")
```

After you have loaded the `patchwork` package you can use `+` to place plots
next to each other, `/` to arrange them vertically, and `plot_layout()` to
determine how much space each plot uses:

```r
library(patchwork)

plot_weight <- olympics_small %>% ggplot(aes(x=noc, y=weight)) +
  geom_boxplot() +
  labs(x = "NOC", y = expression(log[10](Weight))) +
  scale_y_log10()

plot_height <- olympics_small %>% ggplot(aes(x=noc, y=height)) +
  geom_boxplot() +
  labs(x = "NOC", y = expression(log[10](Height))) +
  scale_y_log10()

plot_weight / plot_height + plot_layout(heights = c(3, 2))
```

You can also use parentheses `()` to create more complex layouts. There are
many useful examples on the [patchwork website](https://patchwork.data-imaginist.com/)

## Exporting plots

After creating your plot, you can save it to a file in your favorite format. The
Export tab in the **Plot** pane in RStudio will save your plots at low
resolution, which will not be accepted by many journals and will not scale well
for posters. The [`ggplot2` extensions website](https://exts.ggplot2.tidyverse.org/) provides a list
of packages that extend the capabilities of `ggplot2`, including additional
themes.

Instead, use the `ggsave()` function, which allows you to easily change the
dimension and resolution of your plot by adjusting the appropriate arguments
(`width`, `height` and `dpi`):

```r
my_plot <- year_detail_counts %>% ggplot(aes(x = year, y = n, fill = medal))  +
    geom_col() +
    facet_grid(rows = vars(noc), cols =  vars(sport)) +
    labs(title = "Participants and medals over the years",
         x = "Year",
         y = "Number of individuals") +
    theme_bw()
    theme(axis.text.x = element_text(colour = "grey20", size = 12, angle = 90, hjust = 0.5, vjust = 0.5),
                        axis.text.y = element_text(colour = "grey20", size = 12),
                        strip.text = element_text(face = "italic"),
                        text = element_text(size = 16))

ggsave("name_of_file.png", my_plot, width = 15, height = 10)

## This also works for plots combined with patchwork
plot_combined <- plot_weight / plot_height + plot_layout(heights = c(3, 2))
ggsave("plot_combined.png", plot_combined, width = 10, dpi = 300)
```

Note: The parameters `width` and `height` also determine the font size in the
saved plot.

```r
### Final plotting challenge:
##  With all of this information in hand, please take another five
##  minutes to either improve one of the plots generated in this
##  exercise or create a beautiful graph of your own. Use the RStudio
##  ggplot2 cheat sheet for inspiration:
##  https://posit.co/wp-content/uploads/2022/10/data-visualization-1.pdf
```


topic: data-science
tutorial: python-basics-recap
title: Python - Introductory Graduation
difficulty: introductory
time_estimation: 1H30M
tools_json: []
requirements_json: []
tags_json: []
content_hash: b72bb235ded6b5449a31d3e038a87636
last_modified: 2026-05-15T22:28:11.100862
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-basics-recap/tutorial.html

content:
This module provide something like a recap of everything covered by the modular Python Introductory level curriculum. This serves as something of a graduation into the Intermediate tutorials which cover more advanced topics.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Review

This recapitulates the main points from all of the previous modular tutorials

## Math

Math in python works a lot like math in real life (from algebra onwards). Variables can be assigned, and worked with in the place of numbers

```python
x = 1
y = 2
z = x * y
```

We can use familiar math operators:

Operator | Operation
--- | ---
`+` | Addition
`-` | Subtraction
`*` | Multiplication
`/` | Division (`//` for rounded, integer division)

And some familiar operations require the use of the `math` module:

```python
import math
print(math.pow(2, 8))
print(math.sqrt(9))
```

## Functions

Functions were similarly analogous to algebra and mathematics, we can express `f(x) = x * 3` in python as:

```python
def f(x, y=3):
    z = y * 2
    return x * z
```

There are a few basic parts of a function:

- `def` starts a function *definition*
- it needs a name, here it's `f`
- between some parentheses are the arguments
    - the are arguments, just the variable name (`x`)
    - and keyword arguments, where there is a variable and a value (`y=3`)
- The function body
    - With one or more lines
    - Usually ending in a `return`

And we know we can nest functions, using functional composition, just like in math. In math functional composition was written `f(g(x))` and in python it's exactly the same:

```python
print(math.sqrt(math.pow(2, 4)))
```

Here we've nested three different functions (print is a function!). To read this we start in the middle (math.pow) and move outwards (math.sqrt, print).

## Types

There are lots of different datatypes in Python! The basic types are `bool`, `int`, `float`, `str`. Then we have more complex datatypes like `list` and `dict` which can contain the basic types (as well as other lists/dicts nested.)

Data type       | Examples                  | When to use it | When **not** to use it
---             | ---                       | ---- | ----
Boolean (`bool`) | `True`, `False` | If there are only two possible states, true or false | If your data is not binary
Integer (`int`) | 1, 0, -1023, 42           | Countable, singular items. How many patients are there, how many events did you record, how many variants are there in the sequence | If doubling or halving the value would not make sense: do not use for e.g. patient IDs, or phone numbers. If these are integers you might accidentally do math on the value.
Float (`float`) | 123.49, 3.14159, -3.33334 | If you need more precision or partial values. Recording distance between places, height, mass, etc.
Strings (`str`) | 'patient_12312', 'Jane Doe', '火锅' | To store free text, identifiers, sequence IDs, etc. | If it's truly a numeric value you can do calculations with, like adding or subtracting or doing statistics.
List / Array (`list`) | `['A', 1, 3.4, ['Nested']]` | If you need to store a list of items, like sequences from a file. Especially if you're reading in a table of data from a file. | If you want to retrieve individual values, and there are clear identifiers it might be better as a dict.
Dictionary / Associative Array / map (`dict`) | `{"weight": 3.4, "age": 12, "name": "Fluffy"}` | When you have identifiers for your data, and want to look them up by that value. E.g. looking up sequences by an identifier, or data about students based on their name. Counting values. | If you just have a list of items without identifiers, it makes more sense to just use a list.

There are a couple more datatypes we didn't cover in detail: `set`s, `tuple`, `None`, `enum`, `byte`, all of which [can be read about in Python's documentation.](https://docs.python.org/3/library/datatypes.html)

## Comparators

We have a couple of comparators available to use specifically for numeric values:

- `>`: greater than
- `<`: less than
- `>=`: greater than or equal to
- `<=`: less than or equal to

And a couple that can be used with numbers and strings (or other values!)

- `==`: equal to
- `!=`: does not equal

## Iterables

In Python, there is a class of things which can be easily looped over, called "iterables". All of the following are examples of iterable items:

- `range(10)`
- `'abcd'`, a string
- `['a', 'b' , 'c' , 'd']`, a list

## Flow Control

Basic flow control looks like `if`, `elif`, `else`:

```python
blood_oxygen_saturation = 92.3
altitude = 0

if blood_oxygen_saturation > 96 and altitude == 0:
    print("Healthy individual at sea level")
elif blood_oxygen_saturation > 92 and altitude >= 1000:
    print("Healthy value above sea level")
elif blood_oxygen_saturation == 0:
    print("Monitor failure")
else:
    print("Not good")
```

We must start with an `if`, then we can have one or more `elif`s, and 0 or 1 else to end our clause. If there is no `else`, it's like nothing happens, we just check the `if` and `elif`s and if none match, nothing happens by default.

We could use `and` to check if both conditions are true, and `or` to check if one condition is true.

```python
# And
for i in (True, False):
    for j in (True, False):
        print(f"{i} AND {j} => {i and j}")
# Or
for i in (True, False):
    for j in (True, False):
        print(f"{i} OR {j} => {i or j}")
```

And if we needed, we can invert conditions with not.

```python
# Not
for i in (True, False):
    print(f"NOT {i} => {not i}")
```

All of these components (`if/elif/else`, `and`, `or`, `not`, numerical and value comparators) let us build up

## Loops

Loops let us *loop* over the contents of something iterable (a string, a list, lines in a file). We write

```
for loopVariable in myIterable:
    # Do something
    print(loopVariable)
```

Each loop has:

- `for`, a keyword to start the loop
- a loop variable, here named `loopVariable` which is set automatically every iteration of the loop
- `in`, a keyword used in a loop
- something we want to iterate over like a list or string (which is really just a list of single characters) or lines in a file.
- a loop body where all the action happens

```python
a = 0
c = 0
t = 0
g = 0

for base in 'ACTGATGCYGGCA':
    if base == 'A':
        a = a + 1
    elif base == 'C':
        c = c + 1
    elif base == 'T':
        t = t + 1
    elif base == 'G':
        g = g + 1
    else:
        print("Unexpected base!")

print(f"a={a} c={c} t={t} g={g}")
```

## Files

In python you must `open()` a file handle, using one of the three modes (read, write, or append). Normally you must also later `close()` that file, but your life can be a bit

```python
with open('out.txt', 'w') as handle:
    handle.write("Здравствуйте ")
    handle.write("世界!\n")
    handle.write("Welcome!\n")

# Can no longer handle.write(), once we've exited the with block, the file is automatically closed for us.
```

There are several basic parts

- We use `with` to indicate we want to use the context manager for file opening (this is what automatically closes the file afterwards)
- `open(path, mode)` opens a file
- `as` is a keyword
- `handle` is the name of a file handle, something that represents the file which we can write to, or read from.

Additionally if you need a newline in your file, you *must* write it yourself with a `\n`.

The above code is equivalent to this, but it is not recommended, it's a bit harder to read, and it is very very common to forget to close files which is not ideal.

```python
handle = open('out.txt', 'w')
handle.write("Здравствуйте ")
handle.write("世界!\n")
handle.write("Welcome!\n")
handle.close()

# Can no longer write.
```

You can also read from a file:

```python
# Read the entire file as one giant string
with open('out.txt', 'r') as handle:
    print(handle.read())

# Or read it as separate lines.
with open('out.txt', 'r') as handle:
    print(handle.readlines())
```

## Exceptions

Sometimes things go wrong!

- You divide by zero (`ZeroDivisionError`)
- You try and add a string and a number (`TypeError`)
- You have incorrect indentation (`IndentationError`)
- Files are unreadable
- Files have the wrong permissions

If you expect that something will go wrong you can guard against it with a `try`/`except`

```python
try:
    print(1 / 0)
except ZeroDivisionError:
    print("Nope! I expected that!")
```

Some of the most common reasons to do this are when you're processing user input data. Users often input invalid data, unfortunately.

# Exercises

## Series Approximation

Write a program that computes the sum of an alternating series where each element of the series is an expression of the form

$$4\cdot\sum_{k=1}^{N} \dfrac{(-1)^{k+1}}{2 * k-1}$$

Use that expression and calculate the sum for various values of N like `10`, `1000`, `1000000`

```python
# Write your approximation here!
def calculate_sum(N):
    # for values in the range [1, N] (inclusive!)
        # calculate the expression (the bit after the Sigma)
    # sum up all of those numbers!
    return value

print(calculate_sum(10))
```

## Monte Carlo Simulation

You can use a monte carlo simulation to calculate the value of π. The easy way to do this is to take the region `x = [0, 1], y = [0, 1]`, and fill it with random points. For each point, calculate the distance to the origin. Calculate the ratio of the inside points to the total points, and multiply the value by 4 to estimate π.

You can use the `random` module to generate random values:

```python
import random

print(random.random())
print(random.random())
print(random.random())
```

Using the `random.random()` to generate x and y coordinates, write a function that:

- generates N random points
- calculate their distance to the origin
- Calculate the number of points that are `distance<=1`
- divide that by `N`, and multiply by `4`.

![Gif of the unit square 0, 1 on the x and y axes. Points are being randomly generated and points inside the unit circle are marked in red, outside in blue. A quarter of a circle starts to appear, and the numbers at the top show as N increases, the approximation of Pi improves.](https://upload.wikimedia.org/wikipedia/commons/8/84/Pi_30K.gif "Gif from the Wikipedia Article for Monte Carlo Method")

```python
# Write your code here!
import math

# Just a suggestion: write a function to calculate the distance to origin.
#
def distance(x, y):
    # ...
    return

def generate_random_point():
    # ....
    return [x, y]

def approximate(N=1000):
    # For every point in the range [0, N]
    #   check if it's distance is great than 1

    # Find the ratio of how many are distance<=1
    # and return 4 times that ratio.
    return 4 * x

# Try it with a couple N values like 1, 100, 100000,
n = 10000
# Since we're using a random number function, the result is different every
# time we run the simulation.
print(approximate(n))
print(approximate(n))
print(approximate(n))
```

## Sixpack

Sixpack is an old EMBOSS program which takes in a DNA sequence, and then for every frame, for both strands, emits every {ORF} that it sees.

```
  G  R  G  F  W  C  L  G  G  K  A  A  K  N  Y  R  E  K  S  V  D  V  A  G  Y  D  X   F1
   G  V  A  S  G  A  W  A  V  K  R  Q  K  T  T  V  K  S  R  W  M  W  R  V  M  M     F2
    A  W  L  L  V  P  G  R  *  S  G  K  K  L  P  *  K  V  G  G  C  G  G  L  *  X    F3
1 GGGCGTGGCTTCTGGTGCCTGGGCGGTAAAGCGGCAAAAAACTACCGTGAAAAGTCGGTGGATGTGGCGGGTTATGATG 79
  ----:----|----:----|----:----|----:----|----:----|----:----|----:----|----:----
1 CCCGCACCGAAGACCACGGACCCGCCATTTCGCCGTTTTTTGATGGCACTTTTCAGCCACCTACACCGCCCAATACTAC 79
   P  R  P  K  Q  H  R  P  P  L  A  A  F  F  *  R  S  F  D  T  S  T  A  P  *  S     F6
  X  A  H  S  R  T  G  P  R  Y  L  P  L  F  S  G  H  F  T  P  P  H  P  P  N  H  H   F5
    P  T  A  E  P  A  Q  A  T  F  R  C  F  V  V  T  F  L  R  H  I  H  R  T  I  I    F4
```

Here we see a DNA sequence `GGGCGTGGCTTCTGGTGCCTGGGCGGTAAAGCGGCAAAAAACTACCGTGAAAAGTCGGTGGATGTGGCGGGTTATGATG` which you'll use as input. Above is the translation of the sequence to protein, for each of the three frames (F1-6). Below is the reverse complement of the sequence, and the three frame translation again.

What sixpack does is:

```
orfs = []

for sequence in [forward, reverse_complement(forward)]:
    for frame in [sequence, sequence[1:], sequence[2:]]:
        # Remembering
        for potential start_codon:
            # accumulate until it sees a stop codon
            # and append it to the orfs array once it does.
```

Here are some variables for your convenience:

```python
start_codons = ['TTG', 'CTG', 'ATG']
stop_codons = ['TAA', 'TAG', 'TGA']

# And some convenience functions
def is_start_codon(codon):
    return codon in start_codons

def is_stop_codon(codon):
    return codon in stop_codons
```

It's a good exercise to rewrite `sixpack` in a very simplified version without most of the features in sixpack:

```python
# Write your code here!

# Some recommendations:
def reverse_complement(sequence):
    return ...

orfs = []

for sequence in [forward, reverse_complement(forward)]:
    for frame in [sequence, sequence[1:], sequence[2:]]:
        # Remembering
        for potential start_codon:
            # accumulate until it sees a stop codon
            # and append it to the orfs array once it does.
```


topic: data-science
tutorial: python-warmup-stat-ml
title: Python - Warm-up for statistics and machine learning
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["elixir", "ai-ml"]
content_hash: 34bea2aa5e1d4cf9c000cc69cba73367
last_modified: 2026-05-27T09:01:47.965082
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-warmup-stat-ml/tutorial.html

content:
> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Basic python



```python

X = []

for i in range(10):
    X.append( i**2 )

print(X)
```

    [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]



```python

for x in X:
    print(x)

```

    0
    1
    4
    9
    16
    25
    36
    49
    64
    81



```python
for x in X:
    if x%2 == 1:
        print(x,'is odd')
    else:
        print(x,'is even')
```

    0 is even
    1 is odd
    4 is even
    9 is odd
    16 is even
    25 is odd
    36 is even
    49 is odd
    64 is even
    81 is odd



```python
# list comprehension is a very fine way of compressing all this

X = [ i**2 for i in range(10) ]

Xeven = [ x for x in X if x%2 == 0 ]
Xodd = [ x for x in X if x%2 == 1 ]


print( 'X    ', X )
print( 'Xeven', Xeven )
print( 'Xodd ', Xodd )
```

    X     [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
    Xeven [0, 4, 16, 36, 64]
    Xodd  [1, 9, 25, 49, 81]


[back to the top](#top)

## numpy and vectorized operations


```python
import numpy as np

X_array = np.array(X)

print(X_array)
```

    [ 0  1  4  9 16 25 36 49 64 81]



```python
print(X_array / 2 )
```

    [ 0.   0.5  2.   4.5  8.  12.5 18.  24.5 32.  40.5]



```python
print( np.exp(X_array ) )
print( np.log(X_array ) )
```

    [1.00000000e+00 2.71828183e+00 5.45981500e+01 8.10308393e+03
     8.88611052e+06 7.20048993e+10 4.31123155e+15 1.90734657e+21
     6.23514908e+27 1.50609731e+35]
    [      -inf 0.         1.38629436 2.19722458 2.77258872 3.21887582
     3.58351894 3.8918203  4.15888308 4.39444915]


    /tmp/ipykernel_490123/2855859755.py:2: RuntimeWarning: divide by zero encountered in log
      print( np.log(X_array ) )



```python
print( 'shape' , X_array.shape )
print( 'mean ' , np.mean(X_array) )
print( 'standard deviation' , np.std(X_array) )
```

    shape (10,)
    mean  28.5
    standard deviation 26.852374196707448


### linspace and arange


```python
print( 'linspace 0,2,9 :' , np.linspace(0,2,9) , sep='\t' )
print( 'linspace -0.5,0.5,11 :' , np.linspace(-0.5,0.5,11) , sep='\t' )
print( 'linspace 10,0,11 :' , np.linspace(10,0,11) , sep='\t' )
```

    linspace 0,2,9 :	[0.   0.25 0.5  0.75 1.   1.25 1.5  1.75 2.  ]
    linspace -0.5,0.5,11 :	[-0.5 -0.4 -0.3 -0.2 -0.1  0.   0.1  0.2  0.3  0.4  0.5]
    linspace 10,0,11 :	[10.  9.  8.  7.  6.  5.  4.  3.  2.  1.  0.]



```python
print( "arange 0,2,0.1 :", np.arange(1.5,2,0.1) , sep='\t' )
print( "arange -1,1,0.125 :", np.arange(-1,1,0.125) , sep='\t' )
print( "arange 10,2 :", np.arange(10,2,1) , sep='\t' ) # reverse does not work!
```

    arange 0,2,0.1 :	[1.5 1.6 1.7 1.8 1.9]
    arange -1,1,0.125 :	[-1.    -0.875 -0.75  -0.625 -0.5   -0.375 -0.25  -0.125  0.     0.125
      0.25   0.375  0.5    0.625  0.75   0.875]
    arange 10,2 :	[]


## Basic plotting


```python
import matplotlib.pyplot as plt

plt.plot( [0,1,2,3] , [10,5,7,0.2] )
plt.show()
```



### Adding color, symbols, ...

`matplotlib` offers many options to customize the appearance of your plot.

Here are the (some) common arguments to `plot()` (which can also be applied to many other graphical representations):
 * `color` : could be given as a (red,green,blue) tuple, a [name](https://matplotlib.org/3.1.0/gallery/color/named_colors.html), a hex code, ...  (see [Something better here](https://matplotlib.org/tutorials/colors/colors.html) for all the options)
 * `marker` : symbols for the data point. `'.'` is a point, `'v'` a down triangle, ... see [Something better here](https://matplotlib.org/3.3.3/api/markers_api.html#module-matplotlib.markers) for the list of possibilities.
 * `linestyle` : style of the line. `'-'` is solid, `'--'` is dashed, `''` for no line. See [Something better here](https://matplotlib.org/3.3.3/gallery/lines_bars_and_markers/linestyles.html) for more options
 * `linewidth` : width of the lines
 * `markersize` : size of the markers

You are invited to experiment and explore these options. Here are a few examples:



```python
y1 = [1,2,3,10,5]
y2 = [10,9,7,5.5,6]
y3 = [4,3,1.5,1]

# green, dashed line, with circle markers
plt.plot( y1, color = 'green', marker = 'o', linestyle = '--', linewidth = 2, markersize = 8 )

# blue triangle with no line
plt.plot( y2, color = 'blue', marker = 'v', linestyle = '' , markersize = 16 )

# solid orange line
plt.plot(y3, color = 'orange', marker = '', linestyle = '-', linewidth = 4 )

plt.show()
```
    


Note that:
 * you can call plot several time in a row to make several lines appear (only `plt.show()` causes the figure to appear)
 * the frame of the picture automatically adjust to what it needs to show

### multiple subplots

Now would normally be when we show you how to add labels, titles and legends to figures. 

However, the way `matplotlib` is built, it is actually a bit more efficient to first learn how to create multiple subplots.


Creating multiple plots is possible with the function `plt.subplots()`.
Amon its many arguments, it takes:
 * `nrows` : number of subplot rows
 * `ncols` : number of subplot columns
 * `figsize` : tuple (width,height) of the figure

This function creates a Figure and an Axes object.
The Axes object can be either : 
 * a simple Axe is there is 1 row and 1 columns
 * a list of Axe objects if there is 1 row and multiple columns, or 1 column and multiple rows
 * a list of lists of Axes objects if there is multiple rows and multiple columns



```python
y1 = [1,2,3,10,5]
y2 = [10,9,7,5.5,6]
y3 = [4,3,1.5,1]


# subplots returns a Figure and an Axes object
fig, ax = plt.subplots(nrows=1, ncols=2) # 2 columns and 1 row

# ax is a list with two objects. Each object correspond to 1 subplot

# accessing to the first column ax[0]
ax[0].plot( y1, color = 'green', marker = 'o', linestyle = '--', linewidth = 2, markersize = 8 )

# accessing to the second column ax[1]
ax[1].plot( y2, color = 'blue', marker = 'v', linestyle = '' , markersize = 16 )
ax[1].plot( y3, color = 'orange', marker = '', linestyle = '-' )

plt.show()
```
    


Notice how we call `ax[0].plot(...)` instead of `plt.plot(...)` to specify in which subplots we want to plot.

### multiple subplots - continued

Let's see the same thing with several lines and several columns


```python
y1 = [1,2,3,10,5]
y2 = [10,9,7,5.5,6]
y3 = [4,3,1.5,1]
y4 = [1,2,3,7,5]

# 2 columns and 2 rows, and we also set the figure size
fig, ax = plt.subplots(nrows=2, ncols=2 , figsize = (12,12))

# ax is a list of two lists with two objects each.

# accessing to the first row, first column : ax[0][0]
ax[0][0].plot( y1, color = 'green', marker = 'o', linestyle = '--', linewidth = 2, markersize = 8 )

# accessing to the first row, second column : ax[0][1]
ax[0][1].plot( y2, color = 'blue', marker = 'v', linestyle = '' , markersize = 16 )

# accessing to the second row, first column : ax[1][0]
ax[1][0].plot( y3, color = 'orange', marker = 'x', linestyle = '-' )

# accessing to the first row, second column : ax[1][1]
ax[1][1].plot( y4, color = 'teal', linestyle = '-.' , linewidth=5 )

plt.show()
```
    


### setting up labels

To set the labels at the x-axis, y-axis and title, we use the method of the Axe object:
 * `.set_xlabel(...)`
 * `.set_ylabel(...)`
 * `.set_title(...) `



```python
y1 = [1,2,3,10,5]
y2 = [10,9,7,5.5,6]
y3 = [4,3,1.5,1]

# subplots returns a Figure and an Axes object
fig, ax = plt.subplots(nrows=1, ncols=2 , figsize=(10,5)) # 2 columns and 1 row


# accessing to the first column ax[0]
ax[0].plot( y1, color = 'green', marker = 'o', linestyle = '--', linewidth = 2, markersize = 8 )
ax[0].set_xlabel('x-axis label')
ax[0].set_ylabel('y-axis label')
ax[0].set_title('plot 1')


# accessing to the second column ax[1]
ax[1].plot( y2, color = 'blue', marker = 'v', linestyle = '' , markersize = 16 )
ax[1].plot( y3, color = 'orange', marker = '', linestyle = '-' )
ax[1].set_xlabel('x-axis label')
ax[1].set_ylabel('y-axis label')
ax[1].set_title('plot 2')

plt.show()
```
    


**setting up a legend** 

Each element we add to the figure using `plot()` can be given a label using the `label` argument.
Then, a legend may be added to the figure using the `legend()` method.

This `legend()` method can take a `loc` argument that specifies where it should be plotted. 
Possible values for this argument are: `'best' , 'upper right' , 'upper left' , 'lower left' , 'lower right' , 'right' , 'center left' , 'center right' , 'lower center' , 'upper center' , 'center'` (the default is `best`).



```python

fig, ax = plt.subplots(nrows=1, ncols=1 , figsize=(10,5)) # 2 columns and 1 row

# NB : with 1 col and 1 row, ax is directly the sole subplot we have
#      so to call it we just use ax.plot , ax.set_xlabel , ...

ax.plot( y1, color = 'green', marker = 'o', linestyle = '--', linewidth = 2 , label = 'line A' )
ax.plot( y2, color = 'blue', marker = 'v', linestyle = '' , markersize =  8 , label = 'line B' )
ax.plot( y3, color = 'orange', marker = '', linestyle = '-' , linewidth = 2 , label = 'line C' )

ax.set_xlabel('x-axis label')
ax.set_ylabel('y-axis label')
ax.set_title('plot with a legend')

#adding a legend in the upper right
ax.legend( loc='upper right')

plt.show()

```

    


### additional : writing a figure to a file

Writing a matplotlib figure to a file can be achieved simply by replacing the call to `plt.show()` to `plt.savefig(...)`.

`plt.savefig` takes a number of argument, the most commons are :
 * `fname` : name of the file to write the figure. The extension is used to determine the output format (.pdf,.png, .jpg , .svg ,  ...). Many formats are supported, you can get a list with this command : `plt.gcf().canvas.get_supported_filetypes()`
 * `dpi` : dots per inches , useful to set-up when saving to raster formats (ie., pixel-based such as png or jpeg). The actual size of the image is set using the argument `figsize` of `plt.subplots()`


> <comment-title></comment-title>
>
> in a jupyter notebook the figure will still be shown, whereas in a standard .py script it will not appear on screen.
>
{: .comment}

Here is a demonstration. Apply in on your side and verify that the file `testPlot.png` was created:


```python
import matplotlib.pyplot as plt

y1 = [1,2,3,10,5]
y2 = [10,9,7,5.5,6]
y3 = [4,3,1.5,1]


# subplots returns a Figure and an Axes object
fig, ax = plt.subplots(nrows=1, ncols=2 , figsize = (10,6) ) # 2 columns and 1 row

# ax is a list with two objects. Each object correspond to 1 subplot

# accessing to the first column ax[0]
ax[0].plot( y1, color = 'green', marker = 'o', linestyle = '--', linewidth = 2, markersize = 8 )

# accessing to the second column ax[1]
ax[1].plot( y2, color = 'blue', marker = 'v', linestyle = '' , markersize = 16 )
ax[1].plot( y3, color = 'orange', marker = '', linestyle = '-' )

plt.savefig( 'testPlot.png' , dpi = 90  )
```


## Exercise 00.01 : bringing together numpy and matplotlib

Numpy arrays can be plotted as if they were lists.

1. plot x and y, where:
    * y = 1/(1+exp(-x))
    * x varies between -5 and 5 (plotting around a 100 points should suffice).

2. **Bonus :** plot multiples lines : y = 1/(1+exp(-x*b)) , for the following values of b: 0.5 , 1 , 2 , 4.
    * x still varies between -5 and 5 (plotting around a 100 points should suffice).
    * put a legend in your plot




```python

```

You can load the solution directly in this notebook by uncommenting and running the following line:


```python
# %load  -r -8 solutions/solution_00_01.py
```

bonus question solution:


```python
# %load  -r 9- solutions/solution_00_01.py
```


## Generating random numbers


### the basics


```python
import numpy.random as rd

# random floats between 0 and 1
for i in range(4):
    print( rd.random() )

```

    0.6696103730869407
    0.7426639266737763
    0.6767219223242785
    0.8602105555191791



```python
print( rd.random(size=10) ) # draw directly 10 numbers
```

    [0.37971723 0.80354745 0.4168427  0.70867247 0.17547126 0.43760884
     0.75933345 0.06571168 0.45772397 0.67191214]


### setting the seed: pseudorandomness and reproducibility


```python
rd.seed(42) # setting the seed to 42
print( '1st draw' , rd.random(size=5) )
print( '2nd draw' , rd.random(size=5) )
rd.seed(42)
print( 'after resetting seed' , rd.random(size=5) )
```

    1st draw [0.37454012 0.95071431 0.73199394 0.59865848 0.15601864]
    2nd draw [0.15599452 0.05808361 0.86617615 0.60111501 0.70807258]
    after resetting seed [0.37454012 0.95071431 0.73199394 0.59865848 0.15601864]


### beyond the uniform distribution

numpy offers you quite a large [set of distributions you can draw from](https://docs.scipy.org/doc/numpy-1.15.0/reference/routines.random.html#distributions).

Let's look at the normal distribution:


```python

normalDraw = rd.normal(size = 1000 )

print( 'mean ' , np.mean( normalDraw ) )
print( 'stdev' , np.std( normalDraw ) )
```

    mean  0.025354699638558926
    stdev 1.0003731428167348



```python
normalDraw2 = rd.normal( loc = -2 , scale = 3 , size = 300 ) # loc chnages the location (mean), and scale changes the standard deviation

print( 'mean ' , np.mean( normalDraw2 ) )
print( 'stdev' , np.std( normalDraw2 ) )
```

    mean  -1.9773491637651965
    stdev 2.964622032924749


of course, we could want to plot these drawn numbers:


```python
plt.hist( normalDraw  , alpha = 0.5 , label='loc=0  , scale=1')
plt.hist( normalDraw2 , alpha = 0.5 , label='loc=-2 , scale=3')
plt.legend()
plt.show()
```


## Statistical testing

`numpy.random` let's you draw random numbers ;
`scipy.stats` implements the probability density functions, and Percent point function, as well as the most statistical tests.



```python
import scipy.stats as stats

# plotting the probability density function for 1 of the random draw we just made:

x = np.linspace(-10,10,1001)

normPDF = stats.norm.pdf( x , loc = -2 , scale = 3 )

plt.hist( normalDraw2 , alpha = 0.5 , label='random draw' , density = True) # don't forget density=True
plt.plot(x,normPDF , label='PDF' )
plt.legend()
plt.show()
```


We can also get the expected quantiles of a distribution:


```python
print( '95% quantile of a Chi-square distribution with 3 degrees of freedom:', stats.chi2.ppf(0.95 , df=3))
print( 'fraction of a Chi-square distribution with 3 degrees of freedom above of equal to 5' ,  
      1 - stats.chi2.cdf( 5 , df=3 ) )
```

    95% quantile of a Chi-square distribution with 3 degrees of freedom: 7.814727903251179
    fraction of a Chi-square distribution with 3 degrees of freedom above of equal to 5 0.17179714429673354


And you can apply some classical statistical tests:


```python
# t-test of independance between two random samples:
rd.seed(73)

s1 = rd.normal(size=67)
s2 = rd.normal(size=54 , loc = 0.2)

testStat , pval = stats.ttest_ind(s1,s2 , equal_var=True)  # equal variance : Student's t-test ; unequal : Welch's
#almost all of these stat functions return the same test-statistic , pvalue tuple

print('result of the t-test')
print('\tt:',testStat)
print('\tp-value:',pval)
```

    result of the t-test
        t: 0.26673986193074073
        p-value: 0.7901311339594405


### What is our conclusion for these tests results? What do you think about this?


```python

# Kolmogorov-smirnov test for a chi-square distribution

sample = rd.chisquare(df=13 , size = 43)


# kstest expect as second argument the cdf function of the reference distribution
# this is how to handle the fact that me must set an argument (degree of freedom)
refDistribution = stats.chi2(df=13).cdf

testStat , pval = stats.kstest( sample , refDistribution )
# alternative : 
# testStat , pval = stats.kstest( sample , lambda x : stats.chi2.cdf(x , df=13 ) )

print('result of the Kolmogorov-Smirnov test comparing our sample to a Chi-square distribution with 13 degrees of freedom')
print('\tK:',testStat)
print('\tp-value:',pval)

```

    result of the Kolmogorov-Smirnov test comparing our sample to a Chi-square distribution with 13 degrees of freedom
        K: 0.12249766392962913
        p-value: 0.5003109000967569


If you are interested, this [webpage](https://machinelearningmastery.com/statistical-hypothesis-tests-in-python-cheat-sheet/) references all implemented tests, with examples.

[back to the top](#top)

## Bringing together numpy, numpy.random, and matplotlib

The random generation function return a numpy array, meaning it is fairly trivial to combine it with other arrays:



```python
# combining 

x = np.sort( rd.normal(loc=170 , scale = 23 , size = 100) )

y_theoretical = 0.75 * x + 100 # simple linear relationship : y = a * x + b

measurement_noise = rd.normal(scale = 10 , size = 100) # some noise associated to the measure

y_observed = y_theoretical + measurement_noise # observed = expected + noise

fig,ax = plt.subplots(figsize=(8,8))
plt.plot( x , y_theoretical , label = 'expected' )
plt.plot( x , y_observed , marker = '.' , linestyle='' , alpha = 0.7 , label = 'observed')
plt.legend()
plt.show()
```



## The briefest intro to pandas

`pandas` is a powerful library when doing data analysis, especially in the forms of table.

Basically, it reimplements R data.frame as a DataFrame object and ties together neatly with the libraries we've just seen.



```python
 import pandas as pd
    
df = pd.read_table( 'data/beetle.csv' , sep=',' , index_col=0 ) # pandas automatically detects header.

df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>dose</th>
      <th>nexp</th>
      <th>ndied</th>
      <th>prop</th>
      <th>nalive</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1</th>
      <td>49.1</td>
      <td>59</td>
      <td>6</td>
      <td>0.102</td>
      <td>53</td>
    </tr>
    <tr>
      <th>2</th>
      <td>53.0</td>
      <td>60</td>
      <td>13</td>
      <td>0.217</td>
      <td>47</td>
    </tr>
    <tr>
      <th>3</th>
      <td>56.9</td>
      <td>62</td>
      <td>18</td>
      <td>0.290</td>
      <td>44</td>
    </tr>
    <tr>
      <th>4</th>
      <td>60.8</td>
      <td>56</td>
      <td>28</td>
      <td>0.500</td>
      <td>28</td>
    </tr>
    <tr>
      <th>5</th>
      <td>64.8</td>
      <td>63</td>
      <td>52</td>
      <td>0.825</td>
      <td>11</td>
    </tr>
  </tbody>
</table>
</div>




```python
Nrows, Ncols = df.shape
print( 'number of rows:',Nrows, 'number of columns:', Ncols )
print( 'column names' , df.columns )
```

    number of rows: 8 number of columns: 5
    column names Index(['dose', 'nexp', 'ndied', 'prop', 'nalive'], dtype='object')



```python
df.describe()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>dose</th>
      <th>nexp</th>
      <th>ndied</th>
      <th>prop</th>
      <th>nalive</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>count</th>
      <td>8.000000</td>
      <td>8.000000</td>
      <td>8.000000</td>
      <td>8.000000</td>
      <td>8.000000</td>
    </tr>
    <tr>
      <th>mean</th>
      <td>62.800000</td>
      <td>60.125000</td>
      <td>36.375000</td>
      <td>0.602000</td>
      <td>23.750000</td>
    </tr>
    <tr>
      <th>std</th>
      <td>9.599702</td>
      <td>2.232071</td>
      <td>22.557466</td>
      <td>0.367937</td>
      <td>21.985385</td>
    </tr>
    <tr>
      <th>min</th>
      <td>49.100000</td>
      <td>56.000000</td>
      <td>6.000000</td>
      <td>0.102000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>25%</th>
      <td>55.925000</td>
      <td>59.000000</td>
      <td>16.750000</td>
      <td>0.271750</td>
      <td>4.750000</td>
    </tr>
    <tr>
      <th>50%</th>
      <td>62.800000</td>
      <td>60.000000</td>
      <td>40.000000</td>
      <td>0.662500</td>
      <td>19.500000</td>
    </tr>
    <tr>
      <th>75%</th>
      <td>69.675000</td>
      <td>62.000000</td>
      <td>54.750000</td>
      <td>0.919500</td>
      <td>44.750000</td>
    </tr>
    <tr>
      <th>max</th>
      <td>76.500000</td>
      <td>63.000000</td>
      <td>61.000000</td>
      <td>1.000000</td>
      <td>53.000000</td>
    </tr>
  </tbody>
</table>
</div>




```python
# select a single column:
df['dose']
```




    1    49.1
    2    53.0
    3    56.9
    4    60.8
    5    64.8
    6    68.7
    7    72.6
    8    76.5
    Name: dose, dtype: float64




```python
df[ ['ndied','nalive'] ] # select several columns
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>ndied</th>
      <th>nalive</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1</th>
      <td>6</td>
      <td>53</td>
    </tr>
    <tr>
      <th>2</th>
      <td>13</td>
      <td>47</td>
    </tr>
    <tr>
      <th>3</th>
      <td>18</td>
      <td>44</td>
    </tr>
    <tr>
      <th>4</th>
      <td>28</td>
      <td>28</td>
    </tr>
    <tr>
      <th>5</th>
      <td>52</td>
      <td>11</td>
    </tr>
    <tr>
      <th>6</th>
      <td>53</td>
      <td>6</td>
    </tr>
    <tr>
      <th>7</th>
      <td>61</td>
      <td>1</td>
    </tr>
    <tr>
      <th>8</th>
      <td>60</td>
      <td>0</td>
    </tr>
  </tbody>
</table>
</div>



### Plotting DataFrame Columns

Because `DataFrame` columns are iterable, they can seamlessly be given as argument to `plot()`.


```python

# plotting the column dose along the x-axis and prop along the y-axis
# I use the + marker, with a teal color.
plt.plot(df['dose'] , df['prop'] , color = 'teal' , linestyle='' , marker = '+' , markersize=10 )
plt.xlabel( 'dose' )
plt.ylabel( 'proportion of dead' )
plt.show()
```
    


DataFrame column can be manipulated like numpy array:


```python

## we can combine columns using normal operators
Odds = df['nalive'] /df['ndied'] # the odds of being alive is nalive / ndead

## adding a new column to the DataFrame is trivial:
df['Odds'] = Odds


## we can also apply numpy function to them
df['logOdds'] = np.log( df['Odds'] )


plt.plot(df['dose'] , df['logOdds'] , color = 'teal' , linestyle='' , marker = '+' , markersize=10 )
plt.xlabel( 'dose' )
plt.ylabel( 'log Odds' )
plt.show()

```



## Exercise 00.02 : tying everything together

1. Read the file `'data/kyphosis.csv'`.
2. how many columns are there ?
3. What is the maximum Age ? 
4. create a new column `Stop` , corresponding to the addition of columns `'Start'` and `'Number'`
5. plot the relationship between `'Age'` and `'Number'` (bonus point : use colors to indicate the presence or absence of kyphosis ).





```python

```

Solutions:


```python
# %load  -r -7 solutions/solution_00_02.py
```


```python
# %load  -r 8-9 solutions/solution_00_02.py
```


```python
# %load  -r 11-12 solutions/solution_00_02.py
```


```python
# %load  -r 14-15 solutions/solution_00_02.py
```


```python
# %load  -r 17-22 solutions/solution_00_02.py
```


```python
# %load  -r 24- solutions/solution_00_02.py
```


topic: data-science
tutorial: data-manipulation-olympics
title: Data Manipulation Olympics
difficulty: introductory
time_estimation: 1h
requirements_json: []
tags_json: ["cyoa"]
content_hash: 8912bee40567881eba6ddded731cfa42
last_modified: 2026-05-27T09:01:49.212310
zenodo_link: https://zenodo.org/record/6803028
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/data-manipulation-olympics/tutorial.html

content:
Scientific analyses often consist of a number of tools that run one after the other, in order to go from the raw data to scientific insight. Between these specialized tools, simple data manipulation steps are often needed as a kind of "glue" between tools. For example, the output of tool A may produce a file that contains all the information needed as input for tool B, but tool B expects the columns in a different order. Or in genomic data analysis, some tools expect chromosome X to be listed as `chrX`, while others simply expect `X`. In these situations, extra data manipulation steps are needed to prepare files for input to analysis tools.

Galaxy has a large collection of tools to perform such basic data manipulation tasks, and becoming familiar with these operations will allow to perform your analysis more easily in Galaxy (and outside).


<!--
Note to contributors: feel free to add sections here to include additional data manipulation options.
Make sure each section is independent of each other, i.e. each section should start with the olympics.tsv file.
Also make sure to include many exercises (with answers) for your section!
-->


<!-- set up some variables to easily update tool versions throughout tutorial, since most tools are used many times %} -->
{% assign version_cat="cat1" %}
{% assign version_compute="toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.0" %}
{% assign version_count="Count1" %}
{% assign version_cut_columns="Cut1" %}
{% assign version_cut_advanced="toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0" %}
{% assign version_datamash="toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.1.0+galaxy2" %}
{% assign version_filter="Filter1" %}
{% assign version_join="join1" %}

{% assign version_remove_beginning="Remove beginning1" %}
{% assign version_remove_columns_by_header="toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/0.0.1" %}
{% assign version_replace_text_column="toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.1" %}
{% assign version_replace_text_line="toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.1" %}
{% assign version_paste="Paste1" %}

{% assign version_select_first="Show beginning1" %}
{% assign version_sort="toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/1.1.1" %}
{% assign version_split="toolshed.g2.bx.psu.edu/repos/bgruening/split_file_on_column/tp_split_on_column/0.6" %}
{% assign version_unique="toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sorted_uniq/1.1.0" %}
{% assign version_wc="wc_gnu" %}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Cheatsheet

Here is an overview table of the different data manipulations in this tutorial, with links to the tools in Galaxy.

If you've opened this tutorial via the {% icon level %} icon in Galaxy (top menu bar), you can click on the tool names in the last column to quickly open them in Galaxy and start using them on your own!

<!-- GTN:IGNORE:006 -->

| Operation              | Description                        | Galaxy Tool    |
|------------------------|------------------------------------|----------------|
| Convert format         | Change the file format             | {% icon galaxy-pencil%} Edit attributes |
| Word count             | Count the number of lines, words and characters in a file | {% tool [Line/Word/Character count]({{version_wc}}) %} |
| Sort on a column       | Change the order of the rows based on values in one or more columns | {% tool [Sort]({{version_sort}}) %} |
| Filter                 | Remove rows based on values in one or more columns | {% tool [Filter]({{version_filter}}) %}|
| Counting               | Count occurrences of values in a column   | {% tool [Count]({{version_count}}) %} , {% tool [Datamash]({{version_datamash}}) %} |
| Group on a column      | And perform simple operations (count, mean, min, max etc) | {% tool [Group](Grouping1) %} , {% tool [Datamash]({{version_datamash}}) %} |
| Compute on rows        | to derive new column values from existing ones | {% tool [Compute]({{version_compute}}) %} |
| Find and Replace       | in a specific column               | {% tool [Column Regex Find and Replace]({{version_replace_text_column}}) %}|
| Find and Replace       | on every line                      | {% tool [Regex Find and Replace]({{version_replace_text_line}}) %}|
| Join two Datasets      | side by side on a specified field  | {% tool [Join two Datasets]({{version_join}}) %} |
| Concatenate datasets   | one after the other                | {% tool [Concatenate multiple datasets or collections]({{version_cat}}) %} |
| Remove Beginning       | Good for removing header lines     | {% tool [Remove beginning of a file]({{version_remove_beginning}}) %} |
| Select First lines     | Good for finding top 10s or saving header lines | {% tool [Select first lines]({{version_select_first}}) %} |
| Cut Columns            | By header name                     | {% tool [Remove columns by heading]({{version_remove_columns_by_header}}) %}|
| Cut Columns            | By column number                   | {% tool [Cut columns from a table]({{version_cut_columns}}) %}, {% tool [Advanced Cut columns from a table]({{version_cut_advanced}}) %}|
| Paste                  | Two files side by side             | {% tool [Paste]({{version_paste}}) %} |
| Split file             | Based on values of a column        | {% tool [Split]({{version_split}}) %} |
| Unique                 | Remove duplicate rows              | {% tool [Unique]({{version_unique}}) %} |

**TIP: Adding tools to your Favourites:** If you find yourself frequently using the same tool often but struggle to find it back in the long list of tools, you can **star** your favourite tools in Galaxy!

{% snippet faqs/galaxy/tools_favorite.md %}


In this tutorial, these tools are explained in more detail, and we provide some exercises for you to practice.


# Background

In this tutorial, we will use as our dataset a table with results from the Olympics, from the games in Athens in 1896 until Tokyo in 2020. The objective is to familiarize you with a large number of the most important data manipulation tools in Galaxy. Much like the Olympics, there are many different disciplines (types of operations), and for each operation there are often multiple techniques (tools) available to athletes (data analysts, you) that are great for achieving the goal.


![image of olympic rings, logo and two athletes around the words "Data Analysis Olympics"](./images/cover.jpg)


We will show you many of these commonly needed data manipulation operations, and some examples of how to perform them in Galaxy. We also provide many exercises so that you can train your skills and become a data manipulation Olympian!


# Upload Data

Before we can do any manipulation, we will need some data. Let's upload our table with Olympics results now.

> <hands-on-title>Get data</hands-on-title>
>
> 1. {% tool [Import](upload1) %} the file `olympics.tsv` via link
>
>    ```
>    {{page.zenodo_link}}/files/olympics.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 2. **Expand** on the item in your history to see some metadata and a short preview of the contents.
>
> 3. **View** {% icon galaxy-eye %} the dataset by clicking on the eye icon.
>
>    > <question-title></question-title>
>    >
>    > 1. What is the format of the file?
>    > 2. What does each row represent?
>    > 3. How many lines are in the file? (Hint: {% tool [Line/Word/Character count](wc_gnu) %})
>    > 4. How many columns?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. When you expand the `olympics.tsv` dataset in your history (see also screenshot below), you will see `format: tabular`, this is another term for a tab-separated (`tsv`) file.
>    > > 2. Each row represents an athlete's participation in an event. If an athlete competes in multiple events, there is a line for each event.
>    > > 3. 234,523. Look at the expanded view in your history, this tells us there are ~250,000 rows in the dataset. We can get the exact number using the  {% tool [Line/Word/Character count](wc_gnu) %} tool.
>    > > 4. There are 17 columns in this file. There are multiple ways to find this answer:
>    > >    - Count the columns manually (only doable for small files)
>    > >    - In the expanded view, scroll sideways on the dataset preview, at the top the columns are numbered
>    > >    - Click on the {% icon galaxy-info %} i icon on the dataset; here you will find more detailed information about the file and the job that created it.
>    > >      At the bottom is also a preview (peek) of the dataset and numbered columns
>    > >
>    > > ![a screenshot of the expanded view of the dataset in the history, it shows the datatype, number of lines in the file, and a preview
>    > > of the dataset with numbered columns](./images/columns-number.png)
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


## About this dataset

The data was [obtained](https://github.com/UOSCS/Olympic_Athletes) from [Olympedia](https://www.olympedia.org/). The file `olympics.tsv` contains
234,522 rows and 17 columns. Each row corresponds to an individual athlete competing in an individual Olympic event. The columns are:

- **athlete_id** - Unique number for each athlete
- **name** - Athlete's name
- **sex** - M or F
- **birth_year** - 4-digit number
- **birth_day** - e.g. 24 July
- **birth_place** - town and/or country
- **height** - In centimeters (or `NA` if data not known)
- **weight** - In kilograms (or `NA` if data not known)
- **team** - Team name
- **noc** - National Olympic Committee 3-letter code
- **games** - Year and season
- **year** - Integer
- **season** - Summer or Winter
- **city** - Host city
- **sport** - Sport
- **event** - Event
- **medal** - Gold, Silver, Bronze (or `NA` if no medal was won)

We will use this dataset to practice our data manipulation skills in Galaxy.


# Choose your adventure!

This tutorial is structured a bit differently than most. **You do not have to do the steps in the order they are presented below.** Every section in this tutorial uses the dataset you just uploaded (the `olympics.tsv` file) as input, so you can jump to any section in this tutorial right now if you have a particular data manipulation operation in mind you want to learn more about.




# File Format Conversion

The file we uploaded is a `.tsv` file. This stands for *tab-separated values*. This means that this is a file containing rows and columns, where a TAB character is used to signify a column ends and a new one begins. Galaxy is great at understanding tab-separated files, and most of the data manipulation tools are designed to work with such files.

A similar format you may come across a lot in data science, is the `.csv` file, or *comma-separated values* file. This is the same as `.tsv`, but uses comma (`,`) characters to indicate new columns, instead of TAB (`\t`) characters.

Galaxy can convert these two formats into each other.



> <hands-on-title>Convert Tabular to CSV</hands-on-title>
>
> 1. Convert the `olympics.tsv` file into a CSV file
>
>    {% snippet faqs/galaxy/datasets_convert_datatype.md conversion="csv (using 'Convert tabular to CSV')" %}
>
> 2. {% icon galaxy-eye %} **View the converted dataset**
>
>    > <question-title></question-title>
>    >
>    > 1. What do you notice?
>    > 2. Why are some values in quotes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The display in the center panel should be indistinguishable from before the conversion; however, if you look at the "quick view" of the data in the history step, you will see the commas separating columns.
>    > > 2. If the data in a column contains a comma (e.g. in this file we have events such as `swimming 5,000 meters`), we put the value in quotes to signify that that comma is part of the data, not a column delimiter.
>    > >
>    > {: .solution}
>    {: .question}
>
>
{: .hands_on}


## Converting vs Changing the datatype

The **file format conversion** step changed the dataset itself (by changing TABs to commas), and therefore created another dataset in your history. It is also possible to **change the datatype**; this does not change the underlying file/data, but just tells Galaxy what type of file it is. Galaxy uses this information in a variety of ways, such as:
  - How to display the data when you click the {% icon galaxy-eye %} **View** button
  - Which visualization options to offer
  - Filtering the list of possible inputs for tools (to only allow those of the correct datatype and protect you from mistakes)
  - Which file format conversions to make available
  - etc..


When you upload a file to Galaxy, by default it will attempt to auto-detect the file format. This is very often correct, but some times you may need to manually set the datatype to the correct value.


> <hands-on-title>Change datatype from Tabular to TXT</hands-on-title>
>
> 1. **Change** the datatype of the `olympics.tsv` file to `txt`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="txt" %}
>
> 2. {% icon galaxy-eye %} **View the dataset** again
>
>    > <question-title></question-title>
>    >
>    > 1. What do you see?
>    > 2. Why didn't this step create a new item in your history?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Since Galaxy now no longer know that this is a tabular file with rows and columns,
>    > > it displays the data as-is, no longer arranging the view into neat columns.
>    > >
>    > > 2. The file itself did not change, only the metadata. We simply told Galaxy to treat this file as a plain text file, instead of a tabular file.
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. **Change** the file back from `txt` to `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}



# Sorting

We have a lot of data in this file, but it is ordered by the athlete ID number, which is a somewhat arbitrary and meaningless number. But we can sort the rows in this file to something more convenient, for example alphabetically by name of the athlete, or chronologically by year of the Olympics.

> <hands-on-title>Sort table based on a column</hands-on-title>
>
> 1. We will sort the file in chronological order based on the year of the Olympic games
>
>    > <question-title></question-title>
>    >
>    > 1. Which column contains the year?
>    > 2. Do we want ascending or descending order if we want the oldest games at the top?
>    > 3. What should we do with the very first row (the one with the header names?)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Column 12
>    > > 2. The file should be sorted in ascending (increasing) order
>    > > 3. The header line should always stay on top, so we want to ignore it when we sort the file.
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. Open the {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} tool
>    - Read the help text at the bottom of the tool form
>    - Which settings do you think we need to use?
>
> 3. {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} with the following parameters:
>    - {% icon param-file %} *"Sort Query"*: `olympics.tsv`
>    - {% icon param-text %} *"Number of header lines"*: `1``
>    - Column selections
>      - {% icon param-select %} *"on column"*: `Column 12`
>      - {% icon param-select %} *"in"*: `Ascending order`
>      - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
>
> 4. {% icon galaxy-eye %} **View** the sorted file.
>
>    > <question-title></question-title>
>    >
>    > Which athlete is listed at the top of the file now?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. J. Defert. Who competed in a Tennis event 1896 Summer Olympics in Athens.
>    > >
>    > {: .solution}
>    {: .question}
>
> 5. **Rename** {% icon galaxy-pencil %} the resulting file to somthing meaningful (e.g. `Olympics in chronological order`)
>    - It can be easy to lose track of what steps you have performed and what each output contains, especially if you are doing many
>      data manipulation steps. Do future-you a favour and give datasets good names!
>
{: .hands_on}

This is great, but maybe it would make more sense to sort alphabetically by athlete name *within each year*.

## Sort on multiple columns at once

So we want to sort twice, first by year, and then within each year, we sort again alphabetically by name. The sort tool can do this!


> <hands-on-title>Sort table based on a column</hands-on-title>
>
> 1. We will sort the file in chronological order based on the year of the Olympic games
>
>    > <question-title></question-title>
>    >
>    > 1. Which column contains athlete names?
>    > 2. Do we want ascending or descending order if we want to sort alphabetically with A first?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Column 2
>    > > 2. The file should be sorted in ascending (increasing) order
>    > >
>    > {: .solution}
>    {: .question}
>
>
> 2. {% icon galaxy-refresh %} **Rerun** the sort tool with the following parameters:
>    - All parameter from the first step should already be set for you, and should remain the same
>    - {% icon param-repeat %} Insert Column selections
>      - {% icon param-select %} *"on column"*: `Column 2`
>      - {% icon param-select %} *"in"*: `Ascending order`
>      - {% icon param-select %} *"Flavor"*: `Alphabetical sort`
>
> 3. {% icon galaxy-eye %} **View** the sorted file.
>
>    > <question-title></question-title>
>    >
>    > Which athlete is listed at the top now? Which discipline (sport) did they compete in?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. A. Grigoriadis. He competed in the 500 meters freestyle swimming event.
>    > >
>    > {: .solution}
>    {: .question}
>
> 4. **Rename** {% icon galaxy-pencil %} the resulting file to somthing meaningful
>
{: .hands_on}

## Exercises

Ok, time to train! Let's see if you can use the sort tool to answer the following questions:

> <question-title noprefix>Exercise: Reverse the sort</question-title>
>
> Which athlete comes *last by alphabet*, in the *most recent* Olympics?
>
> > <solution-title></solution-title>
> >
> > `Žolt Peto` who competed in table tennis at the 2020 Summer Olympics in Tokyo.
> > <br>
> > We do this by repeating the previous sort (on year and then name), but changing the order to *descending* for both, to get the answer to the top of the file.
> >
> {: .solution}
{: .question}


> <question-title noprefix>Exercise: sort by height</question-title>
>
> 1. What is the height of the tallest competing athlete? Which athlete(s) are of this height?
> 2. What is the shortest?
> 3. Who was the tallest athlete from the most recent Olympics? How tall were they?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Height is listed in column 7. Since this is a simple number (no scientific notation), we can use *fast numerical sort*,
> >    and because we want the tallest on top, we will need to sort in *descending* (decreasing) order.
> > 2. Rerun the tool for step 1, but change the order to *ascending*
> > 3. First sort by year (descending), then by height (descending)
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. Adam Sandurski from Poland is the tallest athlete in the file, at 214 cm tall.
> >  2. Here we don't get a clear answer. That is because not all athletes have height data available, and blank fields (`NA`) are being sorted to the top. We can filter out rows with missing data to get our answer (see [Filter](#filtering) section to learn how to do this). For now we cannot answer this question with just the sort tool. Some times multiple tools are required to perform such tasks. The [exercise section at the end of this tutorial](#exercises-putting-it-all-together) has many exercises that require a combination of different tools.
> >  3. Gennaro Di Mauro, 210 cm. (2020 Summer Olympics in Tokyo)
> >
> {: .solution}
>
> > <solution-title noprefix>Full Solutions</solution-title>
> >
> > 1. {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} with the following parameters:
> >   - {% icon param-file %} *"Sort Query"*: `olympics.tsv`
> >   - {% icon param-text %} *"Number of header lines"*: `1``
> >   - Column selections
> >     - {% icon param-select %} *"on column"*: `Column 7`
> >     - {% icon param-select %} *"in"*: `Descending order`
> >     - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
> >
> > 2. {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} with the following parameters:
> >   - {% icon param-file %} *"Sort Query"*: `olympics.tsv`
> >   - {% icon param-text %} *"Number of header lines"*: `1``
> >   - Column selections
> >     - {% icon param-select %} *"on column"*: `Column 7`
> >     - {% icon param-select %} *"in"*: `Ascending order`
> >     - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
> >
> > 3. {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} with the following parameters:
> >   - {% icon param-file %} *"Sort Query"*: `olympics.tsv`
> >   - {% icon param-text %} *"Number of header lines"*: `1``
> >   - Column selections
> >     - {% icon param-select %} *"on column"*: `Column 12` (year column)
> >     - {% icon param-select %} *"in"*: `Descending order` (most recent on top)
> >     - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
> >   - {% icon param-repeat %} Insert Column selections
> >     - {% icon param-select %} *"on column"*: `Column 7` (height column)
> >     - {% icon param-select %} *"in"*: `Descending order` (tallest first)
> >     - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
> >
> {: .solution}
>
{: .question}


# Filtering

This file contains a lot of data, but we may only be interested in a subset of this data. For example, we may only want to look at one particular Olympics, or one particular sport. In such cases we can filter the dataset. This will create a new dataset, removing any rows that are not of interest to us (i.e. that don't meet the criteria we provide).

## Filtering based on column contents

> <hands-on-title>Filter table based on a column</hands-on-title>
>
> We will filter the file to show only winter Olympics
>
> 1. Look at the `olympics.tsv` file and answer the following questions
>
>    > <question-title></question-title>
>    >
>    > 1. Which column contains this information?
>    > 2. Which values can this column have? (make sure to notice capitalisation, 'Winter' is not the same as 'winter' to these tools)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Column 13, the column with the *season* header
>    > > 2. The values can be `Summer` or `Winter`
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. Open the tool {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %}
>    - read the help text at the bottom of the tool carefully
>    - get familiar with how to indicate columns and write expressions for filtering
>
>    > <question-title></question-title>
>    >
>    > 1. How would you write the expressions for the following conditions:
>    >    1. column 6 must be 'Yes'
>    >    2. column 13 must be smaller than 75
>    >    3. column 7 cannot be 'NA'
>    >    4. column 2 cannot be empty
>    >
>    > 2. It is also possible to combine multiple conditions, using `and`, `or`, `not` and parentheses
>    >    How would you write expressions for the following filtering conditions:
>    >    1. column 5 is larger than 2 or smaller than -2
>    >    2. column 5 is larger than 2 and smaller than 10
>    >    3. the sum of columns 4 and 6 is greater than or equal to 25,000
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The answers are:
>    > >    1. `c6=='Yes'`
>    > >    2. `c13<75`
>    > >    3. `c7!='NA'`
>    > >    4. `c2!=''`
>    > >
>    > > 2. The answers are:
>    > >    1. `c5>2 or c5<-2`
>    > >    2. `c5>2 and c5<10`
>    > >    3. `c4+c6 >= 25000`
>    > >
>    > {: .solution}
>    {: .question}
>
>    Ok, great, now that you've got the hang of writing expressions for this tool, let's create a file with only Winter Olympics:
>
> 3. {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `olympics.tsv`
>    - {% icon param-text %} *"With the following condition"*: `c13=='Winter'`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
> 4. {% icon galaxy-eye %} **View** the filtered file.
>
>    > <question-title></question-title>
>    >
>    > How many lines are in this file? (Hint: expand the dataset in your history or use {% tool [Line/Word/Character count]({{version_wc}}) %} )
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 44,681 (this is including the header line)
>    > >
>    > {: .solution}
>    {: .question}
>
> 5. **Repeat** the step for the Summer Olympics
>
>    > <question-title></question-title>
>    >
>    > 1. How many lines do you expect in this file?
>    > 2. How many lines are in this file? Were you right?
>    >
>    > > <solution-title noprefix>Hints</solution-title>
>    > >
>    > > 1. Use the {% tool [Line/Word/Character count]({{version_wc}}) %} to find the number of lines in the `olympics.tsv` file and subtract the number of rows in the Winter Olympics file
>    > > 2. Be careful to consider whether these counts include the header line of the file or not
>    > >
>    > {: .solution}
>    >
>    > > <solution-title noprefix>Answers</solution-title>
>    > >
>    > > 1. The original file has 234,523 lines, and the Winter Olympics had 44,681 lines. So we would expect 234,523 - 44,681 = 189,842 rows of data. Since we have subtracted the header line in this equation as well, we expect the Summer Olympics file to have 1 more line that this, so 189,843 total lines.
>    > > 2. 189,843. If you were off by one or two lines, it may have been that you counted the header lines double.
>    > > <br>
>    > > It is always useful to take a moment to think about the expected outcome; this makes it easier to spot mistakes and will save you time in the long run.
>    > >
>    > {: .solution}
>    {: .question}
>
> 6. **Rename** {% icon galaxy-pencil %} both outputs to something descriptive (e.g. `Filter: Winter Olympics only` etc)
>    - This is primarily to get you into the habit. If this is a file you might want to view or use again later, it will be hard to find back again unless you name it well.
>
{: .hands_on}

## Advanced Filtering with Python Syntax

In addition to basic filtering, the Galaxy filtering tool allows for more advanced filtering using Python syntax. This can be particularly useful when you need to apply more complex conditions or manipulate the data in specific ways.

> <comment-title> Understanding Python Syntax for Filtering </comment-title>
>
>    Before we dive into specific examples, let's briefly discuss some basic Python concepts that will help you understand how to write these filtering expressions:
>
> - **Methods**: In Python, methods are functions that are associated with an object. For example, in the expression `c2.startswith('Liu')`, `startswith` is a method that checks if the value in column 2 (`c2`) starts with the substring 'Liu'.
>
> - **Boolean Expressions**: These are expressions that evaluate to either `True` or `False`. For example, `c2.startswith('Liu')` will return `True` if the value in column 2 starts with 'Liu', and `False` otherwise.
>
> - **Comparison Operators**: These include `==` (equal to), `!=` (not equal to), `<` (less than), `>` (greater than), `<=` (less than or equal to), and `>=` (greater than or equal to). These operators are used to compare values and are commonly used in filtering.
>
> - **String Methods**: Strings in Python have several built-in methods that can be used for filtering. Some commonly used ones include:
>   - `startswith(substring)`: Checks if a string starts with a specified substring.
>   - `endswith(substring)`: Checks if a string ends with a specified substring.
>   - `find(substring)`: Returns the lowest index of the substring if it is found in the string; otherwise, it returns `-1`.
>   - `count(substring)`: Returns the number of non-overlapping occurrences of the substring in the string.
>
{: .comment}

These concepts will be used in the filtering expressions we write in Galaxy. Let's now explore some specific scenarios where you might use these expressions.


### Filtering Based on Substring Occurrence

We want to filter rows where a particular column contains a specific substring using the `cX.find()` method, where `X` is the column number.

For example, while you can easily filter Summer or Winter Olympics using `c13=="Summer"` or `c13=="Winter"` if the data is stored in a dedicated column, sometimes the data may not be that straightforward. The relevant information might be embedded in another column, like in `c11` with entries such as "1992 Summer Olympics".

In such cases, you can use substring filtering:

> <question-title></question-title>
>
> 1. How would you filter the rows where the 11th column contains "Summer"?
> 2. How would you filter the rows where the 11th column does **not** contain "Summer"?
>
> > <solution-title>Answers</solution-title>
> >
> > 1. `c11.find('Summer') != -1`
> > 2. `c11.find('Summer') == -1`
> >
> {: .solution}
{: .question}

> <hands-on-title>Filtering Based on Substring Occurrence</hands-on-title>
>
> Ok, great, now that you've got the hang of writing expressions for this tool, let's filter the file for rows where column 11 contains "Summer":
>
> 1. {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `olympics.tsv`
>    - {% icon param-text %} *"With the following condition"*: `c11.find('Summer') != -1`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
> 2. {% icon galaxy-eye %} **View** the filtered file.
>
>    > <question-title></question-title>
>    >
>    > 1. How many rows contained the string "Summer" in column 11? (Hint: use {% tool [Line/Word/Character count]({{version_wc}}) %} )
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. `189843`(this is including the header line)
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. **Rename** {% icon galaxy-pencil %} both outputs to something descriptive.
>
{: .hands-on }


### Filtering Rows that Start or End with a Specific Value

Filter rows based on whether a column starts or ends with a particular value using `cX.startswith()` or `cX.endswith()`.

For example, if we want to:
 - Find all athletes whose names start with "Liu".
 - Find all teams whose names end with "China".


> <question-title></question-title>
>
> 1. How would you filter rows where the 2nd column starts with "Liu"?
> 2. How would you filter rows where the 9th column ends with "China"?
>
> > <solution-title>Answers</solution-title>
> >
> > 1. `c2.startswith('Liu')`
> > 2. `c9.endswith('China')`
> >
> {: .solution}
{: .question}


> <hands-on-title>Filtering Rows that Start or End with a Specific Value</hands-on-title>
>
> Ok, great, now that you've got the hang of writing expressions for this tool, let's filter the file to find all athletes whose names start with "Liu":
>
> 1. {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `olympics.tsv`
>    - {% icon param-text %} *"With the following condition"*: `c2.startswith('Liu')`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
> 2. {% icon galaxy-eye %} **View** the filtered file.
>
>    > <question-title></question-title>
>    >
>    > 1. How many rows contained the string start with "Liu" in column 2? (Hint: expand the dataset in your history or use {% tool [Line/Word/Character count]({{version_wc}}) %} )
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. `324` (this is including the header line)
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. **Rename** {% icon galaxy-pencil %} both outputs to something descriptive.
>
{: .hands-on}


### Filtering Based on the Count of Substring Occurrences

Sometimes, you may need to filter rows based on **how many times** a certain substring appears in a column. This can be done using the `cX.count()` method.

For example, if we want to find athletes with three-part names by counting spaces in the name column.

> <hands-on-title>Filtering Based on the Count of Substring Occurrences</hands-on-title>
>
> We want to filter the dataset to find athletes with three-part names.
>
> 1. Have a look at the `olympics.tsv` file
>
>    > <question-title></question-title>
>    >
>    > 1. Which column contains the names of athletes?
>    > 2. For athletes with 3-part names, how many spaces do we expect in that column?
>    > 3. What expression would you use to find these cases?
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. Column 2 contains the names
>    > > 2. For 3-part names we expect 2 space characters
>    > > 3. `c2.count(' ') == 2`
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %} with the following parameters:
>     - {% icon param-file %} *"Filter"*: `olympics.tsv`
>     - {% icon param-text %} *"With the following condition"*: `c2.count(' ') == 2`
>     - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
> 3. {% icon galaxy-eye %} **View** the filtered file.
>
>     > <question-title></question-title>
>     >
>     > 1. How many rows contain three-part names (i.e., two spaces) in column 2?
>     >
>     > > <solution-title>Answers</solution-title>
>     > >
>     > > 1. `11259` (this is including the header line)
>     > >
>     > {: .solution}
>     {: .question}
>
> 4. **Exercises.** Now let's take it a step further and filter athletes with even longer names.
>
>    > <question-title></question-title>
>    >
>    > 1. How would you filter rows where the 2nd column contains **three spaces** (i.e., four-part names)?
>    > 2. How many medals were won by athletes with five-part names **or more**?
>    > 3. How many parts does the longest name have? Give an example of such a name.
>    > 4. Bonus: **How many** athletes have 6-part names won a medal?
>    >
>    > > <solution-title>Hints</solution-title>
>    > >
>    > > 1. None ;)
>    > > 2. You can also use greater than or less than symbols in your expression. You can filter on two lines using  logical operators like `and`.
>    > > 3. Keep increasing the count until no results are returned
>    > > 4. The number of lines returned indicates the number of medals won by long-named athletes, to
>    > >    find out how many different athletes have such a name, we can use the {% tool [Count - occurrences of each record](Count1) %}
>    > >    tool on Column 2.
>    > {: .solution}
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. `c2.count(' ') == 3`
>    > > 2. `c17 != 'NA' and c2.count(' ') >= 4` returns 27 lines, meaning 26 medals won accounting for the header line excluding the header line. Appart from the name you need to additionally filter out athletes that did not won a medal.
>    > > 3. `c2.count(' ') == 5` is the maximum value to still return results.
>    > >    Five spaces means a 6-part name. For example Patricia Galvin de la Tour d'Auvergne
>    > > 4. `c17 != 'NA' and c2.count(' ') == 5` returns 5 records. So 5 medals were won by these long-named athletes.
>    > >     Using the {% tool [Count - occurrences of each record](Count1) %} to count occurrences in column 2, we
>    > >     discover that there are **3 different athletes with 6-part names**.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


## Exercises

Ok, time to train! let's see if you can use the {% tool [Filter]({{version_filter}}) %} tool to answer the following questions:


> <question-title noprefix>Exercise: Medal winners</question-title>
>
> 1. How many gold medals were handed out?
> 2. How many total medals?
> 3. How many medals were handed out during the 2018 Olympics?
> 4. How many medals were won by individuals with a height between 170 and 180 cm?
> 5. How many gold medals were won by individuals shorter than 160cm or taller than 190?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > - Column 17 contains information about medals
> > - The possible values are `Gold`, `Silver`, `Bronze`, and `` (empty).
> > - Expand the output or use the tool {% tool [Line/Word/Character count]({{version_wc}}) %} to see the number of lines in the file
> > - Don't forget that the output (and line count) may include the header line
> > - Do not use quotes on number columns (e.g. year)
> > - You may need parentheses for complex conditions
> >
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. 8,110   (Expression: `c17=='Gold'`)
> >  2. 24,633  (Expression: `c17=='Gold' or c17=='Silver' or c17=='Bronze'`, or `c17!='NA'`)
> >  3. 131     (Expression: `c17=='Gold' and c12==2018` (note: do not use quotes around `2018`, as it is a numerical value))
> >  4. 8,086   (Expression: `c17!='NA' and c7>=170 and c7<=180`)
> >  5. 812     (Expression: `c17=='Gold' and (c7<160 or c7>190)` (note: parentheses are important here))
> >
> > Note: these numbers are found by determining the number of lines in the file after each filtering step, and subtracting 1 for the header line.
> >
> {: .solution}
{: .question}


# Counting

A common operation we might want to perform on tables of data, is simple counting. How many times does a certain value appear? For our dataset for instance, we might want to know how many countries participated in each Olympics, how many women, etc; any column that has categorical data that we can count. The tool {% tool [**Count** occurrences of each record]({{version_count}}) %} does exactly this.


> <hands-on-title>Count occurrences of values in columns</hands-on-title>
>
> Let's start by simply counting how many different Olympic Games we have in our dataset, and how many times it appears (so how many participations there were each year)
>
> 1. {% tool [**Count** occurrences of each record]({{version_count}})%} with the following parameters
>    - {% icon param-file %} *"from dataset"*: `olympics.tsv`
>    - {% icon param-select %} *"Count occurrences of values in column(s)"*: `Column 11`
>
> 2. {% icon galaxy-eye %} **View** the results.
>
>    > <question-title></question-title>
>    >
>    > 1. How many different Olympic games are in our file?
>    > 2. Which Olympic games had the most participations? (Tip: set the parameter *"How should the results be sorted?"* to `most common values first`)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. 52; there are 53 lines in the resulting file, with one line containing the value of the column header (`games`).
>    > >
>    > >    The resulting file looks something like:
>    > >
>    > >     ```
>    > >     615	1896 Summer Olympics
>    > >     2503	1900 Summer Olympics
>    > >     2643	1904 Summer Olympics
>    > >     3213	1908 Summer Olympics
>    > >     4610	1912 Summer Olympics
>    > >     3448	1920 Summer Olympics
>    > >     5242	1924 Summer Olympics
>    > >     358	1924 Winter Olympics
>    > >     4493	1928 Summer Olympics
>    > >     ...
>    > >     ```
>    > >
>    > > 2. 1996 Summer Olympics. (10501 participations)
>    > >
>    > {: .solution}
>    {: .question}
>
>    You may have noticed that we could have selected multiple columns in this tool. This lets us count on combinations of columns.
>    Let's try counting the number of men and women in each olympic games.
>
> 3. {% tool [**Count** occurrences of each record]({{version_count}})%} with the following parameters
>    - {% icon param-file %} *"from dataset"*: `olympics.tsv`
>    - {% icon param-select %} *"Count occurrences of values in column(s)"*: `Column 11, Column 3`
>
> 4. {% icon galaxy-eye %} **View** the results.
>
>    > <question-title></question-title>
>    >
>    > You see the resulting file has a line for every combination of the two columns (games and sex), providing the count for each.
>    >
>    > 1. How many women were in the first Olympic games?
>    >
>    > 2. Which Olympic games had the most women participants? (Tip: set the parameter *"How should the results be sorted?"* to `most common values first`)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. 2 women participated in the 1896 Olympics. (note that we cannot be sure if this is two different women, or 1 woman participating twice).
>    > >    The file looks something like this:
>    > >    ```
>    > >    2	F	1896 Summer Olympics
>    > >    43	F	1900 Summer Olympics
>    > >    17	F	1904 Summer Olympics
>    > >    55	F	1908 Summer Olympics
>    > >    97	F	1912 Summer Olympics
>    > >    132	F	1920 Summer Olympics
>    > >    269	F	1924 Summer Olympics
>    > >    ```
>    > >
>    > > 2. 2020 Summer Olympics (4652)
>    > >
>    > {: .solution}
>    {: .question}
>
>
{: .hands_on}


Let's say we wanted to know how many different sports there were in each Olympics. If we used the counting tool above, we would get a results file for each combination of sport and olympics, with the number of lines (participations) of each. But we don't really care about the number of lines that have this combination, just the total number of unique sports in each games.

To answer these types of questions we can use a slightly more advanced tool, called {% tool [Datamash]({{version_datamash}}) %}. This tool can do a lot more than counting, but here we will show you how to use it to count in this way.


> <hands-on-title>Counting number of different sports per Olympic Games</hands-on-title>
>
> We will now determine how many different sport there were in each of the different Olympics
>
> 1. {% tool [Datamash]({{version_datamash}}) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `olympics.tsv`
>    - {% icon param-text %} *"Group by fields"*: `11` (the `games` column)
>    - {% icon param-toggle %} *"Input file has a header line"*: `Yes`
>    - {% icon param-toggle %} *"Sort Input"*: `Yes`
>    - *"Operation to perform on each group"*:
>      - {% icon param-select %} *"Type"*: `Count Unique values`
>      - {% icon param-select %} *"On Column"*: `Column: 15` (the `sport` column)
>
> 2. {% icon galaxy-eye %} **View** the results.
>
>    > <question-title></question-title>
>    >
>    > 1. Why did we tell the tool to sort the file?
>    > 2. How many sport were in the first Olympics? How many in the latest?
>    > 3. Which Olympics had the most different sports?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. This is for technical reasons; the tool assumes that your file is sorted (on the column that you supplied to the "Group by fields" parameter).
>    > >    This assumption can greatly speed up the calculation, but will give incorrect results if the file was not actually sorted.
>    > >    If you know for sure that your data is sorted in the right way, you can skip this step. If you are unsure, enable this parameter to be safe, and the
>    > >    tool will perform the sorting step before its calculation.
>    > > 2. 10 and 38.
>    > > 3. The 2020 Summer Olympics had the most different sports (38)
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. **Rename** {% icon galaxy-pencil %} the output to something descriptive
>
{: .hands_on}


Note: Datamash can do a lot more than counting,  and we will showcase some of these other operatins in the [Grouping](#grouping) section.

## Exercises

Ok, let's practice!

> <question-title noprefix>Exercise: Number of participations per country</question-title>
>
> 1. Which country has had the most participations in the Olympics?
> 2. How many countries participated in the first Olympics? How many in the last?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Since we are counting participations (rows), we can use the simple {% tool [Count]({{version_count}})%} tool
> > 2. Since we are counting a bit more complex question, we need the {% tool [Datamash]({{version_datamash}}) %} tool
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. The United States with 17,286 participations
> >  2. 15 and 250.
> >
> {: .solution}
>
> > <solution-title noprefix>Full Solutions</solution-title>
> >
> >  1. {% tool [Count]({{version_count}}) %} with the following parameters:
> >     - {% icon param-file %} *"from dataset"*: `olympics.tsv`
> >     - {% icon param-select %} *"Count occurrences of values in column(s)"*: `Column 9` (the `team` column)
> >     - {% icon param-select %} *"How should the results be sorted?"*: `With the most common values first`
> >
> >     This gives an output like:
> >     ```
> >     17286	United States
> >     11700	France
> >     10230	Great Britain
> >     8898	Italy
> >     7988	Canada
> >     ```
> >
> >  2. {% tool [Datamash]({{version_datamash}}) %} with the following parameters:
> >   - {% icon param-file %} *"Input tabular dataset"*: `olympics.tsv`
> >   - {% icon param-text %} *"Group by fields"*: `11` (the `games` column)
> >   - {% icon param-toggle %} *"Input file has a header line"*: `yes`
> >   - {% icon param-toggle %} *"Sort Input"*: `yes`
> >   - *"Operation to perform on each group"*:
> >       - {% icon param-select %} *"Type"*: `Count Unique values`
> >       - {% icon param-select %} *"On Column"*: `Column: 9` (the `team` column)
> >
> >   This gives an output like:
> >   ```
> >   1896 Summer Olympics	15
> >   1900 Summer Olympics	29
> >   1904 Summer Olympics	13
> >   ...
> >   2016 Summer Olympics	280
> >   2018 Winter Olympics	108
> >   2020 Summer Olympics	250
> >   ```
> >
> {: .solution}
{: .question}








# Grouping

Often we may want to group rows based on a value in a column, and perform some operation on the resulting rows. For example we would like to group the olympics data by one value (e.g. year, country, sport), and determine some value for each group (e.g. number of medals won, average age of athletes, etc).

In the [counting](#counting) section of this tutorial we show how to get answers that require a count (e.g. number of medals won), but sometimes we want to do something more complex, like calculating the average height of athletes in a group, say per country or per sport. This section will show some example of these types of questions.

We can use the {% tool [Datamash]({{version_datamash}}) %} tool for this purpose.

> <hands-on-title>Tallest athlete per sport</hands-on-title>
>
> We would like to answer the following question: *How tall was the tallest athlete of each sport?*
>
> 1. Open the {% tool [Datamash]({{version_datamash}}) %} tool and read the help section at the bottom
>
>    > <question-title></question-title>
>    >
>    > Which settings do you think we need to provide to answer our question?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > - *"Group by fields"*: We want to group by sport (Column 15).
>    > > - *"Sort"*: `Yes`. This may not be obvious, but because our file is currently not sorted by our chosen group (sport), we need to tell the tool to do this.
>    > > - *"Skip NA or NaN values"*: since we do have NA values for athletes for whom height data is unknown, we should set this to `Yes`
>    > > - Our file has a header line, so we should indicate this as well
>    > > - *"Operation"*: we want to determine the **Maximum** height (Column 7)
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. {% tool [Datamash]({{version_datamash}}) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `olympics.tsv`
>    - {% icon param-text %} *"Group by fields"*: `15`
>    - {% icon param-toggle %} *"Sort input"*: `yes`
>    - {% icon param-toggle %} *"Input file has a header line"*: `yes`
>    - {% icon param-toggle %} *"Skip NA or NaN values"*: `yes`
>    - *"Operation to perform on each group"*:
>      - {% icon param-select %} *"Type"*: `Maximum`
>      - {% icon param-select %} *"On Column"*: `Column: 7`
>
> 3. {% icon galaxy-eye %} **View** the results.
>
>    > <question-title></question-title>
>    >
>    > 1. How tall was the tallest athlete in basketball? And what about karate?
>    > 2. Why do some sports have a value of `inf`?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Basketball's tallest athlete was 192cm. For Karate it is 163.
>    > > 2. Our dataset had quite a number of `NA` (unknown) values in the height column, especially for the earlier Olympics. For sports that had only NA values, there is no maximum so the tool outputs `inf` instead.
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


## Grouping on multiple columns

You may have noticed that we could also provide multiple columns to group on. If we do this, we can compute values for combinations of groups, such as sex and sport, to find e.g. the tallest woman in basketball or the shortest man per Olympics. There are also many more options for the computation we perform, so perhaps we are more interested not in the tallest athlete, but the average height. Let's perform some 

tools_json:
["Line/Word/Character count", "{{version_wc}}", "Sort", "{{version_sort}}", "Filter", "{{version_filter}}", "Count", "{{version_count}}", "Datamash", "{{version_datamash}}", "Group", "Grouping1", "Compute", "{{version_compute}}", "Column Regex Find and Replace", "{{version_replace_text_column}}", "Regex Find and Replace", "{{version_replace_text_line}}", "Join two Datasets", "{{version_join}}", "Concatenate multiple datasets or collections", "{{version_cat}}", "Remove beginning of a file", "{{version_remove_beginning}}", "Select first lines", "{{version_select_first}}", "Remove columns by heading", "{{version_remove_columns_by_header}}", "Cut columns from a table", "{{version_cut_columns}}", "Advanced Cut columns from a table", "{{version_cut_advanced}}", "Paste", "{{version_paste}}", "Split", "{{version_split}}", "Unique", "{{version_unique}}", "Import", "upload1", "wc_gnu", "Sort - data in ascending or descending order", "**Filter** data on any column using simple expressions", "Count - occurrences of each record", "Count1", "**Count** occurrences of each record", "Compute on rows", "Column Regex Finda and Replace", "Unique occurrences of each record", "Unique - occurrences of each record", "Join two Datasets side by side on a specified field", "Split by group", "Filter - data on any column using simple expressions", "**Count**"]


